Keras Generator

Updated March 16, 2023

Introduction to Keras Generator

Keras generator is requiring two generator one generator is used in data training and another generator is used for the purpose of validation. In the model class of keras, there are three types of method generators used i.e. evaluate generator, predict generator and fit generator. All three are data generators but not all the generators were created equally. It is basically used to generate the data models.

Key Takeaways

• Data augmentation encompasses the range of techniques used to generate the training samples from the original by applying jitters.
• Generator in keras like function instead of return keyword it will use the yield keyword. The return keyword will be terminating the function and return all the values of the dataset.

What is Keras Generator?

Python string is identifying the sample of the dataset. During the time using or training the classifier, we are not able to load the images into memory. Suppose we are using a small dataset then it is possible in this condition, but it is not good for large datasets. We need to use the potential to available the data. If we were to use a data generator, we could read the images while they were being used for training. At the time of reading images on the go, we can save the memory of our system using it.

How to Use Keras Generator?

The image generator class in keras is very useful for the classification of images. There are multiple ways to use it, basically, it is based on which method we are using. Below we are using the flow from the directory that will take the path to the containing directory. Images will be sorted into the subdirectories and augmentation parameters images.

The below example shows how we can use the keras generator as follows. First, we are importing the required libraries and then we are creating the data generators by using image augmentation as follows.

Code:

from tensorflow.keras.preprocessing.image import ImageDataGenerator
t_data = ImageDataGenerator (
rescale = 1./254,
shear_range = 0.3,
zoom_range = 0.7,
horizontal_flip = True)
test_data = ImageDataGenerator (rescale=1./254)
t_gen = t_data.flow_from_directory (
'keras',
target_size = (350, 350),
batch_size = 22,
class_mode = 'binary')
v_gen = test_data.flow_from_directory (
  'keras',
target_size = (350, 350),
batch_size = 22,
class_mode = 'binary')

Output:

In the below example, we can see how we use the keras generator. We are using the fit generator as follows.

Code:

model.fit_generator (
t_gen,
steps_per_epoch = 3000,
epochs = 50,
validation_data = v_gen,
validation_steps = 800)

Output:

The below example shows how we can use the flexible data generator. We are inheriting the gen class from seq class.

Code:

class DataGenerator (Seq):
'Keras generator'
def __init__(gen, list_ID, lab, img_path, m_path,
to_fit = True, dim = (256,256),
n_channels = 1, shuffle = True):
  'Initialization'
gen.list_ID = list_IDs
gen.lab = lab
gen.img_path = img_path
gen.m_path = m_path
gen.to_fit = to_fit
gen.batch_size = batch_size
gen.dim = dim
gen.n_channels = n_channels
gen.n_classes = n_classes
gen.shuffle = shuffle
gen.on_epoch_end ()

Output:

How to Create a Data Keras Generator?

Keras is providing generators for the image datasets, the same was available in the tf keras processing image into the generator class of image. The main advantage of using the image data generator class is that we can generate batches of data using the image data generator. The below example shows how we can create a keras data generator as follows.

Code:

from tensorflow.keras.preprocessing.image import ImageDataGenerator
t_gen = ImageDataGenerator(rescale=1./254)
t_size = (324, 324)
train_gen = t_gen.flow_from_directory (
  'keras',
target_size = t_size,
batch_size = 42)
model.fit (train_gen,
  epochs = num_epochs)

Output:

To create the custom data generator we need to write the simple generator itself. The custom generator is works with model.fit method. The below example shows a custom generator as follows.

Code:

class CustomDataGen (tf.keras.utils.Sequence):
def __init__(gen, df, col1, col2,
  batch_size,
  ip_size = (124, 124, 4),
  shuf = True):
gen.df = df.copy ()
gen.col1 = col1
gen.col1 = col2
gen.batch_size = batch_size
gen.ip_size = ip_size
gen.shuf = shuffle
gen.n = len (gen.df)
gen.n_name = df [col1['name']].nunique ()
gen.n_type = df[col1['type']].nunique()
def on_epoch_end (gen)

Output:

Method

In the model class of keras, there are three methods used. All three methods are requiring data generators.

• evaluate_generator – The data generator into the evaluate generator contains the same requirement which was in a fit generator.
• predict_generator – The predict generator only returns the inputs. In this type of data generator method, we are focusing on data generator methods.
• fit_generator – This method is used in it for data validation purposes.

The below example shows the keras generator method. We are creating the data generator as follows.

Code:

model.fit_generator(
t_gen,
steps_epoch = 1000,
epochs = 60,
val_data = validation_generator,
val_steps = 500)

Output:

Below is the keras data image generator method which was used to image data processing as follows:

1. flow_from_directory method – This method is very useful when the image was sorted and placed into a respective class. This method will be identifying the class automatically from the name of the folder. By using this method, we are using classes and directory values. Below is the method of flow from the directory method as follows.

Code:

model.flow_from_directory (
  t_gen,
  steps_epoch = 1000,
  epochs = 60,
  val_data = validation_generator,
  val_steps = 500)

Output:

2. flow_from_dataframe method – This method is useful when the image is clustered into any folder. By using this method, we are using the data frame and directory value. Below is the method of flow from the directory method as follows.

Code:

model.flow_from_dataframe (
t_gen,
steps_epoch = 1000,
epochs = 60,
val_data = validation_generator,
val_steps = 500)

Output:

Keras fit_generator Function

This function requires two generators one is for validation, and another is for data. Both methods is returning the tuple and both contain sequence class instances. Keras is providing the data generator which was limited to the specified capabilities. Every task in the fit generator function requires a data loader. Sometimes image contains the mask which was saved in the image. The below example shows the keras fit generator function as follows.

Code:

model.fit_generator (
t_gen,
steps_epoch = 3000,
epochs = 40,
val_data = validation_generator,
val_steps = 700)

Output:

Examples of Keras Generator

Given below are the examples mentioned:

Example #1

In the following example, we are importing the keras and numpy model.

Code:

import numpy as np
from keras.models import Sequential
par = {'dim': (12, 12,12),
  'batch_size': 34,
  'n_classes': 5,
  'n_channels': 1,
  'shuffle': True}
t_gen = DataGenerator(partition['train'], labels, **par)
v_gen = DataGenerator(partition['validation'], labels, **par)
model = Sequential()
model.compile ()
model.fit_generator(generator = t_gen,
validation_data = v_gen,
use_multiprocessing = True,
workers = 5)

Output:

Example #2

In the below example, we are using predict generator as follows.

Code:

model.predict_generator (
t_gen,
steps_epoch = 3000,
epochs = 40,
val_data = validation_generator,
val_steps = 700)

Output:

FAQ

Given below are the FAQs mentioned:

Q1. What is the use of keras generator in python?

Answer: Data generator in python is a mandatory module for the deep learning-based method which was defined in it.

Q2. What is the use of fit generator in keras?

Answer: This generator will require two generator first is for validation and the other is for data. Both the generator is returning the tuple values.

Q3. What is the use of evaluate_generator method in keras?

Answer: The data generator and evaluate generator contains the same requirement which was fit into the generator and it is the same as a generator for training.

Conclusion

Python string is identifying the sample of the dataset. At the time of using or training the classifier, we are not able to load the images into memory. It is requiring two generator one generator is used in data training and another generator is used for the purpose of validation.

Recommended Articles

This is a guide to Keras Generator. Here we discuss the introduction, and how to create a data keras generator. methods, functions, and examples. You may also have a look at the following articles to learn more –

1. Keras Tuner
2. What is Keras GPU?
3. Keras Embedding
4. Keras predict