TENSORFLOW DATA GENERATOR HOW TO
TENSORFLOW DATA GENERATOR CODE

The iterator is not aware of the number of elements present in the Dataset. We have learned various ways to create Datasets and apply various transformations to it, but how do we consume the data? Tensorflow provides Iterators to do that. The output is 0 2 4 6 8 10 12 14 16 18 Creating Iterators Useful in cases where you want to transform your raw data before feeding into the model.ĭef map_fnc(x): return x*2 data = np.arange(10) dataset = tf._tensor_slices(data) dataset = dataset.map(map_fnc) iterator = dataset.make_one_shot_iterator() next_ele = iterator.get_next() with tf.Session() as sess: try: while True: val = n(next_ele) print(val) except tf.errors.OutOfRangeError: pass Map: Used to transform the elements of the Dataset.The output is (array(), array()) (array(), array()) (array(), array()) (array(), array()) (array(), array())ĭataset = tf._tensor_slices(tf.range(10)) dataset = dataset.repeat(count=2) iterator = dataset.make_one_shot_iterator() next_ele = iterator.get_next() with tf.Session() as sess: try: while True: val = n(next_ele) print(val) except tf.errors.OutOfRangeError: pass Useful in scenarios where you have features and labels and you need to provide the pair of feature and label for training the model.ĭatax = np.arange(10,20) datay = np.arange(11,21) datasetx = tf._tensor_slices(datax) datasety = tf._tensor_slices(datay) dcombined = tf.((datasetx, datasety)).batch(2) iterator = dcombined.make_one_shot_iterator() next_ele = iterator.get_next() with tf.Session() as sess: try: while True: val = n(next_ele) print(val) except tf.errors.OutOfRangeError: pass

Zip: Creates a Dataset by zipping together datasets.We will learn about Iterators in detail in the later part of this blog.

TENSORFLOW DATA GENERATOR CODE

You can skip the code where we create an iterator and print the elements of the Dataset. Useful when you want to train smaller batches of data to avoid out of memory errors.ĭata = np.arange(10,40) # create batches of 10 dataset = tf._tensor_slices(data).batch(10) # creates the iterator to consume the data iterator = dataset.make_one_shot_iterator() next_ele = iterator.get_next() with tf.Session() as sess: try: while True: val = n(next_ele) print(val) except tf.errors.OutOfRangeError: pass

Batches: Combines consecutive elements of the Dataset into a single batch.

def generator(): for i in range(10): yield 2*i dataset = tf._generator(generator, (tf.int32)) Operations on Datasets from_generator: Creates a Dataset whose elements are generated by a function. Dataset created using this method will emit all the data at once.ĭata = tf.arange(10) dataset = tf._tensors(data)ģ.

from_tensors: It also accepts single or multiple numpy arrays or tensors.

The object dataset is a tensorflow Dataset object. # source data - numpy array data = np.arange(10) # create a dataset from numpy array dataset = tf._tensor_slices(data)

Dataset created using this method will emit only one data at a time. from_tensor_slices: It accepts single or multiple numpy arrays or tensors.Tensorflow provides various methods to create Datasets from numpy arrays, text files, CSV files, tensors, etc.

TENSORFLOW DATA GENERATOR HOW TO

We will learn how to create Datasets from source data, apply the transformation to Dataset and then consume the data using Iterators. In this blog post we will cover Datasets and Iterators.

Tensorflow data generator