===============
Reproducibility
===============

Reproducible-ML results are a very important aspect in research.
Unfortunately, often not much time is spent on it. In the following
we show how reproducible deals with randomness.

Setting a random seed
---------------------

Reproducibility is achieved by fixing random seeds.
First we consider the :ref:`input_fn_label`.

1. Shuffle, repeat and prefetch function:

.. code-block:: python
   :emphasize-lines: 7

   def shuffle_repeat_prefetch(dataset, buffer_size, n_epochs, random_seed):

       dataset = dataset.apply(
           tf.contrib.data.shuffle_and_repeat(
               buffer_size=buffer_size,
               count=n_epochs,
               seed=random_seed))
       dataset = dataset.prefetch(buffer_size)

   return dataset

2. Datasets from records function:

.. code-block:: python
   :emphasize-lines: 6

   def dataset_from_records(split, record_dir, record_pattern, random_seed):

       record_pattern = os.path.join(record_dir,
                                     record_pattern.format(split=split, idx="*"))

       dataset = tf.data.Dataset.list_files(record_pattern, seed=random_seed)

   return dataset

By adding a fixed random seed to this two functions, we achieve the desired reproducibility
of the experiment.

These two functions are then called within the generic input function:

.. code-block:: python
   :emphasize-lines: 10,14

   def input_fn(split,
                batch_size,
                buffer_size,
                num_parallel,
                compression,
                random_seed):
      def fn():
         ...

         dataset = dataset_from_records(split, random_seed=random_seed)

         ...

         dataset = shuffle_repeat_prefetch(dataset, random_seed)



The second part is controlling the randomness within the experiment itself. Lets see
how this works for the MNIST GAN (:ref:`mnist_exp_label`).

1. The extended input function:

.. code-block:: python
   :emphasize-lines: 7

   def extended_input_fn(noise_dims, mnist_feeder, gan_type):

       def add_noise_and_swap(feature_dict, label_dict):
           X = (tf.to_float(feature_dict.pop("X")) - 128.0) / 128.0

           feature_dict["noise"] = tf.random_normal(
                [mnist_feeder["batch_size"], noise_dims], seed=mnist_feeder["random_seed"])

           feature_dict["labels"] = label_dict["y"]

       return feature_dict, X

       def fn():
           dataset = mnist_input_fn()()

           dataset = dataset.map(add_noise_and_swap,
               num_parallel_calls=mnist_feeder["num_parallel"])

       return dataset

   return fn

2. The main function in the MNIST GAN experiment:

.. code-block:: python
   :emphasize-lines: 6

   def main(model_dir, save_summary_steps, save_checkpoints_steps,
       log_step_count_steps, gen_lr, crit_lr, max_train_steps, gan_type, mnist_feeder):

        config = tf.estimator.RunConfig(
            model_dir=model_dir,
            tf_random_seed=mnist_feeder["random_seed"],
            save_summary_steps=save_summary_steps,
            save_checkpoints_steps=save_checkpoints_steps,
            log_step_count_steps=log_step_count_steps
    )



Results
-------

To show the reproducibility of an experiment we ran the MNIST unconditional GAN experiment
two times on CPUs and compared the loss values.

.. code-block:: console

   python -m exps.mnist.gan with mnist_feeder.random_seed=123


.. figure:: /images/reproducibility/exp.png
   :align: center

Loss 1 and Loss 2 are identical. Therefore perfect reproducibility is achieved.

.. note::
   If you run this experiments on a GPU, reproducible results can not
   be achieved. It is impossible because on a GPU some non-deterministic results
   are expected.