.. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_plot_unet_tutorial.py: UNet Tutorial ================================ A tentative tutorial on the usage of the unet framework in inferno Preface -------------- We start with some unspectacular multi purpose imports needed for this example .. code-block:: python import matplotlib.pyplot as plt import torch import numpy should CUDA be used .. code-block:: python USE_CUDA = True Dataset -------------- For simplicity we will use a toy dataset where we need to perform a binary segmentation task. .. code-block:: python from inferno.io.box.binary_blobs import get_binary_blob_loaders # convert labels from long to float as needed by # binary cross entropy loss def label_transform(x): return torch.from_numpy(x).float() #label_transform = lambda x : torch.from_numpy(x).float() train_loader, test_loader, validate_loader = get_binary_blob_loaders( size=8, # how many images per {train,test,validate} train_batch_size=2, length=256, # <= size of the images gaussian_noise_sigma=1.4, # <= how noise are the images train_label_transform = label_transform, validate_label_transform = label_transform ) image_channels = 1 # <-- number of channels of the image pred_channels = 1 # <-- number of channels needed for the prediction Visualize Dataset ~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: python fig = plt.figure() for i,(image, target) in enumerate(train_loader): ax = fig.add_subplot(1, 2, 1) ax.imshow(image[0,0,...]) ax.set_title('raw data') ax = fig.add_subplot(1, 2, 2) ax.imshow(target[0,...]) ax.set_title('ground truth') break fig.tight_layout() plt.show() .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_001.png :class: sphx-glr-single-img Simple UNet ---------------------------- We start with a very simple predefined res block UNet. By default, this UNet uses ReLUs (in conjunction with batchnorm) as nonlinearities With :code:`activated=False` we make sure that the last layer is not activated since we chain the UNet with a sigmoid activation function. .. code-block:: python from inferno.extensions.layers import ResBlockUNet from inferno.extensions.layers import RemoveSingletonDimension model = torch.nn.Sequential( ResBlockUNet(dim=2, in_channels=image_channels, out_channels=pred_channels, activated=False), RemoveSingletonDimension(dim=1), torch.nn.Sigmoid() ) while the model above will work in principal, it has some drawbacks. Within the UNet, the number of features is increased by a multiplicative factor while going down, the so-called gain. The default value for the gain is 2. Since we start with only a single channel we could either increase the gain, or use a some convolutions to increase the number of channels before the the UNet. .. code-block:: python from inferno.extensions.layers import ConvReLU2D model_a = torch.nn.Sequential( ConvReLU2D(in_channels=image_channels, out_channels=5, kernel_size=3), ResBlockUNet(dim=2, in_channels=5, out_channels=pred_channels, activated=False, res_block_kwargs=dict(batchnorm=True,size=2)) , RemoveSingletonDimension(dim=1) # torch.nn.Sigmoid() ) Training ---------------------------- To train the unet, we use the infernos Trainer class of inferno. Since we train many models later on in this example we encapsulate the training in a function (see :ref:`sphx_glr_auto_examples_trainer.py` for an example dedicated to the trainer itself). .. code-block:: python from inferno.trainers import Trainer from inferno.utils.python_utils import ensure_dir def train_model(model, loaders, **kwargs): trainer = Trainer(model) trainer.build_criterion('BCEWithLogitsLoss') trainer.build_optimizer('Adam', lr=kwargs.get('lr', 0.0001)) #trainer.validate_every((kwargs.get('validate_every', 10), 'epochs')) #trainer.save_every((kwargs.get('save_every', 10), 'epochs')) #trainer.save_to_directory(ensure_dir(kwargs.get('save_dir', 'save_dor'))) trainer.set_max_num_epochs(kwargs.get('max_num_epochs', 200)) # bind the loaders trainer.bind_loader('train', loaders[0]) trainer.bind_loader('validate', loaders[1]) if USE_CUDA: trainer.cuda() # do the training trainer.fit() return trainer trainer = train_model(model=model_a, loaders=[train_loader, validate_loader], save_dir='model_a', lr=0.01) .. rst-class:: sphx-glr-script-out Out: .. code-block:: none [+][2018-08-15 16:20:17.295373] [PROGRESS] Training iteration 0 (batch 0 of epoch 0). [+][2018-08-15 16:20:18.036916] [PROGRESS] Training iteration 1 (batch 1 of epoch 0). [+][2018-08-15 16:20:18.068146] [PROGRESS] Training iteration 2 (batch 2 of epoch 0). [+][2018-08-15 16:20:18.100219] [PROGRESS] Training iteration 3 (batch 3 of epoch 0). [+][2018-08-15 16:20:18.131403] [PROGRESS] Training iteration 4 (batch 4 of epoch 0). [+][2018-08-15 16:20:18.229081] [PROGRESS] Training iteration 5 (batch 1 of epoch 1). [+][2018-08-15 16:20:18.259421] [PROGRESS] Training iteration 6 (batch 2 of epoch 1). [+][2018-08-15 16:20:18.288840] [PROGRESS] Training iteration 7 (batch 3 of epoch 1). [+][2018-08-15 16:20:18.318687] [PROGRESS] Training iteration 8 (batch 4 of epoch 1). [+][2018-08-15 16:20:18.397091] [PROGRESS] Training iteration 9 (batch 1 of epoch 2). [+][2018-08-15 16:20:18.428243] [PROGRESS] Training iteration 10 (batch 2 of epoch 2). [+][2018-08-15 16:20:18.457211] [PROGRESS] Training iteration 11 (batch 3 of epoch 2). [+][2018-08-15 16:20:18.484822] [PROGRESS] Training iteration 12 (batch 4 of epoch 2). [+][2018-08-15 16:20:18.553671] [PROGRESS] Training iteration 13 (batch 1 of epoch 3). [+][2018-08-15 16:20:18.581987] [PROGRESS] Training iteration 14 (batch 2 of epoch 3). [+][2018-08-15 16:20:18.609276] [PROGRESS] Training iteration 15 (batch 3 of epoch 3). [+][2018-08-15 16:20:18.635813] [PROGRESS] Training iteration 16 (batch 4 of epoch 3). [+][2018-08-15 16:20:18.703494] [PROGRESS] Training iteration 17 (batch 1 of epoch 4). [+][2018-08-15 16:20:18.731083] [PROGRESS] Training iteration 18 (batch 2 of epoch 4). [+][2018-08-15 16:20:18.758724] [PROGRESS] Training iteration 19 (batch 3 of epoch 4). [+][2018-08-15 16:20:18.785728] [PROGRESS] Training iteration 20 (batch 4 of epoch 4). [+][2018-08-15 16:20:18.855972] [PROGRESS] Training iteration 21 (batch 1 of epoch 5). [+][2018-08-15 16:20:18.883729] [PROGRESS] Training iteration 22 (batch 2 of epoch 5). [+][2018-08-15 16:20:18.911043] [PROGRESS] Training iteration 23 (batch 3 of epoch 5). [+][2018-08-15 16:20:18.937953] [PROGRESS] Training iteration 24 (batch 4 of epoch 5). [+][2018-08-15 16:20:19.006560] [PROGRESS] Training iteration 25 (batch 1 of epoch 6). [+][2018-08-15 16:20:19.035473] [PROGRESS] Training iteration 26 (batch 2 of epoch 6). [+][2018-08-15 16:20:19.062102] [PROGRESS] Training iteration 27 (batch 3 of epoch 6). [+][2018-08-15 16:20:19.089259] [PROGRESS] Training iteration 28 (batch 4 of epoch 6). [+][2018-08-15 16:20:19.158295] [PROGRESS] Training iteration 29 (batch 1 of epoch 7). [+][2018-08-15 16:20:19.186058] [PROGRESS] Training iteration 30 (batch 2 of epoch 7). [+][2018-08-15 16:20:19.213578] [PROGRESS] Training iteration 31 (batch 3 of epoch 7). [+][2018-08-15 16:20:19.240970] [PROGRESS] Training iteration 32 (batch 4 of epoch 7). [+][2018-08-15 16:20:19.309748] [PROGRESS] Training iteration 33 (batch 1 of epoch 8). [+][2018-08-15 16:20:19.337749] [PROGRESS] Training iteration 34 (batch 2 of epoch 8). [+][2018-08-15 16:20:19.365190] [PROGRESS] Training iteration 35 (batch 3 of epoch 8). [+][2018-08-15 16:20:19.392030] [PROGRESS] Training iteration 36 (batch 4 of epoch 8). [+][2018-08-15 16:20:19.460941] [PROGRESS] Training iteration 37 (batch 1 of epoch 9). [+][2018-08-15 16:20:19.488064] [PROGRESS] Training iteration 38 (batch 2 of epoch 9). [+][2018-08-15 16:20:19.515067] [PROGRESS] Training iteration 39 (batch 3 of epoch 9). [+][2018-08-15 16:20:19.542502] [PROGRESS] Training iteration 40 (batch 4 of epoch 9). [+][2018-08-15 16:20:19.610859] [PROGRESS] Training iteration 41 (batch 1 of epoch 10). [+][2018-08-15 16:20:19.639195] [PROGRESS] Training iteration 42 (batch 2 of epoch 10). [+][2018-08-15 16:20:19.666690] [PROGRESS] Training iteration 43 (batch 3 of epoch 10). [+][2018-08-15 16:20:19.693920] [PROGRESS] Training iteration 44 (batch 4 of epoch 10). [+][2018-08-15 16:20:19.766611] [PROGRESS] Training iteration 45 (batch 1 of epoch 11). [+][2018-08-15 16:20:19.794987] [PROGRESS] Training iteration 46 (batch 2 of epoch 11). [+][2018-08-15 16:20:19.822192] [PROGRESS] Training iteration 47 (batch 3 of epoch 11). [+][2018-08-15 16:20:19.849083] [PROGRESS] Training iteration 48 (batch 4 of epoch 11). [+][2018-08-15 16:20:19.918012] [PROGRESS] Training iteration 49 (batch 1 of epoch 12). [+][2018-08-15 16:20:19.946303] [PROGRESS] Training iteration 50 (batch 2 of epoch 12). [+][2018-08-15 16:20:19.973725] [PROGRESS] Training iteration 51 (batch 3 of epoch 12). [+][2018-08-15 16:20:20.000821] [PROGRESS] Training iteration 52 (batch 4 of epoch 12). [+][2018-08-15 16:20:20.069068] [PROGRESS] Training iteration 53 (batch 1 of epoch 13). [+][2018-08-15 16:20:20.097288] [PROGRESS] Training iteration 54 (batch 2 of epoch 13). [+][2018-08-15 16:20:20.125114] [PROGRESS] Training iteration 55 (batch 3 of epoch 13). [+][2018-08-15 16:20:20.151991] [PROGRESS] Training iteration 56 (batch 4 of epoch 13). [+][2018-08-15 16:20:20.220698] [PROGRESS] Training iteration 57 (batch 1 of epoch 14). [+][2018-08-15 16:20:20.248793] [PROGRESS] Training iteration 58 (batch 2 of epoch 14). [+][2018-08-15 16:20:20.276577] [PROGRESS] Training iteration 59 (batch 3 of epoch 14). [+][2018-08-15 16:20:20.303909] [PROGRESS] Training iteration 60 (batch 4 of epoch 14). [+][2018-08-15 16:20:20.373201] [PROGRESS] Training iteration 61 (batch 1 of epoch 15). [+][2018-08-15 16:20:20.401363] [PROGRESS] Training iteration 62 (batch 2 of epoch 15). [+][2018-08-15 16:20:20.428829] [PROGRESS] Training iteration 63 (batch 3 of epoch 15). [+][2018-08-15 16:20:20.456270] [PROGRESS] Training iteration 64 (batch 4 of epoch 15). [+][2018-08-15 16:20:20.525587] [PROGRESS] Training iteration 65 (batch 1 of epoch 16). [+][2018-08-15 16:20:20.553782] [PROGRESS] Training iteration 66 (batch 2 of epoch 16). [+][2018-08-15 16:20:20.581056] [PROGRESS] Training iteration 67 (batch 3 of epoch 16). [+][2018-08-15 16:20:20.608441] [PROGRESS] Training iteration 68 (batch 4 of epoch 16). [+][2018-08-15 16:20:20.678230] [PROGRESS] Training iteration 69 (batch 1 of epoch 17). [+][2018-08-15 16:20:20.706819] [PROGRESS] Training iteration 70 (batch 2 of epoch 17). [+][2018-08-15 16:20:20.734523] [PROGRESS] Training iteration 71 (batch 3 of epoch 17). [+][2018-08-15 16:20:20.761732] [PROGRESS] Training iteration 72 (batch 4 of epoch 17). [+][2018-08-15 16:20:20.830710] [PROGRESS] Training iteration 73 (batch 1 of epoch 18). [+][2018-08-15 16:20:20.859097] [PROGRESS] Training iteration 74 (batch 2 of epoch 18). [+][2018-08-15 16:20:20.886499] [PROGRESS] Training iteration 75 (batch 3 of epoch 18). [+][2018-08-15 16:20:20.915018] [PROGRESS] Training iteration 76 (batch 4 of epoch 18). [+][2018-08-15 16:20:20.984153] [PROGRESS] Training iteration 77 (batch 1 of epoch 19). [+][2018-08-15 16:20:21.013017] [PROGRESS] Training iteration 78 (batch 2 of epoch 19). [+][2018-08-15 16:20:21.040640] [PROGRESS] Training iteration 79 (batch 3 of epoch 19). [+][2018-08-15 16:20:21.067162] [PROGRESS] Training iteration 80 (batch 4 of epoch 19). [+][2018-08-15 16:20:21.137025] [PROGRESS] Training iteration 81 (batch 1 of epoch 20). [+][2018-08-15 16:20:21.165138] [PROGRESS] Training iteration 82 (batch 2 of epoch 20). [+][2018-08-15 16:20:21.192404] [PROGRESS] Training iteration 83 (batch 3 of epoch 20). [+][2018-08-15 16:20:21.220172] [PROGRESS] Training iteration 84 (batch 4 of epoch 20). [+][2018-08-15 16:20:21.293651] [PROGRESS] Training iteration 85 (batch 1 of epoch 21). [+][2018-08-15 16:20:21.321479] [PROGRESS] Training iteration 86 (batch 2 of epoch 21). [+][2018-08-15 16:20:21.349344] [PROGRESS] Training iteration 87 (batch 3 of epoch 21). [+][2018-08-15 16:20:21.376952] [PROGRESS] Training iteration 88 (batch 4 of epoch 21). [+][2018-08-15 16:20:21.445531] [PROGRESS] Training iteration 89 (batch 1 of epoch 22). [+][2018-08-15 16:20:21.474089] [PROGRESS] Training iteration 90 (batch 2 of epoch 22). [+][2018-08-15 16:20:21.501119] [PROGRESS] Training iteration 91 (batch 3 of epoch 22). [+][2018-08-15 16:20:21.528224] [PROGRESS] Training iteration 92 (batch 4 of epoch 22). [+][2018-08-15 16:20:21.597698] [PROGRESS] Training iteration 93 (batch 1 of epoch 23). [+][2018-08-15 16:20:21.625692] [PROGRESS] Training iteration 94 (batch 2 of epoch 23). [+][2018-08-15 16:20:21.653124] [PROGRESS] Training iteration 95 (batch 3 of epoch 23). [+][2018-08-15 16:20:21.680642] [PROGRESS] Training iteration 96 (batch 4 of epoch 23). [+][2018-08-15 16:20:21.750041] [PROGRESS] Training iteration 97 (batch 1 of epoch 24). [+][2018-08-15 16:20:21.778559] [PROGRESS] Training iteration 98 (batch 2 of epoch 24). [+][2018-08-15 16:20:21.805562] [PROGRESS] Training iteration 99 (batch 3 of epoch 24). [+][2018-08-15 16:20:21.832131] [PROGRESS] Training iteration 100 (batch 4 of epoch 24). [+][2018-08-15 16:20:21.900921] [PROGRESS] Training iteration 101 (batch 1 of epoch 25). [+][2018-08-15 16:20:21.928568] [PROGRESS] Training iteration 102 (batch 2 of epoch 25). [+][2018-08-15 16:20:21.955496] [PROGRESS] Training iteration 103 (batch 3 of epoch 25). [+][2018-08-15 16:20:21.983093] [PROGRESS] Training iteration 104 (batch 4 of epoch 25). [+][2018-08-15 16:20:22.053630] [PROGRESS] Training iteration 105 (batch 1 of epoch 26). [+][2018-08-15 16:20:22.082201] [PROGRESS] Training iteration 106 (batch 2 of epoch 26). [+][2018-08-15 16:20:22.109125] [PROGRESS] Training iteration 107 (batch 3 of epoch 26). [+][2018-08-15 16:20:22.136626] [PROGRESS] Training iteration 108 (batch 4 of epoch 26). [+][2018-08-15 16:20:22.205844] [PROGRESS] Training iteration 109 (batch 1 of epoch 27). [+][2018-08-15 16:20:22.234270] [PROGRESS] Training iteration 110 (batch 2 of epoch 27). [+][2018-08-15 16:20:22.261421] [PROGRESS] Training iteration 111 (batch 3 of epoch 27). [+][2018-08-15 16:20:22.288374] [PROGRESS] Training iteration 112 (batch 4 of epoch 27). [+][2018-08-15 16:20:22.357288] [PROGRESS] Training iteration 113 (batch 1 of epoch 28). [+][2018-08-15 16:20:22.385046] [PROGRESS] Training iteration 114 (batch 2 of epoch 28). [+][2018-08-15 16:20:22.412616] [PROGRESS] Training iteration 115 (batch 3 of epoch 28). [+][2018-08-15 16:20:22.439777] [PROGRESS] Training iteration 116 (batch 4 of epoch 28). [+][2018-08-15 16:20:22.508065] [PROGRESS] Training iteration 117 (batch 1 of epoch 29). [+][2018-08-15 16:20:22.535820] [PROGRESS] Training iteration 118 (batch 2 of epoch 29). [+][2018-08-15 16:20:22.564097] [PROGRESS] Training iteration 119 (batch 3 of epoch 29). [+][2018-08-15 16:20:22.590906] [PROGRESS] Training iteration 120 (batch 4 of epoch 29). [+][2018-08-15 16:20:22.665524] [PROGRESS] Training iteration 121 (batch 1 of epoch 30). [+][2018-08-15 16:20:22.693706] [PROGRESS] Training iteration 122 (batch 2 of epoch 30). [+][2018-08-15 16:20:22.721778] [PROGRESS] Training iteration 123 (batch 3 of epoch 30). [+][2018-08-15 16:20:22.749285] [PROGRESS] Training iteration 124 (batch 4 of epoch 30). [+][2018-08-15 16:20:22.819722] [PROGRESS] Training iteration 125 (batch 1 of epoch 31). [+][2018-08-15 16:20:22.847709] [PROGRESS] Training iteration 126 (batch 2 of epoch 31). [+][2018-08-15 16:20:22.875198] [PROGRESS] Training iteration 127 (batch 3 of epoch 31). [+][2018-08-15 16:20:22.903173] [PROGRESS] Training iteration 128 (batch 4 of epoch 31). [+][2018-08-15 16:20:22.972016] [PROGRESS] Training iteration 129 (batch 1 of epoch 32). [+][2018-08-15 16:20:23.000427] [PROGRESS] Training iteration 130 (batch 2 of epoch 32). [+][2018-08-15 16:20:23.027818] [PROGRESS] Training iteration 131 (batch 3 of epoch 32). [+][2018-08-15 16:20:23.055025] [PROGRESS] Training iteration 132 (batch 4 of epoch 32). [+][2018-08-15 16:20:23.123925] [PROGRESS] Training iteration 133 (batch 1 of epoch 33). [+][2018-08-15 16:20:23.152179] [PROGRESS] Training iteration 134 (batch 2 of epoch 33). [+][2018-08-15 16:20:23.179179] [PROGRESS] Training iteration 135 (batch 3 of epoch 33). [+][2018-08-15 16:20:23.206837] [PROGRESS] Training iteration 136 (batch 4 of epoch 33). [+][2018-08-15 16:20:23.276927] [PROGRESS] Training iteration 137 (batch 1 of epoch 34). [+][2018-08-15 16:20:23.304747] [PROGRESS] Training iteration 138 (batch 2 of epoch 34). [+][2018-08-15 16:20:23.332843] [PROGRESS] Training iteration 139 (batch 3 of epoch 34). [+][2018-08-15 16:20:23.359558] [PROGRESS] Training iteration 140 (batch 4 of epoch 34). [+][2018-08-15 16:20:23.430779] [PROGRESS] Training iteration 141 (batch 1 of epoch 35). [+][2018-08-15 16:20:23.458681] [PROGRESS] Training iteration 142 (batch 2 of epoch 35). [+][2018-08-15 16:20:23.485707] [PROGRESS] Training iteration 143 (batch 3 of epoch 35). [+][2018-08-15 16:20:23.512932] [PROGRESS] Training iteration 144 (batch 4 of epoch 35). [+][2018-08-15 16:20:23.581747] [PROGRESS] Training iteration 145 (batch 1 of epoch 36). [+][2018-08-15 16:20:23.610371] [PROGRESS] Training iteration 146 (batch 2 of epoch 36). [+][2018-08-15 16:20:23.637819] [PROGRESS] Training iteration 147 (batch 3 of epoch 36). [+][2018-08-15 16:20:23.665264] [PROGRESS] Training iteration 148 (batch 4 of epoch 36). [+][2018-08-15 16:20:23.734557] [PROGRESS] Training iteration 149 (batch 1 of epoch 37). [+][2018-08-15 16:20:23.762947] [PROGRESS] Training iteration 150 (batch 2 of epoch 37). [+][2018-08-15 16:20:23.790486] [PROGRESS] Training iteration 151 (batch 3 of epoch 37). [+][2018-08-15 16:20:23.817666] [PROGRESS] Training iteration 152 (batch 4 of epoch 37). [+][2018-08-15 16:20:23.885861] [PROGRESS] Training iteration 153 (batch 1 of epoch 38). [+][2018-08-15 16:20:23.914024] [PROGRESS] Training iteration 154 (batch 2 of epoch 38). [+][2018-08-15 16:20:23.941857] [PROGRESS] Training iteration 155 (batch 3 of epoch 38). [+][2018-08-15 16:20:23.968521] [PROGRESS] Training iteration 156 (batch 4 of epoch 38). [+][2018-08-15 16:20:24.037402] [PROGRESS] Training iteration 157 (batch 1 of epoch 39). [+][2018-08-15 16:20:24.066543] [PROGRESS] Training iteration 158 (batch 2 of epoch 39). [+][2018-08-15 16:20:24.094436] [PROGRESS] Training iteration 159 (batch 3 of epoch 39). [+][2018-08-15 16:20:24.122155] [PROGRESS] Training iteration 160 (batch 4 of epoch 39). [+][2018-08-15 16:20:24.190948] [PROGRESS] Training iteration 161 (batch 1 of epoch 40). [+][2018-08-15 16:20:24.218578] [PROGRESS] Training iteration 162 (batch 2 of epoch 40). [+][2018-08-15 16:20:24.246130] [PROGRESS] Training iteration 163 (batch 3 of epoch 40). [+][2018-08-15 16:20:24.273160] [PROGRESS] Training iteration 164 (batch 4 of epoch 40). [+][2018-08-15 16:20:24.341745] [PROGRESS] Training iteration 165 (batch 1 of epoch 41). [+][2018-08-15 16:20:24.369368] [PROGRESS] Training iteration 166 (batch 2 of epoch 41). [+][2018-08-15 16:20:24.396531] [PROGRESS] Training iteration 167 (batch 3 of epoch 41). [+][2018-08-15 16:20:24.423528] [PROGRESS] Training iteration 168 (batch 4 of epoch 41). [+][2018-08-15 16:20:24.496689] [PROGRESS] Training iteration 169 (batch 1 of epoch 42). [+][2018-08-15 16:20:24.525619] [PROGRESS] Training iteration 170 (batch 2 of epoch 42). [+][2018-08-15 16:20:24.553078] [PROGRESS] Training iteration 171 (batch 3 of epoch 42). [+][2018-08-15 16:20:24.579904] [PROGRESS] Training iteration 172 (batch 4 of epoch 42). [+][2018-08-15 16:20:24.648922] [PROGRESS] Training iteration 173 (batch 1 of epoch 43). [+][2018-08-15 16:20:24.676910] [PROGRESS] Training iteration 174 (batch 2 of epoch 43). [+][2018-08-15 16:20:24.704799] [PROGRESS] Training iteration 175 (batch 3 of epoch 43). [+][2018-08-15 16:20:24.731465] [PROGRESS] Training iteration 176 (batch 4 of epoch 43). [+][2018-08-15 16:20:24.800191] [PROGRESS] Training iteration 177 (batch 1 of epoch 44). [+][2018-08-15 16:20:24.828148] [PROGRESS] Training iteration 178 (batch 2 of epoch 44). [+][2018-08-15 16:20:24.855969] [PROGRESS] Training iteration 179 (batch 3 of epoch 44). [+][2018-08-15 16:20:24.883824] [PROGRESS] Training iteration 180 (batch 4 of epoch 44). [+][2018-08-15 16:20:24.953911] [PROGRESS] Training iteration 181 (batch 1 of epoch 45). [+][2018-08-15 16:20:24.981743] [PROGRESS] Training iteration 182 (batch 2 of epoch 45). [+][2018-08-15 16:20:25.009546] [PROGRESS] Training iteration 183 (batch 3 of epoch 45). [+][2018-08-15 16:20:25.036424] [PROGRESS] Training iteration 184 (batch 4 of epoch 45). [+][2018-08-15 16:20:25.106090] [PROGRESS] Training iteration 185 (batch 1 of epoch 46). [+][2018-08-15 16:20:25.134244] [PROGRESS] Training iteration 186 (batch 2 of epoch 46). [+][2018-08-15 16:20:25.161243] [PROGRESS] Training iteration 187 (batch 3 of epoch 46). [+][2018-08-15 16:20:25.188779] [PROGRESS] Training iteration 188 (batch 4 of epoch 46). [+][2018-08-15 16:20:25.259089] [PROGRESS] Training iteration 189 (batch 1 of epoch 47). [+][2018-08-15 16:20:25.287741] [PROGRESS] Training iteration 190 (batch 2 of epoch 47). [+][2018-08-15 16:20:25.315568] [PROGRESS] Training iteration 191 (batch 3 of epoch 47). [+][2018-08-15 16:20:25.342397] [PROGRESS] Training iteration 192 (batch 4 of epoch 47). [+][2018-08-15 16:20:25.411185] [PROGRESS] Training iteration 193 (batch 1 of epoch 48). [+][2018-08-15 16:20:25.439670] [PROGRESS] Training iteration 194 (batch 2 of epoch 48). [+][2018-08-15 16:20:25.467764] [PROGRESS] Training iteration 195 (batch 3 of epoch 48). [+][2018-08-15 16:20:25.495235] [PROGRESS] Training iteration 196 (batch 4 of epoch 48). [+][2018-08-15 16:20:25.564020] [PROGRESS] Training iteration 197 (batch 1 of epoch 49). [+][2018-08-15 16:20:25.592527] [PROGRESS] Training iteration 198 (batch 2 of epoch 49). [+][2018-08-15 16:20:25.619682] [PROGRESS] Training iteration 199 (batch 3 of epoch 49). [+][2018-08-15 16:20:25.646644] [PROGRESS] Training iteration 200 (batch 4 of epoch 49). [+][2018-08-15 16:20:25.715073] [PROGRESS] Training iteration 201 (batch 1 of epoch 50). [+][2018-08-15 16:20:25.742933] [PROGRESS] Training iteration 202 (batch 2 of epoch 50). [+][2018-08-15 16:20:25.770618] [PROGRESS] Training iteration 203 (batch 3 of epoch 50). [+][2018-08-15 16:20:25.797213] [PROGRESS] Training iteration 204 (batch 4 of epoch 50). [+][2018-08-15 16:20:25.866765] [PROGRESS] Training iteration 205 (batch 1 of epoch 51). [+][2018-08-15 16:20:25.895346] [PROGRESS] Training iteration 206 (batch 2 of epoch 51). [+][2018-08-15 16:20:25.922394] [PROGRESS] Training iteration 207 (batch 3 of epoch 51). [+][2018-08-15 16:20:25.949414] [PROGRESS] Training iteration 208 (batch 4 of epoch 51). [+][2018-08-15 16:20:26.017029] [PROGRESS] Training iteration 209 (batch 1 of epoch 52). [+][2018-08-15 16:20:26.045194] [PROGRESS] Training iteration 210 (batch 2 of epoch 52). [+][2018-08-15 16:20:26.072835] [PROGRESS] Training iteration 211 (batch 3 of epoch 52). [+][2018-08-15 16:20:26.099936] [PROGRESS] Training iteration 212 (batch 4 of epoch 52). [+][2018-08-15 16:20:26.169126] [PROGRESS] Training iteration 213 (batch 1 of epoch 53). [+][2018-08-15 16:20:26.197081] [PROGRESS] Training iteration 214 (batch 2 of epoch 53). [+][2018-08-15 16:20:26.224446] [PROGRESS] Training iteration 215 (batch 3 of epoch 53). [+][2018-08-15 16:20:26.251479] [PROGRESS] Training iteration 216 (batch 4 of epoch 53). [+][2018-08-15 16:20:26.320842] [PROGRESS] Training iteration 217 (batch 1 of epoch 54). [+][2018-08-15 16:20:26.348320] [PROGRESS] Training iteration 218 (batch 2 of epoch 54). [+][2018-08-15 16:20:26.375786] [PROGRESS] Training iteration 219 (batch 3 of epoch 54). [+][2018-08-15 16:20:26.403282] [PROGRESS] Training iteration 220 (batch 4 of epoch 54). [+][2018-08-15 16:20:26.472288] [PROGRESS] Training iteration 221 (batch 1 of epoch 55). [+][2018-08-15 16:20:26.500476] [PROGRESS] Training iteration 222 (batch 2 of epoch 55). [+][2018-08-15 16:20:26.527779] [PROGRESS] Training iteration 223 (batch 3 of epoch 55). [+][2018-08-15 16:20:26.555418] [PROGRESS] Training iteration 224 (batch 4 of epoch 55). [+][2018-08-15 16:20:26.625003] [PROGRESS] Training iteration 225 (batch 1 of epoch 56). [+][2018-08-15 16:20:26.653083] [PROGRESS] Training iteration 226 (batch 2 of epoch 56). [+][2018-08-15 16:20:26.680294] [PROGRESS] Training iteration 227 (batch 3 of epoch 56). [+][2018-08-15 16:20:26.708348] [PROGRESS] Training iteration 228 (batch 4 of epoch 56). [+][2018-08-15 16:20:26.776521] [PROGRESS] Training iteration 229 (batch 1 of epoch 57). [+][2018-08-15 16:20:26.804230] [PROGRESS] Training iteration 230 (batch 2 of epoch 57). [+][2018-08-15 16:20:26.831728] [PROGRESS] Training iteration 231 (batch 3 of epoch 57). [+][2018-08-15 16:20:26.858517] [PROGRESS] Training iteration 232 (batch 4 of epoch 57). [+][2018-08-15 16:20:26.928583] [PROGRESS] Training iteration 233 (batch 1 of epoch 58). [+][2018-08-15 16:20:26.956722] [PROGRESS] Training iteration 234 (batch 2 of epoch 58). [+][2018-08-15 16:20:26.983967] [PROGRESS] Training iteration 235 (batch 3 of epoch 58). [+][2018-08-15 16:20:27.011295] [PROGRESS] Training iteration 236 (batch 4 of epoch 58). [+][2018-08-15 16:20:27.081053] [PROGRESS] Training iteration 237 (batch 1 of epoch 59). [+][2018-08-15 16:20:27.108697] [PROGRESS] Training iteration 238 (batch 2 of epoch 59). [+][2018-08-15 16:20:27.136103] [PROGRESS] Training iteration 239 (batch 3 of epoch 59). [+][2018-08-15 16:20:27.163598] [PROGRESS] Training iteration 240 (batch 4 of epoch 59). [+][2018-08-15 16:20:27.233357] [PROGRESS] Training iteration 241 (batch 1 of epoch 60). [+][2018-08-15 16:20:27.261941] [PROGRESS] Training iteration 242 (batch 2 of epoch 60). [+][2018-08-15 16:20:27.289637] [PROGRESS] Training iteration 243 (batch 3 of epoch 60). [+][2018-08-15 16:20:27.317326] [PROGRESS] Training iteration 244 (batch 4 of epoch 60). [+][2018-08-15 16:20:27.386370] [PROGRESS] Training iteration 245 (batch 1 of epoch 61). [+][2018-08-15 16:20:27.414612] [PROGRESS] Training iteration 246 (batch 2 of epoch 61). [+][2018-08-15 16:20:27.442657] [PROGRESS] Training iteration 247 (batch 3 of epoch 61). [+][2018-08-15 16:20:27.469892] [PROGRESS] Training iteration 248 (batch 4 of epoch 61). [+][2018-08-15 16:20:27.539147] [PROGRESS] Training iteration 249 (batch 1 of epoch 62). [+][2018-08-15 16:20:27.567510] [PROGRESS] Training iteration 250 (batch 2 of epoch 62). [+][2018-08-15 16:20:27.594439] [PROGRESS] Training iteration 251 (batch 3 of epoch 62). [+][2018-08-15 16:20:27.621093] [PROGRESS] Training iteration 252 (batch 4 of epoch 62). [+][2018-08-15 16:20:27.690534] [PROGRESS] Training iteration 253 (batch 1 of epoch 63). [+][2018-08-15 16:20:27.719205] [PROGRESS] Training iteration 254 (batch 2 of epoch 63). [+][2018-08-15 16:20:27.746670] [PROGRESS] Training iteration 255 (batch 3 of epoch 63). [+][2018-08-15 16:20:27.773697] [PROGRESS] Training iteration 256 (batch 4 of epoch 63). [+][2018-08-15 16:20:27.843250] [PROGRESS] Training iteration 257 (batch 1 of epoch 64). [+][2018-08-15 16:20:27.871632] [PROGRESS] Training iteration 258 (batch 2 of epoch 64). [+][2018-08-15 16:20:27.898974] [PROGRESS] Training iteration 259 (batch 3 of epoch 64). [+][2018-08-15 16:20:27.926804] [PROGRESS] Training iteration 260 (batch 4 of epoch 64). [+][2018-08-15 16:20:27.996990] [PROGRESS] Training iteration 261 (batch 1 of epoch 65). [+][2018-08-15 16:20:28.025307] [PROGRESS] Training iteration 262 (batch 2 of epoch 65). [+][2018-08-15 16:20:28.052554] [PROGRESS] Training iteration 263 (batch 3 of epoch 65). [+][2018-08-15 16:20:28.079425] [PROGRESS] Training iteration 264 (batch 4 of epoch 65). [+][2018-08-15 16:20:28.148569] [PROGRESS] Training iteration 265 (batch 1 of epoch 66). [+][2018-08-15 16:20:28.177465] [PROGRESS] Training iteration 266 (batch 2 of epoch 66). [+][2018-08-15 16:20:28.205333] [PROGRESS] Training iteration 267 (batch 3 of epoch 66). [+][2018-08-15 16:20:28.232025] [PROGRESS] Training iteration 268 (batch 4 of epoch 66). [+][2018-08-15 16:20:28.301486] [PROGRESS] Training iteration 269 (batch 1 of epoch 67). [+][2018-08-15 16:20:28.329606] [PROGRESS] Training iteration 270 (batch 2 of epoch 67). [+][2018-08-15 16:20:28.357000] [PROGRESS] Training iteration 271 (batch 3 of epoch 67). [+][2018-08-15 16:20:28.384306] [PROGRESS] Training iteration 272 (batch 4 of epoch 67). [+][2018-08-15 16:20:28.453988] [PROGRESS] Training iteration 273 (batch 1 of epoch 68). [+][2018-08-15 16:20:28.481989] [PROGRESS] Training iteration 274 (batch 2 of epoch 68). [+][2018-08-15 16:20:28.509807] [PROGRESS] Training iteration 275 (batch 3 of epoch 68). [+][2018-08-15 16:20:28.537169] [PROGRESS] Training iteration 276 (batch 4 of epoch 68). [+][2018-08-15 16:20:28.606054] [PROGRESS] Training iteration 277 (batch 1 of epoch 69). [+][2018-08-15 16:20:28.634369] [PROGRESS] Training iteration 278 (batch 2 of epoch 69). [+][2018-08-15 16:20:28.662438] [PROGRESS] Training iteration 279 (batch 3 of epoch 69). [+][2018-08-15 16:20:28.689706] [PROGRESS] Training iteration 280 (batch 4 of epoch 69). [+][2018-08-15 16:20:28.759012] [PROGRESS] Training iteration 281 (batch 1 of epoch 70). [+][2018-08-15 16:20:28.787260] [PROGRESS] Training iteration 282 (batch 2 of epoch 70). [+][2018-08-15 16:20:28.815384] [PROGRESS] Training iteration 283 (batch 3 of epoch 70). [+][2018-08-15 16:20:28.842752] [PROGRESS] Training iteration 284 (batch 4 of epoch 70). [+][2018-08-15 16:20:28.911546] [PROGRESS] Training iteration 285 (batch 1 of epoch 71). [+][2018-08-15 16:20:28.939853] [PROGRESS] Training iteration 286 (batch 2 of epoch 71). [+][2018-08-15 16:20:28.967434] [PROGRESS] Training iteration 287 (batch 3 of epoch 71). [+][2018-08-15 16:20:28.994548] [PROGRESS] Training iteration 288 (batch 4 of epoch 71). [+][2018-08-15 16:20:29.064277] [PROGRESS] Training iteration 289 (batch 1 of epoch 72). [+][2018-08-15 16:20:29.092296] [PROGRESS] Training iteration 290 (batch 2 of epoch 72). [+][2018-08-15 16:20:29.119626] [PROGRESS] Training iteration 291 (batch 3 of epoch 72). [+][2018-08-15 16:20:29.146824] [PROGRESS] Training iteration 292 (batch 4 of epoch 72). [+][2018-08-15 16:20:29.215779] [PROGRESS] Training iteration 293 (batch 1 of epoch 73). [+][2018-08-15 16:20:29.244068] [PROGRESS] Training iteration 294 (batch 2 of epoch 73). [+][2018-08-15 16:20:29.271357] [PROGRESS] Training iteration 295 (batch 3 of epoch 73). [+][2018-08-15 16:20:29.298758] [PROGRESS] Training iteration 296 (batch 4 of epoch 73). [+][2018-08-15 16:20:29.366905] [PROGRESS] Training iteration 297 (batch 1 of epoch 74). [+][2018-08-15 16:20:29.395372] [PROGRESS] Training iteration 298 (batch 2 of epoch 74). [+][2018-08-15 16:20:29.422750] [PROGRESS] Training iteration 299 (batch 3 of epoch 74). [+][2018-08-15 16:20:29.449994] [PROGRESS] Training iteration 300 (batch 4 of epoch 74). [+][2018-08-15 16:20:29.519250] [PROGRESS] Training iteration 301 (batch 1 of epoch 75). [+][2018-08-15 16:20:29.546900] [PROGRESS] Training iteration 302 (batch 2 of epoch 75). [+][2018-08-15 16:20:29.574702] [PROGRESS] Training iteration 303 (batch 3 of epoch 75). [+][2018-08-15 16:20:29.601769] [PROGRESS] Training iteration 304 (batch 4 of epoch 75). [+][2018-08-15 16:20:29.670298] [PROGRESS] Training iteration 305 (batch 1 of epoch 76). [+][2018-08-15 16:20:29.698779] [PROGRESS] Training iteration 306 (batch 2 of epoch 76). [+][2018-08-15 16:20:29.726361] [PROGRESS] Training iteration 307 (batch 3 of epoch 76). [+][2018-08-15 16:20:29.753575] [PROGRESS] Training iteration 308 (batch 4 of epoch 76). [+][2018-08-15 16:20:29.822630] [PROGRESS] Training iteration 309 (batch 1 of epoch 77). [+][2018-08-15 16:20:29.850675] [PROGRESS] Training iteration 310 (batch 2 of epoch 77). [+][2018-08-15 16:20:29.878670] [PROGRESS] Training iteration 311 (batch 3 of epoch 77). [+][2018-08-15 16:20:29.906017] [PROGRESS] Training iteration 312 (batch 4 of epoch 77). [+][2018-08-15 16:20:29.974957] [PROGRESS] Training iteration 313 (batch 1 of epoch 78). [+][2018-08-15 16:20:30.003005] [PROGRESS] Training iteration 314 (batch 2 of epoch 78). [+][2018-08-15 16:20:30.030543] [PROGRESS] Training iteration 315 (batch 3 of epoch 78). [+][2018-08-15 16:20:30.057627] [PROGRESS] Training iteration 316 (batch 4 of epoch 78). [+][2018-08-15 16:20:30.126552] [PROGRESS] Training iteration 317 (batch 1 of epoch 79). [+][2018-08-15 16:20:30.154834] [PROGRESS] Training iteration 318 (batch 2 of epoch 79). [+][2018-08-15 16:20:30.182276] [PROGRESS] Training iteration 319 (batch 3 of epoch 79). [+][2018-08-15 16:20:30.209396] [PROGRESS] Training iteration 320 (batch 4 of epoch 79). [+][2018-08-15 16:20:30.277818] [PROGRESS] Training iteration 321 (batch 1 of epoch 80). [+][2018-08-15 16:20:30.306597] [PROGRESS] Training iteration 322 (batch 2 of epoch 80). [+][2018-08-15 16:20:30.334028] [PROGRESS] Training iteration 323 (batch 3 of epoch 80). [+][2018-08-15 16:20:30.360725] [PROGRESS] Training iteration 324 (batch 4 of epoch 80). [+][2018-08-15 16:20:30.429940] [PROGRESS] Training iteration 325 (batch 1 of epoch 81). [+][2018-08-15 16:20:30.458460] [PROGRESS] Training iteration 326 (batch 2 of epoch 81). [+][2018-08-15 16:20:30.486039] [PROGRESS] Training iteration 327 (batch 3 of epoch 81). [+][2018-08-15 16:20:30.513199] [PROGRESS] Training iteration 328 (batch 4 of epoch 81). [+][2018-08-15 16:20:30.582961] [PROGRESS] Training iteration 329 (batch 1 of epoch 82). [+][2018-08-15 16:20:30.610989] [PROGRESS] Training iteration 330 (batch 2 of epoch 82). [+][2018-08-15 16:20:30.638425] [PROGRESS] Training iteration 331 (batch 3 of epoch 82). [+][2018-08-15 16:20:30.666235] [PROGRESS] Training iteration 332 (batch 4 of epoch 82). [+][2018-08-15 16:20:30.734935] [PROGRESS] Training iteration 333 (batch 1 of epoch 83). [+][2018-08-15 16:20:30.762491] [PROGRESS] Training iteration 334 (batch 2 of epoch 83). [+][2018-08-15 16:20:30.790074] [PROGRESS] Training iteration 335 (batch 3 of epoch 83). [+][2018-08-15 16:20:30.816809] [PROGRESS] Training iteration 336 (batch 4 of epoch 83). [+][2018-08-15 16:20:30.885195] [PROGRESS] Training iteration 337 (batch 1 of epoch 84). [+][2018-08-15 16:20:30.913419] [PROGRESS] Training iteration 338 (batch 2 of epoch 84). [+][2018-08-15 16:20:30.940593] [PROGRESS] Training iteration 339 (batch 3 of epoch 84). [+][2018-08-15 16:20:30.967783] [PROGRESS] Training iteration 340 (batch 4 of epoch 84). [+][2018-08-15 16:20:31.036166] [PROGRESS] Training iteration 341 (batch 1 of epoch 85). [+][2018-08-15 16:20:31.064214] [PROGRESS] Training iteration 342 (batch 2 of epoch 85). [+][2018-08-15 16:20:31.092112] [PROGRESS] Training iteration 343 (batch 3 of epoch 85). [+][2018-08-15 16:20:31.119081] [PROGRESS] Training iteration 344 (batch 4 of epoch 85). [+][2018-08-15 16:20:31.187721] [PROGRESS] Training iteration 345 (batch 1 of epoch 86). [+][2018-08-15 16:20:31.215907] [PROGRESS] Training iteration 346 (batch 2 of epoch 86). [+][2018-08-15 16:20:31.242867] [PROGRESS] Training iteration 347 (batch 3 of epoch 86). [+][2018-08-15 16:20:31.269877] [PROGRESS] Training iteration 348 (batch 4 of epoch 86). [+][2018-08-15 16:20:31.338749] [PROGRESS] Training iteration 349 (batch 1 of epoch 87). [+][2018-08-15 16:20:31.366618] [PROGRESS] Training iteration 350 (batch 2 of epoch 87). [+][2018-08-15 16:20:31.394388] [PROGRESS] Training iteration 351 (batch 3 of epoch 87). [+][2018-08-15 16:20:31.420846] [PROGRESS] Training iteration 352 (batch 4 of epoch 87). [+][2018-08-15 16:20:31.489882] [PROGRESS] Training iteration 353 (batch 1 of epoch 88). [+][2018-08-15 16:20:31.518144] [PROGRESS] Training iteration 354 (batch 2 of epoch 88). [+][2018-08-15 16:20:31.545673] [PROGRESS] Training iteration 355 (batch 3 of epoch 88). [+][2018-08-15 16:20:31.572599] [PROGRESS] Training iteration 356 (batch 4 of epoch 88). [+][2018-08-15 16:20:31.642241] [PROGRESS] Training iteration 357 (batch 1 of epoch 89). [+][2018-08-15 16:20:31.670123] [PROGRESS] Training iteration 358 (batch 2 of epoch 89). [+][2018-08-15 16:20:31.698078] [PROGRESS] Training iteration 359 (batch 3 of epoch 89). [+][2018-08-15 16:20:31.725018] [PROGRESS] Training iteration 360 (batch 4 of epoch 89). [+][2018-08-15 16:20:31.793489] [PROGRESS] Training iteration 361 (batch 1 of epoch 90). [+][2018-08-15 16:20:31.821977] [PROGRESS] Training iteration 362 (batch 2 of epoch 90). [+][2018-08-15 16:20:31.849415] [PROGRESS] Training iteration 363 (batch 3 of epoch 90). [+][2018-08-15 16:20:31.876907] [PROGRESS] Training iteration 364 (batch 4 of epoch 90). [+][2018-08-15 16:20:31.945355] [PROGRESS] Training iteration 365 (batch 1 of epoch 91). [+][2018-08-15 16:20:31.973997] [PROGRESS] Training iteration 366 (batch 2 of epoch 91). [+][2018-08-15 16:20:32.001798] [PROGRESS] Training iteration 367 (batch 3 of epoch 91). [+][2018-08-15 16:20:32.028685] [PROGRESS] Training iteration 368 (batch 4 of epoch 91). [+][2018-08-15 16:20:32.100445] [PROGRESS] Training iteration 369 (batch 1 of epoch 92). [+][2018-08-15 16:20:32.128897] [PROGRESS] Training iteration 370 (batch 2 of epoch 92). [+][2018-08-15 16:20:32.157291] [PROGRESS] Training iteration 371 (batch 3 of epoch 92). [+][2018-08-15 16:20:32.183613] [PROGRESS] Training iteration 372 (batch 4 of epoch 92). [+][2018-08-15 16:20:32.253121] [PROGRESS] Training iteration 373 (batch 1 of epoch 93). [+][2018-08-15 16:20:32.281051] [PROGRESS] Training iteration 374 (batch 2 of epoch 93). [+][2018-08-15 16:20:32.308594] [PROGRESS] Training iteration 375 (batch 3 of epoch 93). [+][2018-08-15 16:20:32.335844] [PROGRESS] Training iteration 376 (batch 4 of epoch 93). [+][2018-08-15 16:20:32.405023] [PROGRESS] Training iteration 377 (batch 1 of epoch 94). [+][2018-08-15 16:20:32.433556] [PROGRESS] Training iteration 378 (batch 2 of epoch 94). [+][2018-08-15 16:20:32.460950] [PROGRESS] Training iteration 379 (batch 3 of epoch 94). [+][2018-08-15 16:20:32.488256] [PROGRESS] Training iteration 380 (batch 4 of epoch 94). [+][2018-08-15 16:20:32.557543] [PROGRESS] Training iteration 381 (batch 1 of epoch 95). [+][2018-08-15 16:20:32.586094] [PROGRESS] Training iteration 382 (batch 2 of epoch 95). [+][2018-08-15 16:20:32.614629] [PROGRESS] Training iteration 383 (batch 3 of epoch 95). [+][2018-08-15 16:20:32.642537] [PROGRESS] Training iteration 384 (batch 4 of epoch 95). [+][2018-08-15 16:20:32.710988] [PROGRESS] Training iteration 385 (batch 1 of epoch 96). [+][2018-08-15 16:20:32.738343] [PROGRESS] Training iteration 386 (batch 2 of epoch 96). [+][2018-08-15 16:20:32.767083] [PROGRESS] Training iteration 387 (batch 3 of epoch 96). [+][2018-08-15 16:20:32.793944] [PROGRESS] Training iteration 388 (batch 4 of epoch 96). [+][2018-08-15 16:20:32.863158] [PROGRESS] Training iteration 389 (batch 1 of epoch 97). [+][2018-08-15 16:20:32.891821] [PROGRESS] Training iteration 390 (batch 2 of epoch 97). [+][2018-08-15 16:20:32.919635] [PROGRESS] Training iteration 391 (batch 3 of epoch 97). [+][2018-08-15 16:20:32.947469] [PROGRESS] Training iteration 392 (batch 4 of epoch 97). [+][2018-08-15 16:20:33.016474] [PROGRESS] Training iteration 393 (batch 1 of epoch 98). [+][2018-08-15 16:20:33.044978] [PROGRESS] Training iteration 394 (batch 2 of epoch 98). [+][2018-08-15 16:20:33.072251] [PROGRESS] Training iteration 395 (batch 3 of epoch 98). [+][2018-08-15 16:20:33.099289] [PROGRESS] Training iteration 396 (batch 4 of epoch 98). [+][2018-08-15 16:20:33.168964] [PROGRESS] Training iteration 397 (batch 1 of epoch 99). [+][2018-08-15 16:20:33.197533] [PROGRESS] Training iteration 398 (batch 2 of epoch 99). [+][2018-08-15 16:20:33.224596] [PROGRESS] Training iteration 399 (batch 3 of epoch 99). [+][2018-08-15 16:20:33.251692] [PROGRESS] Training iteration 400 (batch 4 of epoch 99). [+][2018-08-15 16:20:33.320998] [PROGRESS] Training iteration 401 (batch 1 of epoch 100). [+][2018-08-15 16:20:33.348746] [PROGRESS] Training iteration 402 (batch 2 of epoch 100). [+][2018-08-15 16:20:33.375904] [PROGRESS] Training iteration 403 (batch 3 of epoch 100). [+][2018-08-15 16:20:33.402583] [PROGRESS] Training iteration 404 (batch 4 of epoch 100). [+][2018-08-15 16:20:33.471990] [PROGRESS] Training iteration 405 (batch 1 of epoch 101). [+][2018-08-15 16:20:33.500019] [PROGRESS] Training iteration 406 (batch 2 of epoch 101). [+][2018-08-15 16:20:33.527737] [PROGRESS] Training iteration 407 (batch 3 of epoch 101). [+][2018-08-15 16:20:33.554869] [PROGRESS] Training iteration 408 (batch 4 of epoch 101). [+][2018-08-15 16:20:33.624780] [PROGRESS] Training iteration 409 (batch 1 of epoch 102). [+][2018-08-15 16:20:33.653530] [PROGRESS] Training iteration 410 (batch 2 of epoch 102). [+][2018-08-15 16:20:33.681225] [PROGRESS] Training iteration 411 (batch 3 of epoch 102). [+][2018-08-15 16:20:33.709632] [PROGRESS] Training iteration 412 (batch 4 of epoch 102). [+][2018-08-15 16:20:33.778211] [PROGRESS] Training iteration 413 (batch 1 of epoch 103). [+][2018-08-15 16:20:33.806195] [PROGRESS] Training iteration 414 (batch 2 of epoch 103). [+][2018-08-15 16:20:33.833444] [PROGRESS] Training iteration 415 (batch 3 of epoch 103). [+][2018-08-15 16:20:33.860164] [PROGRESS] Training iteration 416 (batch 4 of epoch 103). [+][2018-08-15 16:20:33.929267] [PROGRESS] Training iteration 417 (batch 1 of epoch 104). [+][2018-08-15 16:20:33.957059] [PROGRESS] Training iteration 418 (batch 2 of epoch 104). [+][2018-08-15 16:20:33.984699] [PROGRESS] Training iteration 419 (batch 3 of epoch 104). [+][2018-08-15 16:20:34.011461] [PROGRESS] Training iteration 420 (batch 4 of epoch 104). [+][2018-08-15 16:20:34.081224] [PROGRESS] Training iteration 421 (batch 1 of epoch 105). [+][2018-08-15 16:20:34.109238] [PROGRESS] Training iteration 422 (batch 2 of epoch 105). [+][2018-08-15 16:20:34.136459] [PROGRESS] Training iteration 423 (batch 3 of epoch 105). [+][2018-08-15 16:20:34.163907] [PROGRESS] Training iteration 424 (batch 4 of epoch 105). [+][2018-08-15 16:20:34.233447] [PROGRESS] Training iteration 425 (batch 1 of epoch 106). [+][2018-08-15 16:20:34.260910] [PROGRESS] Training iteration 426 (batch 2 of epoch 106). [+][2018-08-15 16:20:34.288484] [PROGRESS] Training iteration 427 (batch 3 of epoch 106). [+][2018-08-15 16:20:34.315586] [PROGRESS] Training iteration 428 (batch 4 of epoch 106). [+][2018-08-15 16:20:34.384818] [PROGRESS] Training iteration 429 (batch 1 of epoch 107). [+][2018-08-15 16:20:34.413038] [PROGRESS] Training iteration 430 (batch 2 of epoch 107). [+][2018-08-15 16:20:34.440568] [PROGRESS] Training iteration 431 (batch 3 of epoch 107). [+][2018-08-15 16:20:34.467466] [PROGRESS] Training iteration 432 (batch 4 of epoch 107). [+][2018-08-15 16:20:34.537268] [PROGRESS] Training iteration 433 (batch 1 of epoch 108). [+][2018-08-15 16:20:34.565050] [PROGRESS] Training iteration 434 (batch 2 of epoch 108). [+][2018-08-15 16:20:34.592077] [PROGRESS] Training iteration 435 (batch 3 of epoch 108). [+][2018-08-15 16:20:34.619499] [PROGRESS] Training iteration 436 (batch 4 of epoch 108). [+][2018-08-15 16:20:34.687627] [PROGRESS] Training iteration 437 (batch 1 of epoch 109). [+][2018-08-15 16:20:34.715407] [PROGRESS] Training iteration 438 (batch 2 of epoch 109). [+][2018-08-15 16:20:34.743163] [PROGRESS] Training iteration 439 (batch 3 of epoch 109). [+][2018-08-15 16:20:34.770064] [PROGRESS] Training iteration 440 (batch 4 of epoch 109). [+][2018-08-15 16:20:34.839243] [PROGRESS] Training iteration 441 (batch 1 of epoch 110). [+][2018-08-15 16:20:34.867251] [PROGRESS] Training iteration 442 (batch 2 of epoch 110). [+][2018-08-15 16:20:34.894600] [PROGRESS] Training iteration 443 (batch 3 of epoch 110). [+][2018-08-15 16:20:34.921725] [PROGRESS] Training iteration 444 (batch 4 of epoch 110). [+][2018-08-15 16:20:34.990935] [PROGRESS] Training iteration 445 (batch 1 of epoch 111). [+][2018-08-15 16:20:35.019221] [PROGRESS] Training iteration 446 (batch 2 of epoch 111). [+][2018-08-15 16:20:35.047475] [PROGRESS] Training iteration 447 (batch 3 of epoch 111). [+][2018-08-15 16:20:35.074764] [PROGRESS] Training iteration 448 (batch 4 of epoch 111). [+][2018-08-15 16:20:35.145120] [PROGRESS] Training iteration 449 (batch 1 of epoch 112). [+][2018-08-15 16:20:35.173474] [PROGRESS] Training iteration 450 (batch 2 of epoch 112). [+][2018-08-15 16:20:35.201167] [PROGRESS] Training iteration 451 (batch 3 of epoch 112). [+][2018-08-15 16:20:35.229506] [PROGRESS] Training iteration 452 (batch 4 of epoch 112). [+][2018-08-15 16:20:35.298115] [PROGRESS] Training iteration 453 (batch 1 of epoch 113). [+][2018-08-15 16:20:35.325593] [PROGRESS] Training iteration 454 (batch 2 of epoch 113). [+][2018-08-15 16:20:35.352960] [PROGRESS] Training iteration 455 (batch 3 of epoch 113). [+][2018-08-15 16:20:35.379788] [PROGRESS] Training iteration 456 (batch 4 of epoch 113). [+][2018-08-15 16:20:35.449004] [PROGRESS] Training iteration 457 (batch 1 of epoch 114). [+][2018-08-15 16:20:35.476966] [PROGRESS] Training iteration 458 (batch 2 of epoch 114). [+][2018-08-15 16:20:35.504339] [PROGRESS] Training iteration 459 (batch 3 of epoch 114). [+][2018-08-15 16:20:35.532217] [PROGRESS] Training iteration 460 (batch 4 of epoch 114). [+][2018-08-15 16:20:35.602730] [PROGRESS] Training iteration 461 (batch 1 of epoch 115). [+][2018-08-15 16:20:35.630961] [PROGRESS] Training iteration 462 (batch 2 of epoch 115). [+][2018-08-15 16:20:35.657726] [PROGRESS] Training iteration 463 (batch 3 of epoch 115). [+][2018-08-15 16:20:35.685128] [PROGRESS] Training iteration 464 (batch 4 of epoch 115). [+][2018-08-15 16:20:35.755657] [PROGRESS] Training iteration 465 (batch 1 of epoch 116). [+][2018-08-15 16:20:35.783481] [PROGRESS] Training iteration 466 (batch 2 of epoch 116). [+][2018-08-15 16:20:35.810110] [PROGRESS] Training iteration 467 (batch 3 of epoch 116). [+][2018-08-15 16:20:35.837771] [PROGRESS] Training iteration 468 (batch 4 of epoch 116). [+][2018-08-15 16:20:35.907063] [PROGRESS] Training iteration 469 (batch 1 of epoch 117). [+][2018-08-15 16:20:35.936262] [PROGRESS] Training iteration 470 (batch 2 of epoch 117). [+][2018-08-15 16:20:35.963797] [PROGRESS] Training iteration 471 (batch 3 of epoch 117). [+][2018-08-15 16:20:35.994984] [PROGRESS] Training iteration 472 (batch 4 of epoch 117). [+][2018-08-15 16:20:36.063726] [PROGRESS] Training iteration 473 (batch 1 of epoch 118). [+][2018-08-15 16:20:36.092562] [PROGRESS] Training iteration 474 (batch 2 of epoch 118). [+][2018-08-15 16:20:36.119736] [PROGRESS] Training iteration 475 (batch 3 of epoch 118). [+][2018-08-15 16:20:36.146877] [PROGRESS] Training iteration 476 (batch 4 of epoch 118). [+][2018-08-15 16:20:36.215588] [PROGRESS] Training iteration 477 (batch 1 of epoch 119). [+][2018-08-15 16:20:36.243691] [PROGRESS] Training iteration 478 (batch 2 of epoch 119). [+][2018-08-15 16:20:36.271316] [PROGRESS] Training iteration 479 (batch 3 of epoch 119). [+][2018-08-15 16:20:36.298444] [PROGRESS] Training iteration 480 (batch 4 of epoch 119). [+][2018-08-15 16:20:36.367808] [PROGRESS] Training iteration 481 (batch 1 of epoch 120). [+][2018-08-15 16:20:36.396144] [PROGRESS] Training iteration 482 (batch 2 of epoch 120). [+][2018-08-15 16:20:36.424475] [PROGRESS] Training iteration 483 (batch 3 of epoch 120). [+][2018-08-15 16:20:36.451322] [PROGRESS] Training iteration 484 (batch 4 of epoch 120). [+][2018-08-15 16:20:36.520854] [PROGRESS] Training iteration 485 (batch 1 of epoch 121). [+][2018-08-15 16:20:36.549594] [PROGRESS] Training iteration 486 (batch 2 of epoch 121). [+][2018-08-15 16:20:36.576567] [PROGRESS] Training iteration 487 (batch 3 of epoch 121). [+][2018-08-15 16:20:36.604166] [PROGRESS] Training iteration 488 (batch 4 of epoch 121). [+][2018-08-15 16:20:36.673041] [PROGRESS] Training iteration 489 (batch 1 of epoch 122). [+][2018-08-15 16:20:36.701796] [PROGRESS] Training iteration 490 (batch 2 of epoch 122). [+][2018-08-15 16:20:36.729584] [PROGRESS] Training iteration 491 (batch 3 of epoch 122). [+][2018-08-15 16:20:36.757738] [PROGRESS] Training iteration 492 (batch 4 of epoch 122). [+][2018-08-15 16:20:36.827530] [PROGRESS] Training iteration 493 (batch 1 of epoch 123). [+][2018-08-15 16:20:36.855786] [PROGRESS] Training iteration 494 (batch 2 of epoch 123). [+][2018-08-15 16:20:36.883527] [PROGRESS] Training iteration 495 (batch 3 of epoch 123). [+][2018-08-15 16:20:36.910955] [PROGRESS] Training iteration 496 (batch 4 of epoch 123). [+][2018-08-15 16:20:36.980274] [PROGRESS] Training iteration 497 (batch 1 of epoch 124). [+][2018-08-15 16:20:37.008678] [PROGRESS] Training iteration 498 (batch 2 of epoch 124). [+][2018-08-15 16:20:37.035714] [PROGRESS] Training iteration 499 (batch 3 of epoch 124). [+][2018-08-15 16:20:37.063493] [PROGRESS] Training iteration 500 (batch 4 of epoch 124). [+][2018-08-15 16:20:37.132706] [PROGRESS] Training iteration 501 (batch 1 of epoch 125). [+][2018-08-15 16:20:37.160773] [PROGRESS] Training iteration 502 (batch 2 of epoch 125). [+][2018-08-15 16:20:37.188205] [PROGRESS] Training iteration 503 (batch 3 of epoch 125). [+][2018-08-15 16:20:37.215498] [PROGRESS] Training iteration 504 (batch 4 of epoch 125). [+][2018-08-15 16:20:37.284394] [PROGRESS] Training iteration 505 (batch 1 of epoch 126). [+][2018-08-15 16:20:37.312549] [PROGRESS] Training iteration 506 (batch 2 of epoch 126). [+][2018-08-15 16:20:37.341356] [PROGRESS] Training iteration 507 (batch 3 of epoch 126). [+][2018-08-15 16:20:37.367966] [PROGRESS] Training iteration 508 (batch 4 of epoch 126). [+][2018-08-15 16:20:37.438708] [PROGRESS] Training iteration 509 (batch 1 of epoch 127). [+][2018-08-15 16:20:37.467300] [PROGRESS] Training iteration 510 (batch 2 of epoch 127). [+][2018-08-15 16:20:37.494946] [PROGRESS] Training iteration 511 (batch 3 of epoch 127). [+][2018-08-15 16:20:37.522311] [PROGRESS] Training iteration 512 (batch 4 of epoch 127). [+][2018-08-15 16:20:37.591224] [PROGRESS] Training iteration 513 (batch 1 of epoch 128). [+][2018-08-15 16:20:37.620179] [PROGRESS] Training iteration 514 (batch 2 of epoch 128). [+][2018-08-15 16:20:37.647620] [PROGRESS] Training iteration 515 (batch 3 of epoch 128). [+][2018-08-15 16:20:37.674138] [PROGRESS] Training iteration 516 (batch 4 of epoch 128). [+][2018-08-15 16:20:37.748369] [PROGRESS] Training iteration 517 (batch 1 of epoch 129). [+][2018-08-15 16:20:37.776141] [PROGRESS] Training iteration 518 (batch 2 of epoch 129). [+][2018-08-15 16:20:37.803796] [PROGRESS] Training iteration 519 (batch 3 of epoch 129). [+][2018-08-15 16:20:37.831524] [PROGRESS] Training iteration 520 (batch 4 of epoch 129). [+][2018-08-15 16:20:37.900987] [PROGRESS] Training iteration 521 (batch 1 of epoch 130). [+][2018-08-15 16:20:37.928783] [PROGRESS] Training iteration 522 (batch 2 of epoch 130). [+][2018-08-15 16:20:37.956412] [PROGRESS] Training iteration 523 (batch 3 of epoch 130). [+][2018-08-15 16:20:37.983368] [PROGRESS] Training iteration 524 (batch 4 of epoch 130). [+][2018-08-15 16:20:38.051908] [PROGRESS] Training iteration 525 (batch 1 of epoch 131). [+][2018-08-15 16:20:38.079766] [PROGRESS] Training iteration 526 (batch 2 of epoch 131). [+][2018-08-15 16:20:38.107558] [PROGRESS] Training iteration 527 (batch 3 of epoch 131). [+][2018-08-15 16:20:38.134837] [PROGRESS] Training iteration 528 (batch 4 of epoch 131). [+][2018-08-15 16:20:38.203932] [PROGRESS] Training iteration 529 (batch 1 of epoch 132). [+][2018-08-15 16:20:38.232204] [PROGRESS] Training iteration 530 (batch 2 of epoch 132). [+][2018-08-15 16:20:38.259316] [PROGRESS] Training iteration 531 (batch 3 of epoch 132). [+][2018-08-15 16:20:38.286560] [PROGRESS] Training iteration 532 (batch 4 of epoch 132). [+][2018-08-15 16:20:38.356149] [PROGRESS] Training iteration 533 (batch 1 of epoch 133). [+][2018-08-15 16:20:38.383878] [PROGRESS] Training iteration 534 (batch 2 of epoch 133). [+][2018-08-15 16:20:38.411444] [PROGRESS] Training iteration 535 (batch 3 of epoch 133). [+][2018-08-15 16:20:38.438478] [PROGRESS] Training iteration 536 (batch 4 of epoch 133). [+][2018-08-15 16:20:38.507501] [PROGRESS] Training iteration 537 (batch 1 of epoch 134). [+][2018-08-15 16:20:38.535845] [PROGRESS] Training iteration 538 (batch 2 of epoch 134). [+][2018-08-15 16:20:38.563776] [PROGRESS] Training iteration 539 (batch 3 of epoch 134). [+][2018-08-15 16:20:38.591555] [PROGRESS] Training iteration 540 (batch 4 of epoch 134). [+][2018-08-15 16:20:38.660734] [PROGRESS] Training iteration 541 (batch 1 of epoch 135). [+][2018-08-15 16:20:38.688696] [PROGRESS] Training iteration 542 (batch 2 of epoch 135). [+][2018-08-15 16:20:38.716210] [PROGRESS] Training iteration 543 (batch 3 of epoch 135). [+][2018-08-15 16:20:38.743622] [PROGRESS] Training iteration 544 (batch 4 of epoch 135). [+][2018-08-15 16:20:38.813123] [PROGRESS] Training iteration 545 (batch 1 of epoch 136). [+][2018-08-15 16:20:38.840513] [PROGRESS] Training iteration 546 (batch 2 of epoch 136). [+][2018-08-15 16:20:38.867611] [PROGRESS] Training iteration 547 (batch 3 of epoch 136). [+][2018-08-15 16:20:38.895069] [PROGRESS] Training iteration 548 (batch 4 of epoch 136). [+][2018-08-15 16:20:38.964479] [PROGRESS] Training iteration 549 (batch 1 of epoch 137). [+][2018-08-15 16:20:38.992584] [PROGRESS] Training iteration 550 (batch 2 of epoch 137). [+][2018-08-15 16:20:39.020381] [PROGRESS] Training iteration 551 (batch 3 of epoch 137). [+][2018-08-15 16:20:39.047553] [PROGRESS] Training iteration 552 (batch 4 of epoch 137). [+][2018-08-15 16:20:39.119065] [PROGRESS] Training iteration 553 (batch 1 of epoch 138). [+][2018-08-15 16:20:39.149434] [PROGRESS] Training iteration 554 (batch 2 of epoch 138). [+][2018-08-15 16:20:39.178087] [PROGRESS] Training iteration 555 (batch 3 of epoch 138). [+][2018-08-15 16:20:39.205480] [PROGRESS] Training iteration 556 (batch 4 of epoch 138). [+][2018-08-15 16:20:39.274898] [PROGRESS] Training iteration 557 (batch 1 of epoch 139). [+][2018-08-15 16:20:39.303346] [PROGRESS] Training iteration 558 (batch 2 of epoch 139). [+][2018-08-15 16:20:39.330847] [PROGRESS] Training iteration 559 (batch 3 of epoch 139). [+][2018-08-15 16:20:39.358103] [PROGRESS] Training iteration 560 (batch 4 of epoch 139). [+][2018-08-15 16:20:39.426977] [PROGRESS] Training iteration 561 (batch 1 of epoch 140). [+][2018-08-15 16:20:39.455103] [PROGRESS] Training iteration 562 (batch 2 of epoch 140). [+][2018-08-15 16:20:39.482659] [PROGRESS] Training iteration 563 (batch 3 of epoch 140). [+][2018-08-15 16:20:39.509906] [PROGRESS] Training iteration 564 (batch 4 of epoch 140). [+][2018-08-15 16:20:39.578628] [PROGRESS] Training iteration 565 (batch 1 of epoch 141). [+][2018-08-15 16:20:39.606696] [PROGRESS] Training iteration 566 (batch 2 of epoch 141). [+][2018-08-15 16:20:39.634024] [PROGRESS] Training iteration 567 (batch 3 of epoch 141). [+][2018-08-15 16:20:39.661640] [PROGRESS] Training iteration 568 (batch 4 of epoch 141). [+][2018-08-15 16:20:39.731107] [PROGRESS] Training iteration 569 (batch 1 of epoch 142). [+][2018-08-15 16:20:39.760296] [PROGRESS] Training iteration 570 (batch 2 of epoch 142). [+][2018-08-15 16:20:39.789185] [PROGRESS] Training iteration 571 (batch 3 of epoch 142). [+][2018-08-15 16:20:39.816130] [PROGRESS] Training iteration 572 (batch 4 of epoch 142). [+][2018-08-15 16:20:39.885508] [PROGRESS] Training iteration 573 (batch 1 of epoch 143). [+][2018-08-15 16:20:39.914796] [PROGRESS] Training iteration 574 (batch 2 of epoch 143). [+][2018-08-15 16:20:39.942710] [PROGRESS] Training iteration 575 (batch 3 of epoch 143). [+][2018-08-15 16:20:39.969758] [PROGRESS] Training iteration 576 (batch 4 of epoch 143). [+][2018-08-15 16:20:40.039217] [PROGRESS] Training iteration 577 (batch 1 of epoch 144). [+][2018-08-15 16:20:40.066878] [PROGRESS] Training iteration 578 (batch 2 of epoch 144). [+][2018-08-15 16:20:40.094913] [PROGRESS] Training iteration 579 (batch 3 of epoch 144). [+][2018-08-15 16:20:40.122168] [PROGRESS] Training iteration 580 (batch 4 of epoch 144). [+][2018-08-15 16:20:40.191414] [PROGRESS] Training iteration 581 (batch 1 of epoch 145). [+][2018-08-15 16:20:40.220189] [PROGRESS] Training iteration 582 (batch 2 of epoch 145). [+][2018-08-15 16:20:40.247710] [PROGRESS] Training iteration 583 (batch 3 of epoch 145). [+][2018-08-15 16:20:40.274657] [PROGRESS] Training iteration 584 (batch 4 of epoch 145). [+][2018-08-15 16:20:40.344609] [PROGRESS] Training iteration 585 (batch 1 of epoch 146). [+][2018-08-15 16:20:40.373385] [PROGRESS] Training iteration 586 (batch 2 of epoch 146). [+][2018-08-15 16:20:40.400837] [PROGRESS] Training iteration 587 (batch 3 of epoch 146). [+][2018-08-15 16:20:40.427784] [PROGRESS] Training iteration 588 (batch 4 of epoch 146). [+][2018-08-15 16:20:40.497829] [PROGRESS] Training iteration 589 (batch 1 of epoch 147). [+][2018-08-15 16:20:40.526964] [PROGRESS] Training iteration 590 (batch 2 of epoch 147). [+][2018-08-15 16:20:40.555618] [PROGRESS] Training iteration 591 (batch 3 of epoch 147). [+][2018-08-15 16:20:40.583531] [PROGRESS] Training iteration 592 (batch 4 of epoch 147). [+][2018-08-15 16:20:40.653690] [PROGRESS] Training iteration 593 (batch 1 of epoch 148). [+][2018-08-15 16:20:40.682934] [PROGRESS] Training iteration 594 (batch 2 of epoch 148). [+][2018-08-15 16:20:40.711387] [PROGRESS] Training iteration 595 (batch 3 of epoch 148). [+][2018-08-15 16:20:40.739013] [PROGRESS] Training iteration 596 (batch 4 of epoch 148). [+][2018-08-15 16:20:40.806949] [PROGRESS] Training iteration 597 (batch 1 of epoch 149). [+][2018-08-15 16:20:40.834927] [PROGRESS] Training iteration 598 (batch 2 of epoch 149). [+][2018-08-15 16:20:40.862905] [PROGRESS] Training iteration 599 (batch 3 of epoch 149). [+][2018-08-15 16:20:40.890236] [PROGRESS] Training iteration 600 (batch 4 of epoch 149). [+][2018-08-15 16:20:40.959129] [PROGRESS] Training iteration 601 (batch 1 of epoch 150). [+][2018-08-15 16:20:40.987452] [PROGRESS] Training iteration 602 (batch 2 of epoch 150). [+][2018-08-15 16:20:41.014639] [PROGRESS] Training iteration 603 (batch 3 of epoch 150). [+][2018-08-15 16:20:41.043192] [PROGRESS] Training iteration 604 (batch 4 of epoch 150). [+][2018-08-15 16:20:41.114134] [PROGRESS] Training iteration 605 (batch 1 of epoch 151). [+][2018-08-15 16:20:41.142635] [PROGRESS] Training iteration 606 (batch 2 of epoch 151). [+][2018-08-15 16:20:41.170032] [PROGRESS] Training iteration 607 (batch 3 of epoch 151). [+][2018-08-15 16:20:41.197096] [PROGRESS] Training iteration 608 (batch 4 of epoch 151). [+][2018-08-15 16:20:41.266756] [PROGRESS] Training iteration 609 (batch 1 of epoch 152). [+][2018-08-15 16:20:41.294663] [PROGRESS] Training iteration 610 (batch 2 of epoch 152). [+][2018-08-15 16:20:41.322294] [PROGRESS] Training iteration 611 (batch 3 of epoch 152). [+][2018-08-15 16:20:41.349361] [PROGRESS] Training iteration 612 (batch 4 of epoch 152). [+][2018-08-15 16:20:41.419046] [PROGRESS] Training iteration 613 (batch 1 of epoch 153). [+][2018-08-15 16:20:41.446921] [PROGRESS] Training iteration 614 (batch 2 of epoch 153). [+][2018-08-15 16:20:41.474255] [PROGRESS] Training iteration 615 (batch 3 of epoch 153). [+][2018-08-15 16:20:41.501311] [PROGRESS] Training iteration 616 (batch 4 of epoch 153). [+][2018-08-15 16:20:41.577592] [PROGRESS] Training iteration 617 (batch 1 of epoch 154). [+][2018-08-15 16:20:41.606100] [PROGRESS] Training iteration 618 (batch 2 of epoch 154). [+][2018-08-15 16:20:41.634196] [PROGRESS] Training iteration 619 (batch 3 of epoch 154). [+][2018-08-15 16:20:41.661548] [PROGRESS] Training iteration 620 (batch 4 of epoch 154). [+][2018-08-15 16:20:41.731569] [PROGRESS] Training iteration 621 (batch 1 of epoch 155). [+][2018-08-15 16:20:41.759794] [PROGRESS] Training iteration 622 (batch 2 of epoch 155). [+][2018-08-15 16:20:41.787252] [PROGRESS] Training iteration 623 (batch 3 of epoch 155). [+][2018-08-15 16:20:41.814437] [PROGRESS] Training iteration 624 (batch 4 of epoch 155). [+][2018-08-15 16:20:41.884411] [PROGRESS] Training iteration 625 (batch 1 of epoch 156). [+][2018-08-15 16:20:41.912486] [PROGRESS] Training iteration 626 (batch 2 of epoch 156). [+][2018-08-15 16:20:41.939925] [PROGRESS] Training iteration 627 (batch 3 of epoch 156). [+][2018-08-15 16:20:41.967408] [PROGRESS] Training iteration 628 (batch 4 of epoch 156). [+][2018-08-15 16:20:42.036644] [PROGRESS] Training iteration 629 (batch 1 of epoch 157). [+][2018-08-15 16:20:42.065340] [PROGRESS] Training iteration 630 (batch 2 of epoch 157). [+][2018-08-15 16:20:42.092672] [PROGRESS] Training iteration 631 (batch 3 of epoch 157). [+][2018-08-15 16:20:42.119922] [PROGRESS] Training iteration 632 (batch 4 of epoch 157). [+][2018-08-15 16:20:42.189922] [PROGRESS] Training iteration 633 (batch 1 of epoch 158). [+][2018-08-15 16:20:42.218724] [PROGRESS] Training iteration 634 (batch 2 of epoch 158). [+][2018-08-15 16:20:42.245682] [PROGRESS] Training iteration 635 (batch 3 of epoch 158). [+][2018-08-15 16:20:42.272980] [PROGRESS] Training iteration 636 (batch 4 of epoch 158). [+][2018-08-15 16:20:42.341778] [PROGRESS] Training iteration 637 (batch 1 of epoch 159). [+][2018-08-15 16:20:42.370183] [PROGRESS] Training iteration 638 (batch 2 of epoch 159). [+][2018-08-15 16:20:42.398111] [PROGRESS] Training iteration 639 (batch 3 of epoch 159). [+][2018-08-15 16:20:42.425495] [PROGRESS] Training iteration 640 (batch 4 of epoch 159). [+][2018-08-15 16:20:42.494747] [PROGRESS] Training iteration 641 (batch 1 of epoch 160). [+][2018-08-15 16:20:42.523384] [PROGRESS] Training iteration 642 (batch 2 of epoch 160). [+][2018-08-15 16:20:42.551159] [PROGRESS] Training iteration 643 (batch 3 of epoch 160). [+][2018-08-15 16:20:42.578080] [PROGRESS] Training iteration 644 (batch 4 of epoch 160). [+][2018-08-15 16:20:42.648361] [PROGRESS] Training iteration 645 (batch 1 of epoch 161). [+][2018-08-15 16:20:42.677139] [PROGRESS] Training iteration 646 (batch 2 of epoch 161). [+][2018-08-15 16:20:42.704847] [PROGRESS] Training iteration 647 (batch 3 of epoch 161). [+][2018-08-15 16:20:42.731692] [PROGRESS] Training iteration 648 (batch 4 of epoch 161). [+][2018-08-15 16:20:42.801914] [PROGRESS] Training iteration 649 (batch 1 of epoch 162). [+][2018-08-15 16:20:42.830120] [PROGRESS] Training iteration 650 (batch 2 of epoch 162). [+][2018-08-15 16:20:42.857883] [PROGRESS] Training iteration 651 (batch 3 of epoch 162). [+][2018-08-15 16:20:42.885481] [PROGRESS] Training iteration 652 (batch 4 of epoch 162). [+][2018-08-15 16:20:42.955209] [PROGRESS] Training iteration 653 (batch 1 of epoch 163). [+][2018-08-15 16:20:42.983765] [PROGRESS] Training iteration 654 (batch 2 of epoch 163). [+][2018-08-15 16:20:43.011145] [PROGRESS] Training iteration 655 (batch 3 of epoch 163). [+][2018-08-15 16:20:43.038434] [PROGRESS] Training iteration 656 (batch 4 of epoch 163). [+][2018-08-15 16:20:43.107674] [PROGRESS] Training iteration 657 (batch 1 of epoch 164). [+][2018-08-15 16:20:43.136018] [PROGRESS] Training iteration 658 (batch 2 of epoch 164). [+][2018-08-15 16:20:43.163651] [PROGRESS] Training iteration 659 (batch 3 of epoch 164). [+][2018-08-15 16:20:43.191222] [PROGRESS] Training iteration 660 (batch 4 of epoch 164). [+][2018-08-15 16:20:43.260773] [PROGRESS] Training iteration 661 (batch 1 of epoch 165). [+][2018-08-15 16:20:43.288801] [PROGRESS] Training iteration 662 (batch 2 of epoch 165). [+][2018-08-15 16:20:43.316059] [PROGRESS] Training iteration 663 (batch 3 of epoch 165). [+][2018-08-15 16:20:43.343181] [PROGRESS] Training iteration 664 (batch 4 of epoch 165). [+][2018-08-15 16:20:43.414146] [PROGRESS] Training iteration 665 (batch 1 of epoch 166). [+][2018-08-15 16:20:43.441950] [PROGRESS] Training iteration 666 (batch 2 of epoch 166). [+][2018-08-15 16:20:43.468936] [PROGRESS] Training iteration 667 (batch 3 of epoch 166). [+][2018-08-15 16:20:43.496113] [PROGRESS] Training iteration 668 (batch 4 of epoch 166). [+][2018-08-15 16:20:43.564693] [PROGRESS] Training iteration 669 (batch 1 of epoch 167). [+][2018-08-15 16:20:43.593236] [PROGRESS] Training iteration 670 (batch 2 of epoch 167). [+][2018-08-15 16:20:43.620663] [PROGRESS] Training iteration 671 (batch 3 of epoch 167). [+][2018-08-15 16:20:43.647891] [PROGRESS] Training iteration 672 (batch 4 of epoch 167). [+][2018-08-15 16:20:43.716558] [PROGRESS] Training iteration 673 (batch 1 of epoch 168). [+][2018-08-15 16:20:43.744681] [PROGRESS] Training iteration 674 (batch 2 of epoch 168). [+][2018-08-15 16:20:43.771814] [PROGRESS] Training iteration 675 (batch 3 of epoch 168). [+][2018-08-15 16:20:43.799098] [PROGRESS] Training iteration 676 (batch 4 of epoch 168). [+][2018-08-15 16:20:43.868859] [PROGRESS] Training iteration 677 (batch 1 of epoch 169). [+][2018-08-15 16:20:43.897324] [PROGRESS] Training iteration 678 (batch 2 of epoch 169). [+][2018-08-15 16:20:43.924457] [PROGRESS] Training iteration 679 (batch 3 of epoch 169). [+][2018-08-15 16:20:43.951976] [PROGRESS] Training iteration 680 (batch 4 of epoch 169). [+][2018-08-15 16:20:44.022292] [PROGRESS] Training iteration 681 (batch 1 of epoch 170). [+][2018-08-15 16:20:44.050541] [PROGRESS] Training iteration 682 (batch 2 of epoch 170). [+][2018-08-15 16:20:44.077828] [PROGRESS] Training iteration 683 (batch 3 of epoch 170). [+][2018-08-15 16:20:44.105250] [PROGRESS] Training iteration 684 (batch 4 of epoch 170). [+][2018-08-15 16:20:44.175431] [PROGRESS] Training iteration 685 (batch 1 of epoch 171). [+][2018-08-15 16:20:44.203590] [PROGRESS] Training iteration 686 (batch 2 of epoch 171). [+][2018-08-15 16:20:44.231895] [PROGRESS] Training iteration 687 (batch 3 of epoch 171). [+][2018-08-15 16:20:44.259511] [PROGRESS] Training iteration 688 (batch 4 of epoch 171). [+][2018-08-15 16:20:44.329056] [PROGRESS] Training iteration 689 (batch 1 of epoch 172). [+][2018-08-15 16:20:44.357584] [PROGRESS] Training iteration 690 (batch 2 of epoch 172). [+][2018-08-15 16:20:44.385211] [PROGRESS] Training iteration 691 (batch 3 of epoch 172). [+][2018-08-15 16:20:44.411722] [PROGRESS] Training iteration 692 (batch 4 of epoch 172). [+][2018-08-15 16:20:44.482204] [PROGRESS] Training iteration 693 (batch 1 of epoch 173). [+][2018-08-15 16:20:44.510919] [PROGRESS] Training iteration 694 (batch 2 of epoch 173). [+][2018-08-15 16:20:44.538938] [PROGRESS] Training iteration 695 (batch 3 of epoch 173). [+][2018-08-15 16:20:44.565787] [PROGRESS] Training iteration 696 (batch 4 of epoch 173). [+][2018-08-15 16:20:44.634770] [PROGRESS] Training iteration 697 (batch 1 of epoch 174). [+][2018-08-15 16:20:44.663798] [PROGRESS] Training iteration 698 (batch 2 of epoch 174). [+][2018-08-15 16:20:44.691332] [PROGRESS] Training iteration 699 (batch 3 of epoch 174). [+][2018-08-15 16:20:44.719115] [PROGRESS] Training iteration 700 (batch 4 of epoch 174). [+][2018-08-15 16:20:44.789492] [PROGRESS] Training iteration 701 (batch 1 of epoch 175). [+][2018-08-15 16:20:44.817793] [PROGRESS] Training iteration 702 (batch 2 of epoch 175). [+][2018-08-15 16:20:44.845388] [PROGRESS] Training iteration 703 (batch 3 of epoch 175). [+][2018-08-15 16:20:44.873517] [PROGRESS] Training iteration 704 (batch 4 of epoch 175). [+][2018-08-15 16:20:44.943224] [PROGRESS] Training iteration 705 (batch 1 of epoch 176). [+][2018-08-15 16:20:44.971658] [PROGRESS] Training iteration 706 (batch 2 of epoch 176). [+][2018-08-15 16:20:44.999426] [PROGRESS] Training iteration 707 (batch 3 of epoch 176). [+][2018-08-15 16:20:45.026649] [PROGRESS] Training iteration 708 (batch 4 of epoch 176). [+][2018-08-15 16:20:45.095755] [PROGRESS] Training iteration 709 (batch 1 of epoch 177). [+][2018-08-15 16:20:45.123861] [PROGRESS] Training iteration 710 (batch 2 of epoch 177). [+][2018-08-15 16:20:45.151581] [PROGRESS] Training iteration 711 (batch 3 of epoch 177). [+][2018-08-15 16:20:45.178565] [PROGRESS] Training iteration 712 (batch 4 of epoch 177). [+][2018-08-15 16:20:45.248260] [PROGRESS] Training iteration 713 (batch 1 of epoch 178). [+][2018-08-15 16:20:45.276516] [PROGRESS] Training iteration 714 (batch 2 of epoch 178). [+][2018-08-15 16:20:45.304253] [PROGRESS] Training iteration 715 (batch 3 of epoch 178). [+][2018-08-15 16:20:45.331741] [PROGRESS] Training iteration 716 (batch 4 of epoch 178). [+][2018-08-15 16:20:45.400974] [PROGRESS] Training iteration 717 (batch 1 of epoch 179). [+][2018-08-15 16:20:45.428709] [PROGRESS] Training iteration 718 (batch 2 of epoch 179). [+][2018-08-15 16:20:45.456172] [PROGRESS] Training iteration 719 (batch 3 of epoch 179). [+][2018-08-15 16:20:45.483451] [PROGRESS] Training iteration 720 (batch 4 of epoch 179). [+][2018-08-15 16:20:45.552755] [PROGRESS] Training iteration 721 (batch 1 of epoch 180). [+][2018-08-15 16:20:45.580851] [PROGRESS] Training iteration 722 (batch 2 of epoch 180). [+][2018-08-15 16:20:45.608492] [PROGRESS] Training iteration 723 (batch 3 of epoch 180). [+][2018-08-15 16:20:45.634840] [PROGRESS] Training iteration 724 (batch 4 of epoch 180). [+][2018-08-15 16:20:45.704191] [PROGRESS] Training iteration 725 (batch 1 of epoch 181). [+][2018-08-15 16:20:45.732619] [PROGRESS] Training iteration 726 (batch 2 of epoch 181). [+][2018-08-15 16:20:45.761944] [PROGRESS] Training iteration 727 (batch 3 of epoch 181). [+][2018-08-15 16:20:45.793958] [PROGRESS] Training iteration 728 (batch 4 of epoch 181). [+][2018-08-15 16:20:45.869521] [PROGRESS] Training iteration 729 (batch 1 of epoch 182). [+][2018-08-15 16:20:45.903060] [PROGRESS] Training iteration 730 (batch 2 of epoch 182). [+][2018-08-15 16:20:45.936631] [PROGRESS] Training iteration 731 (batch 3 of epoch 182). [+][2018-08-15 16:20:45.966026] [PROGRESS] Training iteration 732 (batch 4 of epoch 182). [+][2018-08-15 16:20:46.047729] [PROGRESS] Training iteration 733 (batch 1 of epoch 183). [+][2018-08-15 16:20:46.079144] [PROGRESS] Training iteration 734 (batch 2 of epoch 183). [+][2018-08-15 16:20:46.112181] [PROGRESS] Training iteration 735 (batch 3 of epoch 183). [+][2018-08-15 16:20:46.144003] [PROGRESS] Training iteration 736 (batch 4 of epoch 183). [+][2018-08-15 16:20:46.219288] [PROGRESS] Training iteration 737 (batch 1 of epoch 184). [+][2018-08-15 16:20:46.251913] [PROGRESS] Training iteration 738 (batch 2 of epoch 184). [+][2018-08-15 16:20:46.281492] [PROGRESS] Training iteration 739 (batch 3 of epoch 184). [+][2018-08-15 16:20:46.309024] [PROGRESS] Training iteration 740 (batch 4 of epoch 184). [+][2018-08-15 16:20:46.380041] [PROGRESS] Training iteration 741 (batch 1 of epoch 185). [+][2018-08-15 16:20:46.408123] [PROGRESS] Training iteration 742 (batch 2 of epoch 185). [+][2018-08-15 16:20:46.436927] [PROGRESS] Training iteration 743 (batch 3 of epoch 185). [+][2018-08-15 16:20:46.464559] [PROGRESS] Training iteration 744 (batch 4 of epoch 185). [+][2018-08-15 16:20:46.534986] [PROGRESS] Training iteration 745 (batch 1 of epoch 186). [+][2018-08-15 16:20:46.564084] [PROGRESS] Training iteration 746 (batch 2 of epoch 186). [+][2018-08-15 16:20:46.593194] [PROGRESS] Training iteration 747 (batch 3 of epoch 186). [+][2018-08-15 16:20:46.620812] [PROGRESS] Training iteration 748 (batch 4 of epoch 186). [+][2018-08-15 16:20:46.689798] [PROGRESS] Training iteration 749 (batch 1 of epoch 187). [+][2018-08-15 16:20:46.718613] [PROGRESS] Training iteration 750 (batch 2 of epoch 187). [+][2018-08-15 16:20:46.746737] [PROGRESS] Training iteration 751 (batch 3 of epoch 187). [+][2018-08-15 16:20:46.774457] [PROGRESS] Training iteration 752 (batch 4 of epoch 187). [+][2018-08-15 16:20:46.845104] [PROGRESS] Training iteration 753 (batch 1 of epoch 188). [+][2018-08-15 16:20:46.873551] [PROGRESS] Training iteration 754 (batch 2 of epoch 188). [+][2018-08-15 16:20:46.901690] [PROGRESS] Training iteration 755 (batch 3 of epoch 188). [+][2018-08-15 16:20:46.929559] [PROGRESS] Training iteration 756 (batch 4 of epoch 188). [+][2018-08-15 16:20:46.999185] [PROGRESS] Training iteration 757 (batch 1 of epoch 189). [+][2018-08-15 16:20:47.026687] [PROGRESS] Training iteration 758 (batch 2 of epoch 189). [+][2018-08-15 16:20:47.055175] [PROGRESS] Training iteration 759 (batch 3 of epoch 189). [+][2018-08-15 16:20:47.082586] [PROGRESS] Training iteration 760 (batch 4 of epoch 189). [+][2018-08-15 16:20:47.159431] [PROGRESS] Training iteration 761 (batch 1 of epoch 190). [+][2018-08-15 16:20:47.190229] [PROGRESS] Training iteration 762 (batch 2 of epoch 190). [+][2018-08-15 16:20:47.220644] [PROGRESS] Training iteration 763 (batch 3 of epoch 190). [+][2018-08-15 16:20:47.250682] [PROGRESS] Training iteration 764 (batch 4 of epoch 190). [+][2018-08-15 16:20:47.320979] [PROGRESS] Training iteration 765 (batch 1 of epoch 191). [+][2018-08-15 16:20:47.348885] [PROGRESS] Training iteration 766 (batch 2 of epoch 191). [+][2018-08-15 16:20:47.376565] [PROGRESS] Training iteration 767 (batch 3 of epoch 191). [+][2018-08-15 16:20:47.404379] [PROGRESS] Training iteration 768 (batch 4 of epoch 191). [+][2018-08-15 16:20:47.477098] [PROGRESS] Training iteration 769 (batch 1 of epoch 192). [+][2018-08-15 16:20:47.505290] [PROGRESS] Training iteration 770 (batch 2 of epoch 192). [+][2018-08-15 16:20:47.533160] [PROGRESS] Training iteration 771 (batch 3 of epoch 192). [+][2018-08-15 16:20:47.562609] [PROGRESS] Training iteration 772 (batch 4 of epoch 192). [+][2018-08-15 16:20:47.635293] [PROGRESS] Training iteration 773 (batch 1 of epoch 193). [+][2018-08-15 16:20:47.665403] [PROGRESS] Training iteration 774 (batch 2 of epoch 193). [+][2018-08-15 16:20:47.695337] [PROGRESS] Training iteration 775 (batch 3 of epoch 193). [+][2018-08-15 16:20:47.724703] [PROGRESS] Training iteration 776 (batch 4 of epoch 193). [+][2018-08-15 16:20:47.795238] [PROGRESS] Training iteration 777 (batch 1 of epoch 194). [+][2018-08-15 16:20:47.825635] [PROGRESS] Training iteration 778 (batch 2 of epoch 194). [+][2018-08-15 16:20:47.854191] [PROGRESS] Training iteration 779 (batch 3 of epoch 194). [+][2018-08-15 16:20:47.882019] [PROGRESS] Training iteration 780 (batch 4 of epoch 194). [+][2018-08-15 16:20:47.952214] [PROGRESS] Training iteration 781 (batch 1 of epoch 195). [+][2018-08-15 16:20:47.980424] [PROGRESS] Training iteration 782 (batch 2 of epoch 195). [+][2018-08-15 16:20:48.008629] [PROGRESS] Training iteration 783 (batch 3 of epoch 195). [+][2018-08-15 16:20:48.036422] [PROGRESS] Training iteration 784 (batch 4 of epoch 195). [+][2018-08-15 16:20:48.118639] [PROGRESS] Training iteration 785 (batch 1 of epoch 196). [+][2018-08-15 16:20:48.150281] [PROGRESS] Training iteration 786 (batch 2 of epoch 196). [+][2018-08-15 16:20:48.183789] [PROGRESS] Training iteration 787 (batch 3 of epoch 196). [+][2018-08-15 16:20:48.216647] [PROGRESS] Training iteration 788 (batch 4 of epoch 196). [+][2018-08-15 16:20:48.293191] [PROGRESS] Training iteration 789 (batch 1 of epoch 197). [+][2018-08-15 16:20:48.321250] [PROGRESS] Training iteration 790 (batch 2 of epoch 197). [+][2018-08-15 16:20:48.349303] [PROGRESS] Training iteration 791 (batch 3 of epoch 197). [+][2018-08-15 16:20:48.377194] [PROGRESS] Training iteration 792 (batch 4 of epoch 197). [+][2018-08-15 16:20:48.448702] [PROGRESS] Training iteration 793 (batch 1 of epoch 198). [+][2018-08-15 16:20:48.476849] [PROGRESS] Training iteration 794 (batch 2 of epoch 198). [+][2018-08-15 16:20:48.505045] [PROGRESS] Training iteration 795 (batch 3 of epoch 198). [+][2018-08-15 16:20:48.532500] [PROGRESS] Training iteration 796 (batch 4 of epoch 198). [+][2018-08-15 16:20:48.602917] [PROGRESS] Training iteration 797 (batch 1 of epoch 199). [+][2018-08-15 16:20:48.631456] [PROGRESS] Training iteration 798 (batch 2 of epoch 199). [+][2018-08-15 16:20:48.659745] [PROGRESS] Training iteration 799 (batch 3 of epoch 199). [+][2018-08-15 16:20:48.687617] [PROGRESS] Training iteration 800 (batch 4 of epoch 199). [+][2018-08-15 16:20:48.757873] [INFO ] Breaking on request from callback. [+][2018-08-15 16:20:48.758211] [INFO ] Exceeded max number of iterations / epochs, breaking. Prediction ---------------------------- The trainer contains the trained model and we can do predictions. We use :code:`unwrap` to convert the results to numpy arrays. Since we want to do many prediction we encapsulate the the prediction in a function .. code-block:: python from inferno.utils.torch_utils import unwrap def predict(trainer, test_loader, save_dir=None): trainer.eval_mode() for image, target in test_loader: # transfer image to gpu image = image.cuda() if USE_CUDA else image # get batch size from image batch_size = image.size()[0] for b in range(batch_size): prediction = trainer.apply_model(image) prediction = torch.nn.functional.sigmoid(prediction) image = unwrap(image, as_numpy=True, to_cpu=True) prediction = unwrap(prediction, as_numpy=True, to_cpu=True) target = unwrap(target, as_numpy=True, to_cpu=True) fig = plt.figure() ax = fig.add_subplot(2, 2, 1) ax.imshow(image[b,0,...]) ax.set_title('raw data') ax = fig.add_subplot(2, 2, 2) ax.imshow(target[b,...]) ax.set_title('ground truth') ax = fig.add_subplot(2, 2, 4) ax.imshow(prediction[b,...]) ax.set_title('prediction') fig.tight_layout() plt.show() do the prediction .. code-block:: python predict(trainer=trainer, test_loader=test_loader) .. rst-class:: sphx-glr-horizontal * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_002.png :class: sphx-glr-multi-img * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_003.png :class: sphx-glr-multi-img * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_004.png :class: sphx-glr-multi-img * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_005.png :class: sphx-glr-multi-img * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_006.png :class: sphx-glr-multi-img * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_007.png :class: sphx-glr-multi-img * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_008.png :class: sphx-glr-multi-img * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_009.png :class: sphx-glr-multi-img Custom UNet ---------------------------- Often one needs to have a UNet with custom layers. Here we show how to implement such a customized UNet. To this end we derive from :code:`UNetBase`. For the sake of this example we will create a rather exotic UNet which uses different types of convolutions/non-linearities in the different branches of the unet .. code-block:: python from inferno.extensions.layers import UNetBase from inferno.extensions.layers import ConvSELU2D, ConvReLU2D, ConvELU2D, ConvSigmoid2D,Conv2D class MySimple2DUnet(UNetBase): def __init__(self, in_channels, out_channels, depth=3, **kwargs): super(MySimple2DUnet, self).__init__(in_channels=in_channels, out_channels=out_channels, dim=2, depth=depth, **kwargs) def conv_op_factory(self, in_channels, out_channels, part, index): if part == 'down': return torch.nn.Sequential( ConvELU2D(in_channels=in_channels, out_channels=out_channels, kernel_size=3), ConvELU2D(in_channels=out_channels, out_channels=out_channels, kernel_size=3) ), False elif part == 'bottom': return torch.nn.Sequential( ConvReLU2D(in_channels=in_channels, out_channels=out_channels, kernel_size=3), ConvReLU2D(in_channels=out_channels, out_channels=out_channels, kernel_size=3), ), False elif part == 'up': # are we in the very last block? if index + 1 == self.depth: return torch.nn.Sequential( ConvELU2D(in_channels=in_channels, out_channels=out_channels, kernel_size=3), Conv2D(in_channels=out_channels, out_channels=out_channels, kernel_size=3) ), False else: return torch.nn.Sequential( ConvELU2D(in_channels=in_channels, out_channels=out_channels, kernel_size=3), ConvReLU2D(in_channels=out_channels, out_channels=out_channels, kernel_size=3) ), False else: raise RuntimeError("something is wrong") # this function CAN be implemented, if not, MaxPooling is used by default def downsample_op_factory(self, index): return torch.nn.MaxPool2d(kernel_size=2, stride=2) # this function CAN be implemented, if not, Upsampling is used by default def upsample_op_factory(self, index): return torch.nn.Upsample(mode='bilinear', align_corners=False,scale_factor=2) model_b = torch.nn.Sequential( ConvReLU2D(in_channels=image_channels, out_channels=5, kernel_size=3), MySimple2DUnet(in_channels=5, out_channels=pred_channels) , RemoveSingletonDimension(dim=1) ) do the training (with the same functions as before) .. code-block:: python trainer = train_model(model=model_b, loaders=[train_loader, validate_loader], save_dir='model_b', lr=0.001) .. rst-class:: sphx-glr-script-out Out: .. code-block:: none [+][2018-08-15 16:20:49.968379] [PROGRESS] Training iteration 0 (batch 0 of epoch 0). [+][2018-08-15 16:20:50.034558] [PROGRESS] Training iteration 1 (batch 1 of epoch 0). [+][2018-08-15 16:20:50.059021] [PROGRESS] Training iteration 2 (batch 2 of epoch 0). [+][2018-08-15 16:20:50.081431] [PROGRESS] Training iteration 3 (batch 3 of epoch 0). [+][2018-08-15 16:20:50.101687] [PROGRESS] Training iteration 4 (batch 4 of epoch 0). [+][2018-08-15 16:20:50.170383] [PROGRESS] Training iteration 5 (batch 1 of epoch 1). [+][2018-08-15 16:20:50.192796] [PROGRESS] Training iteration 6 (batch 2 of epoch 1). [+][2018-08-15 16:20:50.212532] [PROGRESS] Training iteration 7 (batch 3 of epoch 1). [+][2018-08-15 16:20:50.233634] [PROGRESS] Training iteration 8 (batch 4 of epoch 1). [+][2018-08-15 16:20:50.294644] [PROGRESS] Training iteration 9 (batch 1 of epoch 2). [+][2018-08-15 16:20:50.315931] [PROGRESS] Training iteration 10 (batch 2 of epoch 2). [+][2018-08-15 16:20:50.339907] [PROGRESS] Training iteration 11 (batch 3 of epoch 2). [+][2018-08-15 16:20:50.369116] [PROGRESS] Training iteration 12 (batch 4 of epoch 2). [+][2018-08-15 16:20:50.434182] [PROGRESS] Training iteration 13 (batch 1 of epoch 3). [+][2018-08-15 16:20:50.459577] [PROGRESS] Training iteration 14 (batch 2 of epoch 3). [+][2018-08-15 16:20:50.482720] [PROGRESS] Training iteration 15 (batch 3 of epoch 3). [+][2018-08-15 16:20:50.508764] [PROGRESS] Training iteration 16 (batch 4 of epoch 3). [+][2018-08-15 16:20:50.572524] [PROGRESS] Training iteration 17 (batch 1 of epoch 4). [+][2018-08-15 16:20:50.593954] [PROGRESS] Training iteration 18 (batch 2 of epoch 4). [+][2018-08-15 16:20:50.616249] [PROGRESS] Training iteration 19 (batch 3 of epoch 4). [+][2018-08-15 16:20:50.639736] [PROGRESS] Training iteration 20 (batch 4 of epoch 4). [+][2018-08-15 16:20:50.703206] [PROGRESS] Training iteration 21 (batch 1 of epoch 5). [+][2018-08-15 16:20:50.722341] [PROGRESS] Training iteration 22 (batch 2 of epoch 5). [+][2018-08-15 16:20:50.745861] [PROGRESS] Training iteration 23 (batch 3 of epoch 5). [+][2018-08-15 16:20:50.769990] [PROGRESS] Training iteration 24 (batch 4 of epoch 5). [+][2018-08-15 16:20:50.832153] [PROGRESS] Training iteration 25 (batch 1 of epoch 6). [+][2018-08-15 16:20:50.853527] [PROGRESS] Training iteration 26 (batch 2 of epoch 6). [+][2018-08-15 16:20:50.876346] [PROGRESS] Training iteration 27 (batch 3 of epoch 6). [+][2018-08-15 16:20:50.899524] [PROGRESS] Training iteration 28 (batch 4 of epoch 6). [+][2018-08-15 16:20:50.959807] [PROGRESS] Training iteration 29 (batch 1 of epoch 7). [+][2018-08-15 16:20:50.979861] [PROGRESS] Training iteration 30 (batch 2 of epoch 7). [+][2018-08-15 16:20:51.000656] [PROGRESS] Training iteration 31 (batch 3 of epoch 7). [+][2018-08-15 16:20:51.022400] [PROGRESS] Training iteration 32 (batch 4 of epoch 7). [+][2018-08-15 16:20:51.084468] [PROGRESS] Training iteration 33 (batch 1 of epoch 8). [+][2018-08-15 16:20:51.106066] [PROGRESS] Training iteration 34 (batch 2 of epoch 8). [+][2018-08-15 16:20:51.125886] [PROGRESS] Training iteration 35 (batch 3 of epoch 8). [+][2018-08-15 16:20:51.147783] [PROGRESS] Training iteration 36 (batch 4 of epoch 8). [+][2018-08-15 16:20:51.216719] [PROGRESS] Training iteration 37 (batch 1 of epoch 9). [+][2018-08-15 16:20:51.239534] [PROGRESS] Training iteration 38 (batch 2 of epoch 9). [+][2018-08-15 16:20:51.261890] [PROGRESS] Training iteration 39 (batch 3 of epoch 9). [+][2018-08-15 16:20:51.285273] [PROGRESS] Training iteration 40 (batch 4 of epoch 9). [+][2018-08-15 16:20:51.355959] [PROGRESS] Training iteration 41 (batch 1 of epoch 10). [+][2018-08-15 16:20:51.377582] [PROGRESS] Training iteration 42 (batch 2 of epoch 10). [+][2018-08-15 16:20:51.403237] [PROGRESS] Training iteration 43 (batch 3 of epoch 10). [+][2018-08-15 16:20:51.427984] [PROGRESS] Training iteration 44 (batch 4 of epoch 10). [+][2018-08-15 16:20:51.489835] [PROGRESS] Training iteration 45 (batch 1 of epoch 11). [+][2018-08-15 16:20:51.512234] [PROGRESS] Training iteration 46 (batch 2 of epoch 11). [+][2018-08-15 16:20:51.536260] [PROGRESS] Training iteration 47 (batch 3 of epoch 11). [+][2018-08-15 16:20:51.560487] [PROGRESS] Training iteration 48 (batch 4 of epoch 11). [+][2018-08-15 16:20:51.621565] [PROGRESS] Training iteration 49 (batch 1 of epoch 12). [+][2018-08-15 16:20:51.643795] [PROGRESS] Training iteration 50 (batch 2 of epoch 12). [+][2018-08-15 16:20:51.668061] [PROGRESS] Training iteration 51 (batch 3 of epoch 12). [+][2018-08-15 16:20:51.693036] [PROGRESS] Training iteration 52 (batch 4 of epoch 12). [+][2018-08-15 16:20:51.759140] [PROGRESS] Training iteration 53 (batch 1 of epoch 13). [+][2018-08-15 16:20:51.780344] [PROGRESS] Training iteration 54 (batch 2 of epoch 13). [+][2018-08-15 16:20:51.803883] [PROGRESS] Training iteration 55 (batch 3 of epoch 13). [+][2018-08-15 16:20:51.827087] [PROGRESS] Training iteration 56 (batch 4 of epoch 13). [+][2018-08-15 16:20:51.888328] [PROGRESS] Training iteration 57 (batch 1 of epoch 14). [+][2018-08-15 16:20:51.909006] [PROGRESS] Training iteration 58 (batch 2 of epoch 14). [+][2018-08-15 16:20:51.930568] [PROGRESS] Training iteration 59 (batch 3 of epoch 14). [+][2018-08-15 16:20:51.953046] [PROGRESS] Training iteration 60 (batch 4 of epoch 14). [+][2018-08-15 16:20:52.013922] [PROGRESS] Training iteration 61 (batch 1 of epoch 15). [+][2018-08-15 16:20:52.034167] [PROGRESS] Training iteration 62 (batch 2 of epoch 15). [+][2018-08-15 16:20:52.055069] [PROGRESS] Training iteration 63 (batch 3 of epoch 15). [+][2018-08-15 16:20:52.076244] [PROGRESS] Training iteration 64 (batch 4 of epoch 15). [+][2018-08-15 16:20:52.137247] [PROGRESS] Training iteration 65 (batch 1 of epoch 16). [+][2018-08-15 16:20:52.157628] [PROGRESS] Training iteration 66 (batch 2 of epoch 16). [+][2018-08-15 16:20:52.178406] [PROGRESS] Training iteration 67 (batch 3 of epoch 16). [+][2018-08-15 16:20:52.198935] [PROGRESS] Training iteration 68 (batch 4 of epoch 16). [+][2018-08-15 16:20:52.260235] [PROGRESS] Training iteration 69 (batch 1 of epoch 17). [+][2018-08-15 16:20:52.281013] [PROGRESS] Training iteration 70 (batch 2 of epoch 17). [+][2018-08-15 16:20:52.301574] [PROGRESS] Training iteration 71 (batch 3 of epoch 17). [+][2018-08-15 16:20:52.322253] [PROGRESS] Training iteration 72 (batch 4 of epoch 17). [+][2018-08-15 16:20:52.383745] [PROGRESS] Training iteration 73 (batch 1 of epoch 18). [+][2018-08-15 16:20:52.403719] [PROGRESS] Training iteration 74 (batch 2 of epoch 18). [+][2018-08-15 16:20:52.425650] [PROGRESS] Training iteration 75 (batch 3 of epoch 18). [+][2018-08-15 16:20:52.447070] [PROGRESS] Training iteration 76 (batch 4 of epoch 18). [+][2018-08-15 16:20:52.509978] [PROGRESS] Training iteration 77 (batch 1 of epoch 19). [+][2018-08-15 16:20:52.529731] [PROGRESS] Training iteration 78 (batch 2 of epoch 19). [+][2018-08-15 16:20:52.549863] [PROGRESS] Training iteration 79 (batch 3 of epoch 19). [+][2018-08-15 16:20:52.570338] [PROGRESS] Training iteration 80 (batch 4 of epoch 19). [+][2018-08-15 16:20:52.633290] [PROGRESS] Training iteration 81 (batch 1 of epoch 20). [+][2018-08-15 16:20:52.653292] [PROGRESS] Training iteration 82 (batch 2 of epoch 20). [+][2018-08-15 16:20:52.673998] [PROGRESS] Training iteration 83 (batch 3 of epoch 20). [+][2018-08-15 16:20:52.695269] [PROGRESS] Training iteration 84 (batch 4 of epoch 20). [+][2018-08-15 16:20:52.758158] [PROGRESS] Training iteration 85 (batch 1 of epoch 21). [+][2018-08-15 16:20:52.779350] [PROGRESS] Training iteration 86 (batch 2 of epoch 21). [+][2018-08-15 16:20:52.799686] [PROGRESS] Training iteration 87 (batch 3 of epoch 21). [+][2018-08-15 16:20:52.820556] [PROGRESS] Training iteration 88 (batch 4 of epoch 21). [+][2018-08-15 16:20:52.881953] [PROGRESS] Training iteration 89 (batch 1 of epoch 22). [+][2018-08-15 16:20:52.902030] [PROGRESS] Training iteration 90 (batch 2 of epoch 22). [+][2018-08-15 16:20:52.923358] [PROGRESS] Training iteration 91 (batch 3 of epoch 22). [+][2018-08-15 16:20:52.943473] [PROGRESS] Training iteration 92 (batch 4 of epoch 22). [+][2018-08-15 16:20:53.005819] [PROGRESS] Training iteration 93 (batch 1 of epoch 23). [+][2018-08-15 16:20:53.026188] [PROGRESS] Training iteration 94 (batch 2 of epoch 23). [+][2018-08-15 16:20:53.047071] [PROGRESS] Training iteration 95 (batch 3 of epoch 23). [+][2018-08-15 16:20:53.068610] [PROGRESS] Training iteration 96 (batch 4 of epoch 23). [+][2018-08-15 16:20:53.131327] [PROGRESS] Training iteration 97 (batch 1 of epoch 24). [+][2018-08-15 16:20:53.153242] [PROGRESS] Training iteration 98 (batch 2 of epoch 24). [+][2018-08-15 16:20:53.172414] [PROGRESS] Training iteration 99 (batch 3 of epoch 24). [+][2018-08-15 16:20:53.193925] [PROGRESS] Training iteration 100 (batch 4 of epoch 24). [+][2018-08-15 16:20:53.256167] [PROGRESS] Training iteration 101 (batch 1 of epoch 25). [+][2018-08-15 16:20:53.275793] [PROGRESS] Training iteration 102 (batch 2 of epoch 25). [+][2018-08-15 16:20:53.297487] [PROGRESS] Training iteration 103 (batch 3 of epoch 25). [+][2018-08-15 16:20:53.319014] [PROGRESS] Training iteration 104 (batch 4 of epoch 25). [+][2018-08-15 16:20:53.382768] [PROGRESS] Training iteration 105 (batch 1 of epoch 26). [+][2018-08-15 16:20:53.402584] [PROGRESS] Training iteration 106 (batch 2 of epoch 26). [+][2018-08-15 16:20:53.423494] [PROGRESS] Training iteration 107 (batch 3 of epoch 26). [+][2018-08-15 16:20:53.445291] [PROGRESS] Training iteration 108 (batch 4 of epoch 26). [+][2018-08-15 16:20:53.508569] [PROGRESS] Training iteration 109 (batch 1 of epoch 27). [+][2018-08-15 16:20:53.528828] [PROGRESS] Training iteration 110 (batch 2 of epoch 27). [+][2018-08-15 16:20:53.551409] [PROGRESS] Training iteration 111 (batch 3 of epoch 27). [+][2018-08-15 16:20:53.574021] [PROGRESS] Training iteration 112 (batch 4 of epoch 27). [+][2018-08-15 16:20:53.636009] [PROGRESS] Training iteration 113 (batch 1 of epoch 28). [+][2018-08-15 16:20:53.656367] [PROGRESS] Training iteration 114 (batch 2 of epoch 28). [+][2018-08-15 16:20:53.676311] [PROGRESS] Training iteration 115 (batch 3 of epoch 28). [+][2018-08-15 16:20:53.697333] [PROGRESS] Training iteration 116 (batch 4 of epoch 28). [+][2018-08-15 16:20:53.757681] [PROGRESS] Training iteration 117 (batch 1 of epoch 29). [+][2018-08-15 16:20:53.778063] [PROGRESS] Training iteration 118 (batch 2 of epoch 29). [+][2018-08-15 16:20:53.800711] [PROGRESS] Training iteration 119 (batch 3 of epoch 29). [+][2018-08-15 16:20:53.821790] [PROGRESS] Training iteration 120 (batch 4 of epoch 29). [+][2018-08-15 16:20:53.882985] [PROGRESS] Training iteration 121 (batch 1 of epoch 30). [+][2018-08-15 16:20:53.903136] [PROGRESS] Training iteration 122 (batch 2 of epoch 30). [+][2018-08-15 16:20:53.923683] [PROGRESS] Training iteration 123 (batch 3 of epoch 30). [+][2018-08-15 16:20:53.944198] [PROGRESS] Training iteration 124 (batch 4 of epoch 30). [+][2018-08-15 16:20:54.006387] [PROGRESS] Training iteration 125 (batch 1 of epoch 31). [+][2018-08-15 16:20:54.025708] [PROGRESS] Training iteration 126 (batch 2 of epoch 31). [+][2018-08-15 16:20:54.047926] [PROGRESS] Training iteration 127 (batch 3 of epoch 31). [+][2018-08-15 16:20:54.071962] [PROGRESS] Training iteration 128 (batch 4 of epoch 31). [+][2018-08-15 16:20:54.135585] [PROGRESS] Training iteration 129 (batch 1 of epoch 32). [+][2018-08-15 16:20:54.158460] [PROGRESS] Training iteration 130 (batch 2 of epoch 32). [+][2018-08-15 16:20:54.181530] [PROGRESS] Training iteration 131 (batch 3 of epoch 32). [+][2018-08-15 16:20:54.205108] [PROGRESS] Training iteration 132 (batch 4 of epoch 32). [+][2018-08-15 16:20:54.269743] [PROGRESS] Training iteration 133 (batch 1 of epoch 33). [+][2018-08-15 16:20:54.294106] [PROGRESS] Training iteration 134 (batch 2 of epoch 33). [+][2018-08-15 16:20:54.315922] [PROGRESS] Training iteration 135 (batch 3 of epoch 33). [+][2018-08-15 16:20:54.338768] [PROGRESS] Training iteration 136 (batch 4 of epoch 33). [+][2018-08-15 16:20:54.399556] [PROGRESS] Training iteration 137 (batch 1 of epoch 34). [+][2018-08-15 16:20:54.419308] [PROGRESS] Training iteration 138 (batch 2 of epoch 34). [+][2018-08-15 16:20:54.440190] [PROGRESS] Training iteration 139 (batch 3 of epoch 34). [+][2018-08-15 16:20:54.461310] [PROGRESS] Training iteration 140 (batch 4 of epoch 34). [+][2018-08-15 16:20:54.521765] [PROGRESS] Training iteration 141 (batch 1 of epoch 35). [+][2018-08-15 16:20:54.543519] [PROGRESS] Training iteration 142 (batch 2 of epoch 35). [+][2018-08-15 16:20:54.566836] [PROGRESS] Training iteration 143 (batch 3 of epoch 35). [+][2018-08-15 16:20:54.590461] [PROGRESS] Training iteration 144 (batch 4 of epoch 35). [+][2018-08-15 16:20:54.653862] [PROGRESS] Training iteration 145 (batch 1 of epoch 36). [+][2018-08-15 16:20:54.677909] [PROGRESS] Training iteration 146 (batch 2 of epoch 36). [+][2018-08-15 16:20:54.697586] [PROGRESS] Training iteration 147 (batch 3 of epoch 36). [+][2018-08-15 16:20:54.718781] [PROGRESS] Training iteration 148 (batch 4 of epoch 36). [+][2018-08-15 16:20:54.783295] [PROGRESS] Training iteration 149 (batch 1 of epoch 37). [+][2018-08-15 16:20:54.805059] [PROGRESS] Training iteration 150 (batch 2 of epoch 37). [+][2018-08-15 16:20:54.826556] [PROGRESS] Training iteration 151 (batch 3 of epoch 37). [+][2018-08-15 16:20:54.845017] [PROGRESS] Training iteration 152 (batch 4 of epoch 37). [+][2018-08-15 16:20:54.907623] [PROGRESS] Training iteration 153 (batch 1 of epoch 38). [+][2018-08-15 16:20:54.927538] [PROGRESS] Training iteration 154 (batch 2 of epoch 38). [+][2018-08-15 16:20:54.947923] [PROGRESS] Training iteration 155 (batch 3 of epoch 38). [+][2018-08-15 16:20:54.969114] [PROGRESS] Training iteration 156 (batch 4 of epoch 38). [+][2018-08-15 16:20:55.031749] [PROGRESS] Training iteration 157 (batch 1 of epoch 39). [+][2018-08-15 16:20:55.052049] [PROGRESS] Training iteration 158 (batch 2 of epoch 39). [+][2018-08-15 16:20:55.072853] [PROGRESS] Training iteration 159 (batch 3 of epoch 39). [+][2018-08-15 16:20:55.093511] [PROGRESS] Training iteration 160 (batch 4 of epoch 39). [+][2018-08-15 16:20:55.162498] [PROGRESS] Training iteration 161 (batch 1 of epoch 40). [+][2018-08-15 16:20:55.183075] [PROGRESS] Training iteration 162 (batch 2 of epoch 40). [+][2018-08-15 16:20:55.205742] [PROGRESS] Training iteration 163 (batch 3 of epoch 40). [+][2018-08-15 16:20:55.224172] [PROGRESS] Training iteration 164 (batch 4 of epoch 40). [+][2018-08-15 16:20:55.284633] [PROGRESS] Training iteration 165 (batch 1 of epoch 41). [+][2018-08-15 16:20:55.304236] [PROGRESS] Training iteration 166 (batch 2 of epoch 41). [+][2018-08-15 16:20:55.325375] [PROGRESS] Training iteration 167 (batch 3 of epoch 41). [+][2018-08-15 16:20:55.346967] [PROGRESS] Training iteration 168 (batch 4 of epoch 41). [+][2018-08-15 16:20:55.411860] [PROGRESS] Training iteration 169 (batch 1 of epoch 42). [+][2018-08-15 16:20:55.437335] [PROGRESS] Training iteration 170 (batch 2 of epoch 42). [+][2018-08-15 16:20:55.458405] [PROGRESS] Training iteration 171 (batch 3 of epoch 42). [+][2018-08-15 16:20:55.480813] [PROGRESS] Training iteration 172 (batch 4 of epoch 42). [+][2018-08-15 16:20:55.542130] [PROGRESS] Training iteration 173 (batch 1 of epoch 43). [+][2018-08-15 16:20:55.562824] [PROGRESS] Training iteration 174 (batch 2 of epoch 43). [+][2018-08-15 16:20:55.584333] [PROGRESS] Training iteration 175 (batch 3 of epoch 43). [+][2018-08-15 16:20:55.604838] [PROGRESS] Training iteration 176 (batch 4 of epoch 43). [+][2018-08-15 16:20:55.665459] [PROGRESS] Training iteration 177 (batch 1 of epoch 44). [+][2018-08-15 16:20:55.685619] [PROGRESS] Training iteration 178 (batch 2 of epoch 44). [+][2018-08-15 16:20:55.707790] [PROGRESS] Training iteration 179 (batch 3 of epoch 44). [+][2018-08-15 16:20:55.730979] [PROGRESS] Training iteration 180 (batch 4 of epoch 44). [+][2018-08-15 16:20:55.792524] [PROGRESS] Training iteration 181 (batch 1 of epoch 45). [+][2018-08-15 16:20:55.813820] [PROGRESS] Training iteration 182 (batch 2 of epoch 45). [+][2018-08-15 16:20:55.836659] [PROGRESS] Training iteration 183 (batch 3 of epoch 45). [+][2018-08-15 16:20:55.859183] [PROGRESS] Training iteration 184 (batch 4 of epoch 45). [+][2018-08-15 16:20:55.920411] [PROGRESS] Training iteration 185 (batch 1 of epoch 46). [+][2018-08-15 16:20:55.941399] [PROGRESS] Training iteration 186 (batch 2 of epoch 46). [+][2018-08-15 16:20:55.964075] [PROGRESS] Training iteration 187 (batch 3 of epoch 46). [+][2018-08-15 16:20:55.986209] [PROGRESS] Training iteration 188 (batch 4 of epoch 46). [+][2018-08-15 16:20:56.053492] [PROGRESS] Training iteration 189 (batch 1 of epoch 47). [+][2018-08-15 16:20:56.074644] [PROGRESS] Training iteration 190 (batch 2 of epoch 47). [+][2018-08-15 16:20:56.098846] [PROGRESS] Training iteration 191 (batch 3 of epoch 47). [+][2018-08-15 16:20:56.121805] [PROGRESS] Training iteration 192 (batch 4 of epoch 47). [+][2018-08-15 16:20:56.183576] [PROGRESS] Training iteration 193 (batch 1 of epoch 48). [+][2018-08-15 16:20:56.204840] [PROGRESS] Training iteration 194 (batch 2 of epoch 48). [+][2018-08-15 16:20:56.228108] [PROGRESS] Training iteration 195 (batch 3 of epoch 48). [+][2018-08-15 16:20:56.251243] [PROGRESS] Training iteration 196 (batch 4 of epoch 48). [+][2018-08-15 16:20:56.313397] [PROGRESS] Training iteration 197 (batch 1 of epoch 49). [+][2018-08-15 16:20:56.334193] [PROGRESS] Training iteration 198 (batch 2 of epoch 49). [+][2018-08-15 16:20:56.355542] [PROGRESS] Training iteration 199 (batch 3 of epoch 49). [+][2018-08-15 16:20:56.377470] [PROGRESS] Training iteration 200 (batch 4 of epoch 49). [+][2018-08-15 16:20:56.437587] [PROGRESS] Training iteration 201 (batch 1 of epoch 50). [+][2018-08-15 16:20:56.458546] [PROGRESS] Training iteration 202 (batch 2 of epoch 50). [+][2018-08-15 16:20:56.479786] [PROGRESS] Training iteration 203 (batch 3 of epoch 50). [+][2018-08-15 16:20:56.501596] [PROGRESS] Training iteration 204 (batch 4 of epoch 50). [+][2018-08-15 16:20:56.567954] [PROGRESS] Training iteration 205 (batch 1 of epoch 51). [+][2018-08-15 16:20:56.587426] [PROGRESS] Training iteration 206 (batch 2 of epoch 51). [+][2018-08-15 16:20:56.608514] [PROGRESS] Training iteration 207 (batch 3 of epoch 51). [+][2018-08-15 16:20:56.629449] [PROGRESS] Training iteration 208 (batch 4 of epoch 51). [+][2018-08-15 16:20:56.689723] [PROGRESS] Training iteration 209 (batch 1 of epoch 52). [+][2018-08-15 16:20:56.710176] [PROGRESS] Training iteration 210 (batch 2 of epoch 52). [+][2018-08-15 16:20:56.732175] [PROGRESS] Training iteration 211 (batch 3 of epoch 52). [+][2018-08-15 16:20:56.755050] [PROGRESS] Training iteration 212 (batch 4 of epoch 52). [+][2018-08-15 16:20:56.816782] [PROGRESS] Training iteration 213 (batch 1 of epoch 53). [+][2018-08-15 16:20:56.836810] [PROGRESS] Training iteration 214 (batch 2 of epoch 53). [+][2018-08-15 16:20:56.860122] [PROGRESS] Training iteration 215 (batch 3 of epoch 53). [+][2018-08-15 16:20:56.881437] [PROGRESS] Training iteration 216 (batch 4 of epoch 53). [+][2018-08-15 16:20:56.950169] [PROGRESS] Training iteration 217 (batch 1 of epoch 54). [+][2018-08-15 16:20:56.972057] [PROGRESS] Training iteration 218 (batch 2 of epoch 54). [+][2018-08-15 16:20:56.993448] [PROGRESS] Training iteration 219 (batch 3 of epoch 54). [+][2018-08-15 16:20:57.015772] [PROGRESS] Training iteration 220 (batch 4 of epoch 54). [+][2018-08-15 16:20:57.078033] [PROGRESS] Training iteration 221 (batch 1 of epoch 55). [+][2018-08-15 16:20:57.099395] [PROGRESS] Training iteration 222 (batch 2 of epoch 55). [+][2018-08-15 16:20:57.124403] [PROGRESS] Training iteration 223 (batch 3 of epoch 55). [+][2018-08-15 16:20:57.147258] [PROGRESS] Training iteration 224 (batch 4 of epoch 55). [+][2018-08-15 16:20:57.209529] [PROGRESS] Training iteration 225 (batch 1 of epoch 56). [+][2018-08-15 16:20:57.229484] [PROGRESS] Training iteration 226 (batch 2 of epoch 56). [+][2018-08-15 16:20:57.249489] [PROGRESS] Training iteration 227 (batch 3 of epoch 56). [+][2018-08-15 16:20:57.270724] [PROGRESS] Training iteration 228 (batch 4 of epoch 56). [+][2018-08-15 16:20:57.330873] [PROGRESS] Training iteration 229 (batch 1 of epoch 57). [+][2018-08-15 16:20:57.350666] [PROGRESS] Training iteration 230 (batch 2 of epoch 57). [+][2018-08-15 16:20:57.372097] [PROGRESS] Training iteration 231 (batch 3 of epoch 57). [+][2018-08-15 16:20:57.393144] [PROGRESS] Training iteration 232 (batch 4 of epoch 57). [+][2018-08-15 16:20:57.453501] [PROGRESS] Training iteration 233 (batch 1 of epoch 58). [+][2018-08-15 16:20:57.473641] [PROGRESS] Training iteration 234 (batch 2 of epoch 58). [+][2018-08-15 16:20:57.494846] [PROGRESS] Training iteration 235 (batch 3 of epoch 58). [+][2018-08-15 16:20:57.515314] [PROGRESS] Training iteration 236 (batch 4 of epoch 58). [+][2018-08-15 16:20:57.576118] [PROGRESS] Training iteration 237 (batch 1 of epoch 59). [+][2018-08-15 16:20:57.596035] [PROGRESS] Training iteration 238 (batch 2 of epoch 59). [+][2018-08-15 16:20:57.617113] [PROGRESS] Training iteration 239 (batch 3 of epoch 59). [+][2018-08-15 16:20:57.639001] [PROGRESS] Training iteration 240 (batch 4 of epoch 59). [+][2018-08-15 16:20:57.701732] [PROGRESS] Training iteration 241 (batch 1 of epoch 60). [+][2018-08-15 16:20:57.723168] [PROGRESS] Training iteration 242 (batch 2 of epoch 60). [+][2018-08-15 16:20:57.744521] [PROGRESS] Training iteration 243 (batch 3 of epoch 60). [+][2018-08-15 16:20:57.765773] [PROGRESS] Training iteration 244 (batch 4 of epoch 60). [+][2018-08-15 16:20:57.831103] [PROGRESS] Training iteration 245 (batch 1 of epoch 61). [+][2018-08-15 16:20:57.851448] [PROGRESS] Training iteration 246 (batch 2 of epoch 61). [+][2018-08-15 16:20:57.871966] [PROGRESS] Training iteration 247 (batch 3 of epoch 61). [+][2018-08-15 16:20:57.891153] [PROGRESS] Training iteration 248 (batch 4 of epoch 61). [+][2018-08-15 16:20:57.953536] [PROGRESS] Training iteration 249 (batch 1 of epoch 62). [+][2018-08-15 16:20:57.973348] [PROGRESS] Training iteration 250 (batch 2 of epoch 62). [+][2018-08-15 16:20:57.994123] [PROGRESS] Training iteration 251 (batch 3 of epoch 62). [+][2018-08-15 16:20:58.015141] [PROGRESS] Training iteration 252 (batch 4 of epoch 62). [+][2018-08-15 16:20:58.075293] [PROGRESS] Training iteration 253 (batch 1 of epoch 63). [+][2018-08-15 16:20:58.095397] [PROGRESS] Training iteration 254 (batch 2 of epoch 63). [+][2018-08-15 16:20:58.116567] [PROGRESS] Training iteration 255 (batch 3 of epoch 63). [+][2018-08-15 16:20:58.138474] [PROGRESS] Training iteration 256 (batch 4 of epoch 63). [+][2018-08-15 16:20:58.200317] [PROGRESS] Training iteration 257 (batch 1 of epoch 64). [+][2018-08-15 16:20:58.221051] [PROGRESS] Training iteration 258 (batch 2 of epoch 64). [+][2018-08-15 16:20:58.244345] [PROGRESS] Training iteration 259 (batch 3 of epoch 64). [+][2018-08-15 16:20:58.265655] [PROGRESS] Training iteration 260 (batch 4 of epoch 64). [+][2018-08-15 16:20:58.327194] [PROGRESS] Training iteration 261 (batch 1 of epoch 65). [+][2018-08-15 16:20:58.348064] [PROGRESS] Training iteration 262 (batch 2 of epoch 65). [+][2018-08-15 16:20:58.371116] [PROGRESS] Training iteration 263 (batch 3 of epoch 65). [+][2018-08-15 16:20:58.394816] [PROGRESS] Training iteration 264 (batch 4 of epoch 65). [+][2018-08-15 16:20:58.455447] [PROGRESS] Training iteration 265 (batch 1 of epoch 66). [+][2018-08-15 16:20:58.475373] [PROGRESS] Training iteration 266 (batch 2 of epoch 66). [+][2018-08-15 16:20:58.495596] [PROGRESS] Training iteration 267 (batch 3 of epoch 66). [+][2018-08-15 16:20:58.516581] [PROGRESS] Training iteration 268 (batch 4 of epoch 66). [+][2018-08-15 16:20:58.578533] [PROGRESS] Training iteration 269 (batch 1 of epoch 67). [+][2018-08-15 16:20:58.598981] [PROGRESS] Training iteration 270 (batch 2 of epoch 67). [+][2018-08-15 16:20:58.624627] [PROGRESS] Training iteration 271 (batch 3 of epoch 67). [+][2018-08-15 16:20:58.651144] [PROGRESS] Training iteration 272 (batch 4 of epoch 67). [+][2018-08-15 16:20:58.720499] [PROGRESS] Training iteration 273 (batch 1 of epoch 68). [+][2018-08-15 16:20:58.741819] [PROGRESS] Training iteration 274 (batch 2 of epoch 68). [+][2018-08-15 16:20:58.765443] [PROGRESS] Training iteration 275 (batch 3 of epoch 68). [+][2018-08-15 16:20:58.790773] [PROGRESS] Training iteration 276 (batch 4 of epoch 68). [+][2018-08-15 16:20:58.859789] [PROGRESS] Training iteration 277 (batch 1 of epoch 69). [+][2018-08-15 16:20:58.880525] [PROGRESS] Training iteration 278 (batch 2 of epoch 69). [+][2018-08-15 16:20:58.904991] [PROGRESS] Training iteration 279 (batch 3 of epoch 69). [+][2018-08-15 16:20:58.931935] [PROGRESS] Training iteration 280 (batch 4 of epoch 69). [+][2018-08-15 16:20:59.004178] [PROGRESS] Training iteration 281 (batch 1 of epoch 70). [+][2018-08-15 16:20:59.025564] [PROGRESS] Training iteration 282 (batch 2 of epoch 70). [+][2018-08-15 16:20:59.046418] [PROGRESS] Training iteration 283 (batch 3 of epoch 70). [+][2018-08-15 16:20:59.070598] [PROGRESS] Training iteration 284 (batch 4 of epoch 70). [+][2018-08-15 16:20:59.133788] [PROGRESS] Training iteration 285 (batch 1 of epoch 71). [+][2018-08-15 16:20:59.158599] [PROGRESS] Training iteration 286 (batch 2 of epoch 71). [+][2018-08-15 16:20:59.180152] [PROGRESS] Training iteration 287 (batch 3 of epoch 71). [+][2018-08-15 16:20:59.200530] [PROGRESS] Training iteration 288 (batch 4 of epoch 71). [+][2018-08-15 16:20:59.262029] [PROGRESS] Training iteration 289 (batch 1 of epoch 72). [+][2018-08-15 16:20:59.284028] [PROGRESS] Training iteration 290 (batch 2 of epoch 72). [+][2018-08-15 16:20:59.307554] [PROGRESS] Training iteration 291 (batch 3 of epoch 72). [+][2018-08-15 16:20:59.331666] [PROGRESS] Training iteration 292 (batch 4 of epoch 72). [+][2018-08-15 16:20:59.399056] [PROGRESS] Training iteration 293 (batch 1 of epoch 73). [+][2018-08-15 16:20:59.419916] [PROGRESS] Training iteration 294 (batch 2 of epoch 73). [+][2018-08-15 16:20:59.440644] [PROGRESS] Training iteration 295 (batch 3 of epoch 73). [+][2018-08-15 16:20:59.460172] [PROGRESS] Training iteration 296 (batch 4 of epoch 73). [+][2018-08-15 16:20:59.523434] [PROGRESS] Training iteration 297 (batch 1 of epoch 74). [+][2018-08-15 16:20:59.542924] [PROGRESS] Training iteration 298 (batch 2 of epoch 74). [+][2018-08-15 16:20:59.564556] [PROGRESS] Training iteration 299 (batch 3 of epoch 74). [+][2018-08-15 16:20:59.585571] [PROGRESS] Training iteration 300 (batch 4 of epoch 74). [+][2018-08-15 16:20:59.651351] [PROGRESS] Training iteration 301 (batch 1 of epoch 75). [+][2018-08-15 16:20:59.672430] [PROGRESS] Training iteration 302 (batch 2 of epoch 75). [+][2018-08-15 16:20:59.695856] [PROGRESS] Training iteration 303 (batch 3 of epoch 75). [+][2018-08-15 16:20:59.715885] [PROGRESS] Training iteration 304 (batch 4 of epoch 75). [+][2018-08-15 16:20:59.779729] [PROGRESS] Training iteration 305 (batch 1 of epoch 76). [+][2018-08-15 16:20:59.800515] [PROGRESS] Training iteration 306 (batch 2 of epoch 76). [+][2018-08-15 16:20:59.823848] [PROGRESS] Training iteration 307 (batch 3 of epoch 76). [+][2018-08-15 16:20:59.843805] [PROGRESS] Training iteration 308 (batch 4 of epoch 76). [+][2018-08-15 16:20:59.906369] [PROGRESS] Training iteration 309 (batch 1 of epoch 77). [+][2018-08-15 16:20:59.927258] [PROGRESS] Training iteration 310 (batch 2 of epoch 77). [+][2018-08-15 16:20:59.947908] [PROGRESS] Training iteration 311 (batch 3 of epoch 77). [+][2018-08-15 16:20:59.969102] [PROGRESS] Training iteration 312 (batch 4 of epoch 77). [+][2018-08-15 16:21:00.038509] [PROGRESS] Training iteration 313 (batch 1 of epoch 78). [+][2018-08-15 16:21:00.059440] [PROGRESS] Training iteration 314 (batch 2 of epoch 78). [+][2018-08-15 16:21:00.082999] [PROGRESS] Training iteration 315 (batch 3 of epoch 78). [+][2018-08-15 16:21:00.106030] [PROGRESS] Training iteration 316 (batch 4 of epoch 78). [+][2018-08-15 16:21:00.166743] [PROGRESS] Training iteration 317 (batch 1 of epoch 79). [+][2018-08-15 16:21:00.188742] [PROGRESS] Training iteration 318 (batch 2 of epoch 79). [+][2018-08-15 16:21:00.212414] [PROGRESS] Training iteration 319 (batch 3 of epoch 79). [+][2018-08-15 16:21:00.235240] [PROGRESS] Training iteration 320 (batch 4 of epoch 79). [+][2018-08-15 16:21:00.295845] [PROGRESS] Training iteration 321 (batch 1 of epoch 80). [+][2018-08-15 16:21:00.317935] [PROGRESS] Training iteration 322 (batch 2 of epoch 80). [+][2018-08-15 16:21:00.340929] [PROGRESS] Training iteration 323 (batch 3 of epoch 80). [+][2018-08-15 16:21:00.363782] [PROGRESS] Training iteration 324 (batch 4 of epoch 80). [+][2018-08-15 16:21:00.427283] [PROGRESS] Training iteration 325 (batch 1 of epoch 81). [+][2018-08-15 16:21:00.447322] [PROGRESS] Training iteration 326 (batch 2 of epoch 81). [+][2018-08-15 16:21:00.468279] [PROGRESS] Training iteration 327 (batch 3 of epoch 81). [+][2018-08-15 16:21:00.488983] [PROGRESS] Training iteration 328 (batch 4 of epoch 81). [+][2018-08-15 16:21:00.550598] [PROGRESS] Training iteration 329 (batch 1 of epoch 82). [+][2018-08-15 16:21:00.570647] [PROGRESS] Training iteration 330 (batch 2 of epoch 82). [+][2018-08-15 16:21:00.591130] [PROGRESS] Training iteration 331 (batch 3 of epoch 82). [+][2018-08-15 16:21:00.611542] [PROGRESS] Training iteration 332 (batch 4 of epoch 82). [+][2018-08-15 16:21:00.672627] [PROGRESS] Training iteration 333 (batch 1 of epoch 83). [+][2018-08-15 16:21:00.692369] [PROGRESS] Training iteration 334 (batch 2 of epoch 83). [+][2018-08-15 16:21:00.712651] [PROGRESS] Training iteration 335 (batch 3 of epoch 83). [+][2018-08-15 16:21:00.734332] [PROGRESS] Training iteration 336 (batch 4 of epoch 83). [+][2018-08-15 16:21:00.795538] [PROGRESS] Training iteration 337 (batch 1 of epoch 84). [+][2018-08-15 16:21:00.815841] [PROGRESS] Training iteration 338 (batch 2 of epoch 84). [+][2018-08-15 16:21:00.836180] [PROGRESS] Training iteration 339 (batch 3 of epoch 84). [+][2018-08-15 16:21:00.857329] [PROGRESS] Training iteration 340 (batch 4 of epoch 84). [+][2018-08-15 16:21:00.918058] [PROGRESS] Training iteration 341 (batch 1 of epoch 85). [+][2018-08-15 16:21:00.937781] [PROGRESS] Training iteration 342 (batch 2 of epoch 85). [+][2018-08-15 16:21:00.958675] [PROGRESS] Training iteration 343 (batch 3 of epoch 85). [+][2018-08-15 16:21:00.979087] [PROGRESS] Training iteration 344 (batch 4 of epoch 85). [+][2018-08-15 16:21:01.040129] [PROGRESS] Training iteration 345 (batch 1 of epoch 86). [+][2018-08-15 16:21:01.060074] [PROGRESS] Training iteration 346 (batch 2 of epoch 86). [+][2018-08-15 16:21:01.081104] [PROGRESS] Training iteration 347 (batch 3 of epoch 86). [+][2018-08-15 16:21:01.103075] [PROGRESS] Training iteration 348 (batch 4 of epoch 86). [+][2018-08-15 16:21:01.169432] [PROGRESS] Training iteration 349 (batch 1 of epoch 87). [+][2018-08-15 16:21:01.188745] [PROGRESS] Training iteration 350 (batch 2 of epoch 87). [+][2018-08-15 16:21:01.208974] [PROGRESS] Training iteration 351 (batch 3 of epoch 87). [+][2018-08-15 16:21:01.230149] [PROGRESS] Training iteration 352 (batch 4 of epoch 87). [+][2018-08-15 16:21:01.291845] [PROGRESS] Training iteration 353 (batch 1 of epoch 88). [+][2018-08-15 16:21:01.311570] [PROGRESS] Training iteration 354 (batch 2 of epoch 88). [+][2018-08-15 16:21:01.332643] [PROGRESS] Training iteration 355 (batch 3 of epoch 88). [+][2018-08-15 16:21:01.355892] [PROGRESS] Training iteration 356 (batch 4 of epoch 88). [+][2018-08-15 16:21:01.416700] [PROGRESS] Training iteration 357 (batch 1 of epoch 89). [+][2018-08-15 16:21:01.437103] [PROGRESS] Training iteration 358 (batch 2 of epoch 89). [+][2018-08-15 16:21:01.458782] [PROGRESS] Training iteration 359 (batch 3 of epoch 89). [+][2018-08-15 16:21:01.479497] [PROGRESS] Training iteration 360 (batch 4 of epoch 89). [+][2018-08-15 16:21:01.540528] [PROGRESS] Training iteration 361 (batch 1 of epoch 90). [+][2018-08-15 16:21:01.559900] [PROGRESS] Training iteration 362 (batch 2 of epoch 90). [+][2018-08-15 16:21:01.581080] [PROGRESS] Training iteration 363 (batch 3 of epoch 90). [+][2018-08-15 16:21:01.602037] [PROGRESS] Training iteration 364 (batch 4 of epoch 90). [+][2018-08-15 16:21:01.663461] [PROGRESS] Training iteration 365 (batch 1 of epoch 91). [+][2018-08-15 16:21:01.683094] [PROGRESS] Training iteration 366 (batch 2 of epoch 91). [+][2018-08-15 16:21:01.703378] [PROGRESS] Training iteration 367 (batch 3 of epoch 91). [+][2018-08-15 16:21:01.724247] [PROGRESS] Training iteration 368 (batch 4 of epoch 91). [+][2018-08-15 16:21:01.784762] [PROGRESS] Training iteration 369 (batch 1 of epoch 92). [+][2018-08-15 16:21:01.804561] [PROGRESS] Training iteration 370 (batch 2 of epoch 92). [+][2018-08-15 16:21:01.824757] [PROGRESS] Training iteration 371 (batch 3 of epoch 92). [+][2018-08-15 16:21:01.845680] [PROGRESS] Training iteration 372 (batch 4 of epoch 92). [+][2018-08-15 16:21:01.905811] [PROGRESS] Training iteration 373 (batch 1 of epoch 93). [+][2018-08-15 16:21:01.926074] [PROGRESS] Training iteration 374 (batch 2 of epoch 93). [+][2018-08-15 16:21:01.947019] [PROGRESS] Training iteration 375 (batch 3 of epoch 93). [+][2018-08-15 16:21:01.967021] [PROGRESS] Training iteration 376 (batch 4 of epoch 93). [+][2018-08-15 16:21:02.028200] [PROGRESS] Training iteration 377 (batch 1 of epoch 94). [+][2018-08-15 16:21:02.047724] [PROGRESS] Training iteration 378 (batch 2 of epoch 94). [+][2018-08-15 16:21:02.068442] [PROGRESS] Training iteration 379 (batch 3 of epoch 94). [+][2018-08-15 16:21:02.088618] [PROGRESS] Training iteration 380 (batch 4 of epoch 94). [+][2018-08-15 16:21:02.149129] [PROGRESS] Training iteration 381 (batch 1 of epoch 95). [+][2018-08-15 16:21:02.168995] [PROGRESS] Training iteration 382 (batch 2 of epoch 95). [+][2018-08-15 16:21:02.190180] [PROGRESS] Training iteration 383 (batch 3 of epoch 95). [+][2018-08-15 16:21:02.210728] [PROGRESS] Training iteration 384 (batch 4 of epoch 95). [+][2018-08-15 16:21:02.272209] [PROGRESS] Training iteration 385 (batch 1 of epoch 96). [+][2018-08-15 16:21:02.292809] [PROGRESS] Training iteration 386 (batch 2 of epoch 96). [+][2018-08-15 16:21:02.314156] [PROGRESS] Training iteration 387 (batch 3 of epoch 96). [+][2018-08-15 16:21:02.335834] [PROGRESS] Training iteration 388 (batch 4 of epoch 96). [+][2018-08-15 16:21:02.396978] [PROGRESS] Training iteration 389 (batch 1 of epoch 97). [+][2018-08-15 16:21:02.416860] [PROGRESS] Training iteration 390 (batch 2 of epoch 97). [+][2018-08-15 16:21:02.437433] [PROGRESS] Training iteration 391 (batch 3 of epoch 97). [+][2018-08-15 16:21:02.458174] [PROGRESS] Training iteration 392 (batch 4 of epoch 97). [+][2018-08-15 16:21:02.519703] [PROGRESS] Training iteration 393 (batch 1 of epoch 98). [+][2018-08-15 16:21:02.540137] [PROGRESS] Training iteration 394 (batch 2 of epoch 98). [+][2018-08-15 16:21:02.560196] [PROGRESS] Training iteration 395 (batch 3 of epoch 98). [+][2018-08-15 16:21:02.581444] [PROGRESS] Training iteration 396 (batch 4 of epoch 98). [+][2018-08-15 16:21:02.642556] [PROGRESS] Training iteration 397 (batch 1 of epoch 99). [+][2018-08-15 16:21:02.662351] [PROGRESS] Training iteration 398 (batch 2 of epoch 99). [+][2018-08-15 16:21:02.682787] [PROGRESS] Training iteration 399 (batch 3 of epoch 99). [+][2018-08-15 16:21:02.703202] [PROGRESS] Training iteration 400 (batch 4 of epoch 99). [+][2018-08-15 16:21:02.764234] [PROGRESS] Training iteration 401 (batch 1 of epoch 100). [+][2018-08-15 16:21:02.783889] [PROGRESS] Training iteration 402 (batch 2 of epoch 100). [+][2018-08-15 16:21:02.805146] [PROGRESS] Training iteration 403 (batch 3 of epoch 100). [+][2018-08-15 16:21:02.825795] [PROGRESS] Training iteration 404 (batch 4 of epoch 100). [+][2018-08-15 16:21:02.886740] [PROGRESS] Training iteration 405 (batch 1 of epoch 101). [+][2018-08-15 16:21:02.906854] [PROGRESS] Training iteration 406 (batch 2 of epoch 101). [+][2018-08-15 16:21:02.927069] [PROGRESS] Training iteration 407 (batch 3 of epoch 101). [+][2018-08-15 16:21:02.946585] [PROGRESS] Training iteration 408 (batch 4 of epoch 101). [+][2018-08-15 16:21:03.012949] [PROGRESS] Training iteration 409 (batch 1 of epoch 102). [+][2018-08-15 16:21:03.032814] [PROGRESS] Training iteration 410 (batch 2 of epoch 102). [+][2018-08-15 16:21:03.053764] [PROGRESS] Training iteration 411 (batch 3 of epoch 102). [+][2018-08-15 16:21:03.074399] [PROGRESS] Training iteration 412 (batch 4 of epoch 102). [+][2018-08-15 16:21:03.135456] [PROGRESS] Training iteration 413 (batch 1 of epoch 103). [+][2018-08-15 16:21:03.156109] [PROGRESS] Training iteration 414 (batch 2 of epoch 103). [+][2018-08-15 16:21:03.177449] [PROGRESS] Training iteration 415 (batch 3 of epoch 103). [+][2018-08-15 16:21:03.200290] [PROGRESS] Training iteration 416 (batch 4 of epoch 103). [+][2018-08-15 16:21:03.262054] [PROGRESS] Training iteration 417 (batch 1 of epoch 104). [+][2018-08-15 16:21:03.284692] [PROGRESS] Training iteration 418 (batch 2 of epoch 104). [+][2018-08-15 16:21:03.309798] [PROGRESS] Training iteration 419 (batch 3 of epoch 104). [+][2018-08-15 16:21:03.333681] [PROGRESS] Training iteration 420 (batch 4 of epoch 104). [+][2018-08-15 16:21:03.395118] [PROGRESS] Training iteration 421 (batch 1 of epoch 105). [+][2018-08-15 16:21:03.416701] [PROGRESS] Training iteration 422 (batch 2 of epoch 105). [+][2018-08-15 16:21:03.440172] [PROGRESS] Training iteration 423 (batch 3 of epoch 105). [+][2018-08-15 16:21:03.464092] [PROGRESS] Training iteration 424 (batch 4 of epoch 105). [+][2018-08-15 16:21:03.525390] [PROGRESS] Training iteration 425 (batch 1 of epoch 106). [+][2018-08-15 16:21:03.545410] [PROGRESS] Training iteration 426 (batch 2 of epoch 106). [+][2018-08-15 16:21:03.566069] [PROGRESS] Training iteration 427 (batch 3 of epoch 106). [+][2018-08-15 16:21:03.586153] [PROGRESS] Training iteration 428 (batch 4 of epoch 106). [+][2018-08-15 16:21:03.645776] [PROGRESS] Training iteration 429 (batch 1 of epoch 107). [+][2018-08-15 16:21:03.665985] [PROGRESS] Training iteration 430 (batch 2 of epoch 107). [+][2018-08-15 16:21:03.686680] [PROGRESS] Training iteration 431 (batch 3 of epoch 107). [+][2018-08-15 16:21:03.707081] [PROGRESS] Training iteration 432 (batch 4 of epoch 107). [+][2018-08-15 16:21:03.768706] [PROGRESS] Training iteration 433 (batch 1 of epoch 108). [+][2018-08-15 16:21:03.788428] [PROGRESS] Training iteration 434 (batch 2 of epoch 108). [+][2018-08-15 16:21:03.808589] [PROGRESS] Training iteration 435 (batch 3 of epoch 108). [+][2018-08-15 16:21:03.828600] [PROGRESS] Training iteration 436 (batch 4 of epoch 108). [+][2018-08-15 16:21:03.889545] [PROGRESS] Training iteration 437 (batch 1 of epoch 109). [+][2018-08-15 16:21:03.909625] [PROGRESS] Training iteration 438 (batch 2 of epoch 109). [+][2018-08-15 16:21:03.930514] [PROGRESS] Training iteration 439 (batch 3 of epoch 109). [+][2018-08-15 16:21:03.951116] [PROGRESS] Training iteration 440 (batch 4 of epoch 109). [+][2018-08-15 16:21:04.012346] [PROGRESS] Training iteration 441 (batch 1 of epoch 110). [+][2018-08-15 16:21:04.034178] [PROGRESS] Training iteration 442 (batch 2 of epoch 110). [+][2018-08-15 16:21:04.055839] [PROGRESS] Training iteration 443 (batch 3 of epoch 110). [+][2018-08-15 16:21:04.078700] [PROGRESS] Training iteration 444 (batch 4 of epoch 110). [+][2018-08-15 16:21:04.139964] [PROGRESS] Training iteration 445 (batch 1 of epoch 111). [+][2018-08-15 16:21:04.160724] [PROGRESS] Training iteration 446 (batch 2 of epoch 111). [+][2018-08-15 16:21:04.183073] [PROGRESS] Training iteration 447 (batch 3 of epoch 111). [+][2018-08-15 16:21:04.206374] [PROGRESS] Training iteration 448 (batch 4 of epoch 111). [+][2018-08-15 16:21:04.268083] [PROGRESS] Training iteration 449 (batch 1 of epoch 112). [+][2018-08-15 16:21:04.289227] [PROGRESS] Training iteration 450 (batch 2 of epoch 112). [+][2018-08-15 16:21:04.312499] [PROGRESS] Training iteration 451 (batch 3 of epoch 112). [+][2018-08-15 16:21:04.334810] [PROGRESS] Training iteration 452 (batch 4 of epoch 112). [+][2018-08-15 16:21:04.395335] [PROGRESS] Training iteration 453 (batch 1 of epoch 113). [+][2018-08-15 16:21:04.415987] [PROGRESS] Training iteration 454 (batch 2 of epoch 113). [+][2018-08-15 16:21:04.437056] [PROGRESS] Training iteration 455 (batch 3 of epoch 113). [+][2018-08-15 16:21:04.458429] [PROGRESS] Training iteration 456 (batch 4 of epoch 113). [+][2018-08-15 16:21:04.520764] [PROGRESS] Training iteration 457 (batch 1 of epoch 114). [+][2018-08-15 16:21:04.541714] [PROGRESS] Training iteration 458 (batch 2 of epoch 114). [+][2018-08-15 16:21:04.564633] [PROGRESS] Training iteration 459 (batch 3 of epoch 114). [+][2018-08-15 16:21:04.587313] [PROGRESS] Training iteration 460 (batch 4 of epoch 114). [+][2018-08-15 16:21:04.649419] [PROGRESS] Training iteration 461 (batch 1 of epoch 115). [+][2018-08-15 16:21:04.670891] [PROGRESS] Training iteration 462 (batch 2 of epoch 115). [+][2018-08-15 16:21:04.693847] [PROGRESS] Training iteration 463 (batch 3 of epoch 115). [+][2018-08-15 16:21:04.717005] [PROGRESS] Training iteration 464 (batch 4 of epoch 115). [+][2018-08-15 16:21:04.778324] [PROGRESS] Training iteration 465 (batch 1 of epoch 116). [+][2018-08-15 16:21:04.799610] [PROGRESS] Training iteration 466 (batch 2 of epoch 116). [+][2018-08-15 16:21:04.822129] [PROGRESS] Training iteration 467 (batch 3 of epoch 116). [+][2018-08-15 16:21:04.845089] [PROGRESS] Training iteration 468 (batch 4 of epoch 116). [+][2018-08-15 16:21:04.905883] [PROGRESS] Training iteration 469 (batch 1 of epoch 117). [+][2018-08-15 16:21:04.927076] [PROGRESS] Training iteration 470 (batch 2 of epoch 117). [+][2018-08-15 16:21:04.951647] [PROGRESS] Training iteration 471 (batch 3 of epoch 117). [+][2018-08-15 16:21:04.975009] [PROGRESS] Training iteration 472 (batch 4 of epoch 117). [+][2018-08-15 16:21:05.038495] [PROGRESS] Training iteration 473 (batch 1 of epoch 118). [+][2018-08-15 16:21:05.058939] [PROGRESS] Training iteration 474 (batch 2 of epoch 118). [+][2018-08-15 16:21:05.079854] [PROGRESS] Training iteration 475 (batch 3 of epoch 118). [+][2018-08-15 16:21:05.099930] [PROGRESS] Training iteration 476 (batch 4 of epoch 118). [+][2018-08-15 16:21:05.160742] [PROGRESS] Training iteration 477 (batch 1 of epoch 119). [+][2018-08-15 16:21:05.181622] [PROGRESS] Training iteration 478 (batch 2 of epoch 119). [+][2018-08-15 16:21:05.204175] [PROGRESS] Training iteration 479 (batch 3 of epoch 119). [+][2018-08-15 16:21:05.228002] [PROGRESS] Training iteration 480 (batch 4 of epoch 119). [+][2018-08-15 16:21:05.289002] [PROGRESS] Training iteration 481 (batch 1 of epoch 120). [+][2018-08-15 16:21:05.310533] [PROGRESS] Training iteration 482 (batch 2 of epoch 120). [+][2018-08-15 16:21:05.333752] [PROGRESS] Training iteration 483 (batch 3 of epoch 120). [+][2018-08-15 16:21:05.356962] [PROGRESS] Training iteration 484 (batch 4 of epoch 120). [+][2018-08-15 16:21:05.424951] [PROGRESS] Training iteration 485 (batch 1 of epoch 121). [+][2018-08-15 16:21:05.448713] [PROGRESS] Training iteration 486 (batch 2 of epoch 121). [+][2018-08-15 16:21:05.470817] [PROGRESS] Training iteration 487 (batch 3 of epoch 121). [+][2018-08-15 16:21:05.495483] [PROGRESS] Training iteration 488 (batch 4 of epoch 121). [+][2018-08-15 16:21:05.556384] [PROGRESS] Training iteration 489 (batch 1 of epoch 122). [+][2018-08-15 16:21:05.579711] [PROGRESS] Training iteration 490 (batch 2 of epoch 122). [+][2018-08-15 16:21:05.601323] [PROGRESS] Training iteration 491 (batch 3 of epoch 122). [+][2018-08-15 16:21:05.624086] [PROGRESS] Training iteration 492 (batch 4 of epoch 122). [+][2018-08-15 16:21:05.685117] [PROGRESS] Training iteration 493 (batch 1 of epoch 123). [+][2018-08-15 16:21:05.705931] [PROGRESS] Training iteration 494 (batch 2 of epoch 123). [+][2018-08-15 16:21:05.726326] [PROGRESS] Training iteration 495 (batch 3 of epoch 123). [+][2018-08-15 16:21:05.747512] [PROGRESS] Training iteration 496 (batch 4 of epoch 123). [+][2018-08-15 16:21:05.808097] [PROGRESS] Training iteration 497 (batch 1 of epoch 124). [+][2018-08-15 16:21:05.827821] [PROGRESS] Training iteration 498 (batch 2 of epoch 124). [+][2018-08-15 16:21:05.848799] [PROGRESS] Training iteration 499 (batch 3 of epoch 124). [+][2018-08-15 16:21:05.870029] [PROGRESS] Training iteration 500 (batch 4 of epoch 124). [+][2018-08-15 16:21:05.931350] [PROGRESS] Training iteration 501 (batch 1 of epoch 125). [+][2018-08-15 16:21:05.951112] [PROGRESS] Training iteration 502 (batch 2 of epoch 125). [+][2018-08-15 16:21:05.972557] [PROGRESS] Training iteration 503 (batch 3 of epoch 125). [+][2018-08-15 16:21:05.992658] [PROGRESS] Training iteration 504 (batch 4 of epoch 125). [+][2018-08-15 16:21:06.054760] [PROGRESS] Training iteration 505 (batch 1 of epoch 126). [+][2018-08-15 16:21:06.075717] [PROGRESS] Training iteration 506 (batch 2 of epoch 126). [+][2018-08-15 16:21:06.097923] [PROGRESS] Training iteration 507 (batch 3 of epoch 126). [+][2018-08-15 16:21:06.119763] [PROGRESS] Training iteration 508 (batch 4 of epoch 126). [+][2018-08-15 16:21:06.183140] [PROGRESS] Training iteration 509 (batch 1 of epoch 127). [+][2018-08-15 16:21:06.205468] [PROGRESS] Training iteration 510 (batch 2 of epoch 127). [+][2018-08-15 16:21:06.225093] [PROGRESS] Training iteration 511 (batch 3 of epoch 127). [+][2018-08-15 16:21:06.244819] [PROGRESS] Training iteration 512 (batch 4 of epoch 127). [+][2018-08-15 16:21:06.306018] [PROGRESS] Training iteration 513 (batch 1 of epoch 128). [+][2018-08-15 16:21:06.325964] [PROGRESS] Training iteration 514 (batch 2 of epoch 128). [+][2018-08-15 16:21:06.347391] [PROGRESS] Training iteration 515 (batch 3 of epoch 128). [+][2018-08-15 16:21:06.367998] [PROGRESS] Training iteration 516 (batch 4 of epoch 128). [+][2018-08-15 16:21:06.428976] [PROGRESS] Training iteration 517 (batch 1 of epoch 129). [+][2018-08-15 16:21:06.449416] [PROGRESS] Training iteration 518 (batch 2 of epoch 129). [+][2018-08-15 16:21:06.472826] [PROGRESS] Training iteration 519 (batch 3 of epoch 129). [+][2018-08-15 16:21:06.497068] [PROGRESS] Training iteration 520 (batch 4 of epoch 129). [+][2018-08-15 16:21:06.560373] [PROGRESS] Training iteration 521 (batch 1 of epoch 130). [+][2018-08-15 16:21:06.582636] [PROGRESS] Training iteration 522 (batch 2 of epoch 130). [+][2018-08-15 16:21:06.605863] [PROGRESS] Training iteration 523 (batch 3 of epoch 130). [+][2018-08-15 16:21:06.628357] [PROGRESS] Training iteration 524 (batch 4 of epoch 130). [+][2018-08-15 16:21:06.689300] [PROGRESS] Training iteration 525 (batch 1 of epoch 131). [+][2018-08-15 16:21:06.710164] [PROGRESS] Training iteration 526 (batch 2 of epoch 131). [+][2018-08-15 16:21:06.733517] [PROGRESS] Training iteration 527 (batch 3 of epoch 131). [+][2018-08-15 16:21:06.753593] [PROGRESS] Training iteration 528 (batch 4 of epoch 131). [+][2018-08-15 16:21:06.817133] [PROGRESS] Training iteration 529 (batch 1 of epoch 132). [+][2018-08-15 16:21:06.836216] [PROGRESS] Training iteration 530 (batch 2 of epoch 132). [+][2018-08-15 16:21:06.859685] [PROGRESS] Training iteration 531 (batch 3 of epoch 132). [+][2018-08-15 16:21:06.883053] [PROGRESS] Training iteration 532 (batch 4 of epoch 132). [+][2018-08-15 16:21:06.944187] [PROGRESS] Training iteration 533 (batch 1 of epoch 133). [+][2018-08-15 16:21:06.963920] [PROGRESS] Training iteration 534 (batch 2 of epoch 133). [+][2018-08-15 16:21:06.986954] [PROGRESS] Training iteration 535 (batch 3 of epoch 133). [+][2018-08-15 16:21:07.009477] [PROGRESS] Training iteration 536 (batch 4 of epoch 133). [+][2018-08-15 16:21:07.071069] [PROGRESS] Training iteration 537 (batch 1 of epoch 134). [+][2018-08-15 16:21:07.091642] [PROGRESS] Training iteration 538 (batch 2 of epoch 134). [+][2018-08-15 16:21:07.114341] [PROGRESS] Training iteration 539 (batch 3 of epoch 134). [+][2018-08-15 16:21:07.137677] [PROGRESS] Training iteration 540 (batch 4 of epoch 134). [+][2018-08-15 16:21:07.199733] [PROGRESS] Training iteration 541 (batch 1 of epoch 135). [+][2018-08-15 16:21:07.220220] [PROGRESS] Training iteration 542 (batch 2 of epoch 135). [+][2018-08-15 16:21:07.241506] [PROGRESS] Training iteration 543 (batch 3 of epoch 135). [+][2018-08-15 16:21:07.264045] [PROGRESS] Training iteration 544 (batch 4 of epoch 135). [+][2018-08-15 16:21:07.326251] [PROGRESS] Training iteration 545 (batch 1 of epoch 136). [+][2018-08-15 16:21:07.346539] [PROGRESS] Training iteration 546 (batch 2 of epoch 136). [+][2018-08-15 16:21:07.369420] [PROGRESS] Training iteration 547 (batch 3 of epoch 136). [+][2018-08-15 16:21:07.393396] [PROGRESS] Training iteration 548 (batch 4 of epoch 136). [+][2018-08-15 16:21:07.456444] [PROGRESS] Training iteration 549 (batch 1 of epoch 137). [+][2018-08-15 16:21:07.478165] [PROGRESS] Training iteration 550 (batch 2 of epoch 137). [+][2018-08-15 16:21:07.501072] [PROGRESS] Training iteration 551 (batch 3 of epoch 137). [+][2018-08-15 16:21:07.526136] [PROGRESS] Training iteration 552 (batch 4 of epoch 137). [+][2018-08-15 16:21:07.587454] [PROGRESS] Training iteration 553 (batch 1 of epoch 138). [+][2018-08-15 16:21:07.609043] [PROGRESS] Training iteration 554 (batch 2 of epoch 138). [+][2018-08-15 16:21:07.633538] [PROGRESS] Training iteration 555 (batch 3 of epoch 138). [+][2018-08-15 16:21:07.657072] [PROGRESS] Training iteration 556 (batch 4 of epoch 138). [+][2018-08-15 16:21:07.718618] [PROGRESS] Training iteration 557 (batch 1 of epoch 139). [+][2018-08-15 16:21:07.740326] [PROGRESS] Training iteration 558 (batch 2 of epoch 139). [+][2018-08-15 16:21:07.763617] [PROGRESS] Training iteration 559 (batch 3 of epoch 139). [+][2018-08-15 16:21:07.787054] [PROGRESS] Training iteration 560 (batch 4 of epoch 139). [+][2018-08-15 16:21:07.848788] [PROGRESS] Training iteration 561 (batch 1 of epoch 140). [+][2018-08-15 16:21:07.869920] [PROGRESS] Training iteration 562 (batch 2 of epoch 140). [+][2018-08-15 16:21:07.892450] [PROGRESS] Training iteration 563 (batch 3 of epoch 140). [+][2018-08-15 16:21:07.915831] [PROGRESS] Training iteration 564 (batch 4 of epoch 140). [+][2018-08-15 16:21:07.977298] [PROGRESS] Training iteration 565 (batch 1 of epoch 141). [+][2018-08-15 16:21:07.998699] [PROGRESS] Training iteration 566 (batch 2 of epoch 141). [+][2018-08-15 16:21:08.021414] [PROGRESS] Training iteration 567 (batch 3 of epoch 141). [+][2018-08-15 16:21:08.043990] [PROGRESS] Training iteration 568 (batch 4 of epoch 141). [+][2018-08-15 16:21:08.106663] [PROGRESS] Training iteration 569 (batch 1 of epoch 142). [+][2018-08-15 16:21:08.129563] [PROGRESS] Training iteration 570 (batch 2 of epoch 142). [+][2018-08-15 16:21:08.150563] [PROGRESS] Training iteration 571 (batch 3 of epoch 142). [+][2018-08-15 16:21:08.171075] [PROGRESS] Training iteration 572 (batch 4 of epoch 142). [+][2018-08-15 16:21:08.232513] [PROGRESS] Training iteration 573 (batch 1 of epoch 143). [+][2018-08-15 16:21:08.253022] [PROGRESS] Training iteration 574 (batch 2 of epoch 143). [+][2018-08-15 16:21:08.273126] [PROGRESS] Training iteration 575 (batch 3 of epoch 143). [+][2018-08-15 16:21:08.293956] [PROGRESS] Training iteration 576 (batch 4 of epoch 143). [+][2018-08-15 16:21:08.354991] [PROGRESS] Training iteration 577 (batch 1 of epoch 144). [+][2018-08-15 16:21:08.375326] [PROGRESS] Training iteration 578 (batch 2 of epoch 144). [+][2018-08-15 16:21:08.397403] [PROGRESS] Training iteration 579 (batch 3 of epoch 144). [+][2018-08-15 16:21:08.418624] [PROGRESS] Training iteration 580 (batch 4 of epoch 144). [+][2018-08-15 16:21:08.480553] [PROGRESS] Training iteration 581 (batch 1 of epoch 145). [+][2018-08-15 16:21:08.500517] [PROGRESS] Training iteration 582 (batch 2 of epoch 145). [+][2018-08-15 16:21:08.521439] [PROGRESS] Training iteration 583 (batch 3 of epoch 145). [+][2018-08-15 16:21:08.542754] [PROGRESS] Training iteration 584 (batch 4 of epoch 145). [+][2018-08-15 16:21:08.603742] [PROGRESS] Training iteration 585 (batch 1 of epoch 146). [+][2018-08-15 16:21:08.623053] [PROGRESS] Training iteration 586 (batch 2 of epoch 146). [+][2018-08-15 16:21:08.643575] [PROGRESS] Training iteration 587 (batch 3 of epoch 146). [+][2018-08-15 16:21:08.665019] [PROGRESS] Training iteration 588 (batch 4 of epoch 146). [+][2018-08-15 16:21:08.725997] [PROGRESS] Training iteration 589 (batch 1 of epoch 147). [+][2018-08-15 16:21:08.745732] [PROGRESS] Training iteration 590 (batch 2 of epoch 147). [+][2018-08-15 16:21:08.766137] [PROGRESS] Training iteration 591 (batch 3 of epoch 147). [+][2018-08-15 16:21:08.787381] [PROGRESS] Training iteration 592 (batch 4 of epoch 147). [+][2018-08-15 16:21:08.851089] [PROGRESS] Training iteration 593 (batch 1 of epoch 148). [+][2018-08-15 16:21:08.873367] [PROGRESS] Training iteration 594 (batch 2 of epoch 148). [+][2018-08-15 16:21:08.897208] [PROGRESS] Training iteration 595 (batch 3 of epoch 148). [+][2018-08-15 16:21:08.921332] [PROGRESS] Training iteration 596 (batch 4 of epoch 148). [+][2018-08-15 16:21:08.985850] [PROGRESS] Training iteration 597 (batch 1 of epoch 149). [+][2018-08-15 16:21:09.008411] [PROGRESS] Training iteration 598 (batch 2 of epoch 149). [+][2018-08-15 16:21:09.032513] [PROGRESS] Training iteration 599 (batch 3 of epoch 149). [+][2018-08-15 16:21:09.056683] [PROGRESS] Training iteration 600 (batch 4 of epoch 149). [+][2018-08-15 16:21:09.122766] [PROGRESS] Training iteration 601 (batch 1 of epoch 150). [+][2018-08-15 16:21:09.143847] [PROGRESS] Training iteration 602 (batch 2 of epoch 150). [+][2018-08-15 16:21:09.167380] [PROGRESS] Training iteration 603 (batch 3 of epoch 150). [+][2018-08-15 16:21:09.190342] [PROGRESS] Training iteration 604 (batch 4 of epoch 150). [+][2018-08-15 16:21:09.253351] [PROGRESS] Training iteration 605 (batch 1 of epoch 151). [+][2018-08-15 16:21:09.273400] [PROGRESS] Training iteration 606 (batch 2 of epoch 151). [+][2018-08-15 16:21:09.294107] [PROGRESS] Training iteration 607 (batch 3 of epoch 151). [+][2018-08-15 16:21:09.315186] [PROGRESS] Training iteration 608 (batch 4 of epoch 151). [+][2018-08-15 16:21:09.377126] [PROGRESS] Training iteration 609 (batch 1 of epoch 152). [+][2018-08-15 16:21:09.398143] [PROGRESS] Training iteration 610 (batch 2 of epoch 152). [+][2018-08-15 16:21:09.419109] [PROGRESS] Training iteration 611 (batch 3 of epoch 152). [+][2018-08-15 16:21:09.442552] [PROGRESS] Training iteration 612 (batch 4 of epoch 152). [+][2018-08-15 16:21:09.510958] [PROGRESS] Training iteration 613 (batch 1 of epoch 153). [+][2018-08-15 16:21:09.531084] [PROGRESS] Training iteration 614 (batch 2 of epoch 153). [+][2018-08-15 16:21:09.555107] [PROGRESS] Training iteration 615 (batch 3 of epoch 153). [+][2018-08-15 16:21:09.576281] [PROGRESS] Training iteration 616 (batch 4 of epoch 153). [+][2018-08-15 16:21:09.638938] [PROGRESS] Training iteration 617 (batch 1 of epoch 154). [+][2018-08-15 16:21:09.661356] [PROGRESS] Training iteration 618 (batch 2 of epoch 154). [+][2018-08-15 16:21:09.684018] [PROGRESS] Training iteration 619 (batch 3 of epoch 154). [+][2018-08-15 16:21:09.706652] [PROGRESS] Training iteration 620 (batch 4 of epoch 154). [+][2018-08-15 16:21:09.768048] [PROGRESS] Training iteration 621 (batch 1 of epoch 155). [+][2018-08-15 16:21:09.788390] [PROGRESS] Training iteration 622 (batch 2 of epoch 155). [+][2018-08-15 16:21:09.811271] [PROGRESS] Training iteration 623 (batch 3 of epoch 155). [+][2018-08-15 16:21:09.834102] [PROGRESS] Training iteration 624 (batch 4 of epoch 155). [+][2018-08-15 16:21:09.900089] [PROGRESS] Training iteration 625 (batch 1 of epoch 156). [+][2018-08-15 16:21:09.921017] [PROGRESS] Training iteration 626 (batch 2 of epoch 156). [+][2018-08-15 16:21:09.944376] [PROGRESS] Training iteration 627 (batch 3 of epoch 156). [+][2018-08-15 16:21:09.968132] [PROGRESS] Training iteration 628 (batch 4 of epoch 156). [+][2018-08-15 16:21:10.029451] [PROGRESS] Training iteration 629 (batch 1 of epoch 157). [+][2018-08-15 16:21:10.050123] [PROGRESS] Training iteration 630 (batch 2 of epoch 157). [+][2018-08-15 16:21:10.072875] [PROGRESS] Training iteration 631 (batch 3 of epoch 157). [+][2018-08-15 16:21:10.095199] [PROGRESS] Training iteration 632 (batch 4 of epoch 157). [+][2018-08-15 16:21:10.160194] [PROGRESS] Training iteration 633 (batch 1 of epoch 158). [+][2018-08-15 16:21:10.180682] [PROGRESS] Training iteration 634 (batch 2 of epoch 158). [+][2018-08-15 16:21:10.201679] [PROGRESS] Training iteration 635 (batch 3 of epoch 158). [+][2018-08-15 16:21:10.222768] [PROGRESS] Training iteration 636 (batch 4 of epoch 158). [+][2018-08-15 16:21:10.282788] [PROGRESS] Training iteration 637 (batch 1 of epoch 159). [+][2018-08-15 16:21:10.304023] [PROGRESS] Training iteration 638 (batch 2 of epoch 159). [+][2018-08-15 16:21:10.327266] [PROGRESS] Training iteration 639 (batch 3 of epoch 159). [+][2018-08-15 16:21:10.349832] [PROGRESS] Training iteration 640 (batch 4 of epoch 159). [+][2018-08-15 16:21:10.411047] [PROGRESS] Training iteration 641 (batch 1 of epoch 160). [+][2018-08-15 16:21:10.432244] [PROGRESS] Training iteration 642 (batch 2 of epoch 160). [+][2018-08-15 16:21:10.455959] [PROGRESS] Training iteration 643 (batch 3 of epoch 160). [+][2018-08-15 16:21:10.480899] [PROGRESS] Training iteration 644 (batch 4 of epoch 160). [+][2018-08-15 16:21:10.543759] [PROGRESS] Training iteration 645 (batch 1 of epoch 161). [+][2018-08-15 16:21:10.565494] [PROGRESS] Training iteration 646 (batch 2 of epoch 161). [+][2018-08-15 16:21:10.590719] [PROGRESS] Training iteration 647 (batch 3 of epoch 161). [+][2018-08-15 16:21:10.613154] [PROGRESS] Training iteration 648 (batch 4 of epoch 161). [+][2018-08-15 16:21:10.678151] [PROGRESS] Training iteration 649 (batch 1 of epoch 162). [+][2018-08-15 16:21:10.699049] [PROGRESS] Training iteration 650 (batch 2 of epoch 162). [+][2018-08-15 16:21:10.722539] [PROGRESS] Training iteration 651 (batch 3 of epoch 162). [+][2018-08-15 16:21:10.746265] [PROGRESS] Training iteration 652 (batch 4 of epoch 162). [+][2018-08-15 16:21:10.808558] [PROGRESS] Training iteration 653 (batch 1 of epoch 163). [+][2018-08-15 16:21:10.831649] [PROGRESS] Training iteration 654 (batch 2 of epoch 163). [+][2018-08-15 16:21:10.853535] [PROGRESS] Training iteration 655 (batch 3 of epoch 163). [+][2018-08-15 16:21:10.875992] [PROGRESS] Training iteration 656 (batch 4 of epoch 163). [+][2018-08-15 16:21:10.938721] [PROGRESS] Training iteration 657 (batch 1 of epoch 164). [+][2018-08-15 16:21:10.961404] [PROGRESS] Training iteration 658 (batch 2 of epoch 164). [+][2018-08-15 16:21:10.986653] [PROGRESS] Training iteration 659 (batch 3 of epoch 164). [+][2018-08-15 16:21:11.009557] [PROGRESS] Training iteration 660 (batch 4 of epoch 164). [+][2018-08-15 16:21:11.070963] [PROGRESS] Training iteration 661 (batch 1 of epoch 165). [+][2018-08-15 16:21:11.091683] [PROGRESS] Training iteration 662 (batch 2 of epoch 165). [+][2018-08-15 16:21:11.114301] [PROGRESS] Training iteration 663 (batch 3 of epoch 165). [+][2018-08-15 16:21:11.136949] [PROGRESS] Training iteration 664 (batch 4 of epoch 165). [+][2018-08-15 16:21:11.200270] [PROGRESS] Training iteration 665 (batch 1 of epoch 166). [+][2018-08-15 16:21:11.222362] [PROGRESS] Training iteration 666 (batch 2 of epoch 166). [+][2018-08-15 16:21:11.244864] [PROGRESS] Training iteration 667 (batch 3 of epoch 166). [+][2018-08-15 16:21:11.266857] [PROGRESS] Training iteration 668 (batch 4 of epoch 166). [+][2018-08-15 16:21:11.327271] [PROGRESS] Training iteration 669 (batch 1 of epoch 167). [+][2018-08-15 16:21:11.347207] [PROGRESS] Training iteration 670 (batch 2 of epoch 167). [+][2018-08-15 16:21:11.368407] [PROGRESS] Training iteration 671 (batch 3 of epoch 167). [+][2018-08-15 16:21:11.390543] [PROGRESS] Training iteration 672 (batch 4 of epoch 167). [+][2018-08-15 16:21:11.455461] [PROGRESS] Training iteration 673 (batch 1 of epoch 168). [+][2018-08-15 16:21:11.476708] [PROGRESS] Training iteration 674 (batch 2 of epoch 168). [+][2018-08-15 16:21:11.500166] [PROGRESS] Training iteration 675 (batch 3 of epoch 168). [+][2018-08-15 16:21:11.522438] [PROGRESS] Training iteration 676 (batch 4 of epoch 168). [+][2018-08-15 16:21:11.585094] [PROGRESS] Training iteration 677 (batch 1 of epoch 169). [+][2018-08-15 16:21:11.606909] [PROGRESS] Training iteration 678 (batch 2 of epoch 169). [+][2018-08-15 16:21:11.627411] [PROGRESS] Training iteration 679 (batch 3 of epoch 169). [+][2018-08-15 16:21:11.649982] [PROGRESS] Training iteration 680 (batch 4 of epoch 169). [+][2018-08-15 16:21:11.712846] [PROGRESS] Training iteration 681 (batch 1 of epoch 170). [+][2018-08-15 16:21:11.734131] [PROGRESS] Training iteration 682 (batch 2 of epoch 170). [+][2018-08-15 16:21:11.756473] [PROGRESS] Training iteration 683 (batch 3 of epoch 170). [+][2018-08-15 16:21:11.780565] [PROGRESS] Training iteration 684 (batch 4 of epoch 170). [+][2018-08-15 16:21:11.843522] [PROGRESS] Training iteration 685 (batch 1 of epoch 171). [+][2018-08-15 16:21:11.865617] [PROGRESS] Training iteration 686 (batch 2 of epoch 171). [+][2018-08-15 16:21:11.890153] [PROGRESS] Training iteration 687 (batch 3 of epoch 171). [+][2018-08-15 16:21:11.914589] [PROGRESS] Training iteration 688 (batch 4 of epoch 171). [+][2018-08-15 16:21:11.976684] [PROGRESS] Training iteration 689 (batch 1 of epoch 172). [+][2018-08-15 16:21:11.997589] [PROGRESS] Training iteration 690 (batch 2 of epoch 172). [+][2018-08-15 16:21:12.020141] [PROGRESS] Training iteration 691 (batch 3 of epoch 172). [+][2018-08-15 16:21:12.042125] [PROGRESS] Training iteration 692 (batch 4 of epoch 172). [+][2018-08-15 16:21:12.108818] [PROGRESS] Training iteration 693 (batch 1 of epoch 173). [+][2018-08-15 16:21:12.129070] [PROGRESS] Training iteration 694 (batch 2 of epoch 173). [+][2018-08-15 16:21:12.150474] [PROGRESS] Training iteration 695 (batch 3 of epoch 173). [+][2018-08-15 16:21:12.171890] [PROGRESS] Training iteration 696 (batch 4 of epoch 173). [+][2018-08-15 16:21:12.233529] [PROGRESS] Training iteration 697 (batch 1 of epoch 174). [+][2018-08-15 16:21:12.254877] [PROGRESS] Training iteration 698 (batch 2 of epoch 174). [+][2018-08-15 16:21:12.276008] [PROGRESS] Training iteration 699 (batch 3 of epoch 174). [+][2018-08-15 16:21:12.298852] [PROGRESS] Training iteration 700 (batch 4 of epoch 174). [+][2018-08-15 16:21:12.362203] [PROGRESS] Training iteration 701 (batch 1 of epoch 175). [+][2018-08-15 16:21:12.382099] [PROGRESS] Training iteration 702 (batch 2 of epoch 175). [+][2018-08-15 16:21:12.404773] [PROGRESS] Training iteration 703 (batch 3 of epoch 175). [+][2018-08-15 16:21:12.428237] [PROGRESS] Training iteration 704 (batch 4 of epoch 175). [+][2018-08-15 16:21:12.498573] [PROGRESS] Training iteration 705 (batch 1 of epoch 176). [+][2018-08-15 16:21:12.520476] [PROGRESS] Training iteration 706 (batch 2 of epoch 176). [+][2018-08-15 16:21:12.544329] [PROGRESS] Training iteration 707 (batch 3 of epoch 176). [+][2018-08-15 16:21:12.566679] [PROGRESS] Training iteration 708 (batch 4 of epoch 176). [+][2018-08-15 16:21:12.627848] [PROGRESS] Training iteration 709 (batch 1 of epoch 177). [+][2018-08-15 16:21:12.649699] [PROGRESS] Training iteration 710 (batch 2 of epoch 177). [+][2018-08-15 16:21:12.672938] [PROGRESS] Training iteration 711 (batch 3 of epoch 177). [+][2018-08-15 16:21:12.695544] [PROGRESS] Training iteration 712 (batch 4 of epoch 177). [+][2018-08-15 16:21:12.757133] [PROGRESS] Training iteration 713 (batch 1 of epoch 178). [+][2018-08-15 16:21:12.778491] [PROGRESS] Training iteration 714 (batch 2 of epoch 178). [+][2018-08-15 16:21:12.802458] [PROGRESS] Training iteration 715 (batch 3 of epoch 178). [+][2018-08-15 16:21:12.824186] [PROGRESS] Training iteration 716 (batch 4 of epoch 178). [+][2018-08-15 16:21:12.890289] [PROGRESS] Training iteration 717 (batch 1 of epoch 179). [+][2018-08-15 16:21:12.912618] [PROGRESS] Training iteration 718 (batch 2 of epoch 179). [+][2018-08-15 16:21:12.932897] [PROGRESS] Training iteration 719 (batch 3 of epoch 179). [+][2018-08-15 16:21:12.958918] [PROGRESS] Training iteration 720 (batch 4 of epoch 179). [+][2018-08-15 16:21:13.028113] [PROGRESS] Training iteration 721 (batch 1 of epoch 180). [+][2018-08-15 16:21:13.049420] [PROGRESS] Training iteration 722 (batch 2 of epoch 180). [+][2018-08-15 16:21:13.072995] [PROGRESS] Training iteration 723 (batch 3 of epoch 180). [+][2018-08-15 16:21:13.097459] [PROGRESS] Training iteration 724 (batch 4 of epoch 180). [+][2018-08-15 16:21:13.161801] [PROGRESS] Training iteration 725 (batch 1 of epoch 181). [+][2018-08-15 16:21:13.183868] [PROGRESS] Training iteration 726 (batch 2 of epoch 181). [+][2018-08-15 16:21:13.207711] [PROGRESS] Training iteration 727 (batch 3 of epoch 181). [+][2018-08-15 16:21:13.231267] [PROGRESS] Training iteration 728 (batch 4 of epoch 181). [+][2018-08-15 16:21:13.292935] [PROGRESS] Training iteration 729 (batch 1 of epoch 182). [+][2018-08-15 16:21:13.313630] [PROGRESS] Training iteration 730 (batch 2 of epoch 182). [+][2018-08-15 16:21:13.334844] [PROGRESS] Training iteration 731 (batch 3 of epoch 182). [+][2018-08-15 16:21:13.355480] [PROGRESS] Training iteration 732 (batch 4 of epoch 182). [+][2018-08-15 16:21:13.417654] [PROGRESS] Training iteration 733 (batch 1 of epoch 183). [+][2018-08-15 16:21:13.438924] [PROGRESS] Training iteration 734 (batch 2 of epoch 183). [+][2018-08-15 16:21:13.460379] [PROGRESS] Training iteration 735 (batch 3 of epoch 183). [+][2018-08-15 16:21:13.481708] [PROGRESS] Training iteration 736 (batch 4 of epoch 183). [+][2018-08-15 16:21:13.543428] [PROGRESS] Training iteration 737 (batch 1 of epoch 184). [+][2018-08-15 16:21:13.564809] [PROGRESS] Training iteration 738 (batch 2 of epoch 184). [+][2018-08-15 16:21:13.588008] [PROGRESS] Training iteration 739 (batch 3 of epoch 184). [+][2018-08-15 16:21:13.608078] [PROGRESS] Training iteration 740 (batch 4 of epoch 184). [+][2018-08-15 16:21:13.672782] [PROGRESS] Training iteration 741 (batch 1 of epoch 185). [+][2018-08-15 16:21:13.692581] [PROGRESS] Training iteration 742 (batch 2 of epoch 185). [+][2018-08-15 16:21:13.713085] [PROGRESS] Training iteration 743 (batch 3 of epoch 185). [+][2018-08-15 16:21:13.735218] [PROGRESS] Training iteration 744 (batch 4 of epoch 185). [+][2018-08-15 16:21:13.796150] [PROGRESS] Training iteration 745 (batch 1 of epoch 186). [+][2018-08-15 16:21:13.816245] [PROGRESS] Training iteration 746 (batch 2 of epoch 186). [+][2018-08-15 16:21:13.838921] [PROGRESS] Training iteration 747 (batch 3 of epoch 186). [+][2018-08-15 16:21:13.858685] [PROGRESS] Training iteration 748 (batch 4 of epoch 186). [+][2018-08-15 16:21:13.922638] [PROGRESS] Training iteration 749 (batch 1 of epoch 187). [+][2018-08-15 16:21:13.941570] [PROGRESS] Training iteration 750 (batch 2 of epoch 187). [+][2018-08-15 16:21:13.961437] [PROGRESS] Training iteration 751 (batch 3 of epoch 187). [+][2018-08-15 16:21:13.983064] [PROGRESS] Training iteration 752 (batch 4 of epoch 187). [+][2018-08-15 16:21:14.049140] [PROGRESS] Training iteration 753 (batch 1 of epoch 188). [+][2018-08-15 16:21:14.071484] [PROGRESS] Training iteration 754 (batch 2 of epoch 188). [+][2018-08-15 16:21:14.093366] [PROGRESS] Training iteration 755 (batch 3 of epoch 188). [+][2018-08-15 16:21:14.121143] [PROGRESS] Training iteration 756 (batch 4 of epoch 188). [+][2018-08-15 16:21:14.182848] [PROGRESS] Training iteration 757 (batch 1 of epoch 189). [+][2018-08-15 16:21:14.205872] [PROGRESS] Training iteration 758 (batch 2 of epoch 189). [+][2018-08-15 16:21:14.226771] [PROGRESS] Training iteration 759 (batch 3 of epoch 189). [+][2018-08-15 16:21:14.247753] [PROGRESS] Training iteration 760 (batch 4 of epoch 189). [+][2018-08-15 16:21:14.312792] [PROGRESS] Training iteration 761 (batch 1 of epoch 190). [+][2018-08-15 16:21:14.333206] [PROGRESS] Training iteration 762 (batch 2 of epoch 190). [+][2018-08-15 16:21:14.353855] [PROGRESS] Training iteration 763 (batch 3 of epoch 190). [+][2018-08-15 16:21:14.378581] [PROGRESS] Training iteration 764 (batch 4 of epoch 190). [+][2018-08-15 16:21:14.442481] [PROGRESS] Training iteration 765 (batch 1 of epoch 191). [+][2018-08-15 16:21:14.461524] [PROGRESS] Training iteration 766 (batch 2 of epoch 191). [+][2018-08-15 16:21:14.482015] [PROGRESS] Training iteration 767 (batch 3 of epoch 191). [+][2018-08-15 16:21:14.502429] [PROGRESS] Training iteration 768 (batch 4 of epoch 191). [+][2018-08-15 16:21:14.562960] [PROGRESS] Training iteration 769 (batch 1 of epoch 192). [+][2018-08-15 16:21:14.582956] [PROGRESS] Training iteration 770 (batch 2 of epoch 192). [+][2018-08-15 16:21:14.603526] [PROGRESS] Training iteration 771 (batch 3 of epoch 192). [+][2018-08-15 16:21:14.624684] [PROGRESS] Training iteration 772 (batch 4 of epoch 192). [+][2018-08-15 16:21:14.688531] [PROGRESS] Training iteration 773 (batch 1 of epoch 193). [+][2018-08-15 16:21:14.708667] [PROGRESS] Training iteration 774 (batch 2 of epoch 193). [+][2018-08-15 16:21:14.731012] [PROGRESS] Training iteration 775 (batch 3 of epoch 193). [+][2018-08-15 16:21:14.752146] [PROGRESS] Training iteration 776 (batch 4 of epoch 193). [+][2018-08-15 16:21:14.815488] [PROGRESS] Training iteration 777 (batch 1 of epoch 194). [+][2018-08-15 16:21:14.834977] [PROGRESS] Training iteration 778 (batch 2 of epoch 194). [+][2018-08-15 16:21:14.859937] [PROGRESS] Training iteration 779 (batch 3 of epoch 194). [+][2018-08-15 16:21:14.878765] [PROGRESS] Training iteration 780 (batch 4 of epoch 194). [+][2018-08-15 16:21:14.940833] [PROGRESS] Training iteration 781 (batch 1 of epoch 195). [+][2018-08-15 16:21:14.963443] [PROGRESS] Training iteration 782 (batch 2 of epoch 195). [+][2018-08-15 16:21:14.987542] [PROGRESS] Training iteration 783 (batch 3 of epoch 195). [+][2018-08-15 16:21:15.011692] [PROGRESS] Training iteration 784 (batch 4 of epoch 195). [+][2018-08-15 16:21:15.073324] [PROGRESS] Training iteration 785 (batch 1 of epoch 196). [+][2018-08-15 16:21:15.097006] [PROGRESS] Training iteration 786 (batch 2 of epoch 196). [+][2018-08-15 16:21:15.119003] [PROGRESS] Training iteration 787 (batch 3 of epoch 196). [+][2018-08-15 16:21:15.142541] [PROGRESS] Training iteration 788 (batch 4 of epoch 196). [+][2018-08-15 16:21:15.206832] [PROGRESS] Training iteration 789 (batch 1 of epoch 197). [+][2018-08-15 16:21:15.226979] [PROGRESS] Training iteration 790 (batch 2 of epoch 197). [+][2018-08-15 16:21:15.250288] [PROGRESS] Training iteration 791 (batch 3 of epoch 197). [+][2018-08-15 16:21:15.273444] [PROGRESS] Training iteration 792 (batch 4 of epoch 197). [+][2018-08-15 16:21:15.339895] [PROGRESS] Training iteration 793 (batch 1 of epoch 198). [+][2018-08-15 16:21:15.361079] [PROGRESS] Training iteration 794 (batch 2 of epoch 198). [+][2018-08-15 16:21:15.385836] [PROGRESS] Training iteration 795 (batch 3 of epoch 198). [+][2018-08-15 16:21:15.405187] [PROGRESS] Training iteration 796 (batch 4 of epoch 198). [+][2018-08-15 16:21:15.469177] [PROGRESS] Training iteration 797 (batch 1 of epoch 199). [+][2018-08-15 16:21:15.488262] [PROGRESS] Training iteration 798 (batch 2 of epoch 199). [+][2018-08-15 16:21:15.507994] [PROGRESS] Training iteration 799 (batch 3 of epoch 199). [+][2018-08-15 16:21:15.531529] [PROGRESS] Training iteration 800 (batch 4 of epoch 199). [+][2018-08-15 16:21:15.593344] [INFO ] Breaking on request from callback. [+][2018-08-15 16:21:15.593698] [INFO ] Exceeded max number of iterations / epochs, breaking. do the training (with the same functions as before) .. code-block:: python predict(trainer=trainer, test_loader=test_loader) .. rst-class:: sphx-glr-horizontal * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_010.png :class: sphx-glr-multi-img * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_011.png :class: sphx-glr-multi-img * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_012.png :class: sphx-glr-multi-img * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_013.png :class: sphx-glr-multi-img * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_014.png :class: sphx-glr-multi-img * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_015.png :class: sphx-glr-multi-img * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_016.png :class: sphx-glr-multi-img * .. image:: /auto_examples/images/sphx_glr_plot_unet_tutorial_017.png :class: sphx-glr-multi-img **Total running time of the script:** ( 1 minutes 0.683 seconds) .. _sphx_glr_download_auto_examples_plot_unet_tutorial.py: .. only :: html .. container:: sphx-glr-footer :class: sphx-glr-footer-example .. container:: sphx-glr-download :download:`Download Python source code: plot_unet_tutorial.py ` .. container:: sphx-glr-download :download:`Download Jupyter notebook: plot_unet_tutorial.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_