trainable flag does not work for batch normalization layer

[nes123]

I work with keras 1.0.2 with tensorflow.
I try to freeze some layers in the network, it works well for convolutions and FC but not for the batch normalization layer. I print the weights of the layer before and after one epoch and I see a changes. any ideas ?

[fchollet]

Please reformulate your question in a clearer fashion.

[jaekyeom]

I think I'm experiencing the same issue.
My Keras is 1.2.1 and Tensorflow is 0.12.1.
I spent some time wondering why some weights are changing even if I set trainable=False for all models before compiling, and then came here.
If I eliminate the BatchNormalization layers, they stay the same.

[jaekyeom]

As it seems like a problem with calling self.add_update(), not using mode=0 could be a temporary workaround.

[scott-vsi]

You may need to set the learning phase to testing (0): e.g.,

from keras import backend as K
K.set_learning_phase(0)  # all new operations will be in test mode from now on

per https://blog.keras.io/keras-as-a-simplified-interface-to-tensorflow-tutorial.html

[jaekyeom]

Actually, what I wanted was to prevent some part of the network from being trained, and the part contained BatchNormalization layers.

[Tokukawa]

A little bit more of context

from keras.layers import normalization
from keras.models import Sequential
import numpy as np

model0 = Sequential()
norm_m0 = normalization.BatchNormalization(input_shape=(10,), momentum=0.8)
model0.add(norm_m0)

model0.summary()

model1 = Sequential()
norm_m1 = normalization.BatchNormalization(input_shape=(10,), momentum=0.8)
model1.add(norm_m1)
for layer in model1.layers:
    layer.trainable = False
model1.compile(loss='mse', optimizer='sgd')

print("Shape batch normalization: {}.".format(len(model0.layers[-1].get_weights())))
print("Before training")
print([np.array_equal(w0, w1)for w0,w1 in zip(model0.layers[-1].get_weights(), model1.layers[-1].get_weights())])

X = np.random.normal(loc=5.0, scale=10.0, size=(1000, 10))
model1.fit(X, X, nb_epoch=4, verbose=0)

print("After training")
print([np.array_equal(w0,w1)for w0,w1 in zip(model0.layers[-1].get_weights(), model1.layers[-1].get_weights())])

Shape batch normalization: 4.
Before training
[True, True, True, True]
After training
[True, True, False, False]

We instantiate two models.
Before the training all the weights of the two models are equal. After the training of the second model with all the layers frozen, the weights are different.
However if you change the default mode from 0, you get the correct beaviour

from keras.layers import normalization
from keras.models import Sequential
import numpy as np

model0 = Sequential()
norm_m0 = normalization.BatchNormalization(mode=1, input_shape=(10,), momentum=0.8)
model0.add(norm_m0)

model0.summary()

model1 = Sequential()
norm_m1 = normalization.BatchNormalization(mode=1, input_shape=(10,), momentum=0.8)
model1.add(norm_m1)

for layer in model1.layers:
    layer.trainable = False

model1.compile(loss='mse', optimizer='sgd')


print("Shape batch normalization: {}.".format(len(model0.layers[-1].get_weights())))
print("Before training")
print([np.array_equal(w0, w1)for w0,w1 in zip(model0.layers[-1].get_weights(), model1.layers[-1].get_weights())])

X = np.random.normal(loc=5.0, scale=10.0, size=(1000, 10))
model1.fit(X, X, nb_epoch=4, verbose=0)

print("After training")
print([np.array_equal(w0,w1)for w0,w1 in zip(model0.layers[-1].get_weights(), model1.layers[-1].get_weights())])

Shape batch normalization: 4.
Before training
[True, True, True, True]
After training
[True, True, True, True]

[thematrixduo]

Same here, I tried to train a DCGAN model and found that freezing the discriminator with trainable=False only works if the discriminator does not contain any batch norm layer. If discriminator has batch norm layer, the output of the model on the same data input is changed even when dsicriminator.trainable=False. Hope there can be a fix :D @fchollet

[litesaber15]

I am a little confused about freezing BatchNormalization. It has gamma and beta parameters that are initialized with 1s and 0s respectively by default, and they're also trainable by default. How to freeze these? The API documentation and source code both say that BatchNormalization does not have trainable as a parameter.

However, weirdly, if I do pass trainable=False in BatchNormalization(), the number of trainable parameters drops to 0. What am I missing here?

[redsphinx]

@Tokukawa I am also having this issue. I just tried setting mode=1 and it gives the following error: TypeError: The mode argument of BatchNormalization no longer exists. mode=1 and mode=2 are no longer supported.

[Tokukawa]

@redsphinx In keras 2 BatchNormalization has been rewritten from scratch. I didn't tried it yet,but I guess this issue is gone now.

[milsto]

Passing trainable=False in BatchNormalization() will freeze the layer parameters in Keras 2 (tested with Keras 2.0.2). Seams that @Tokukawa is right, and I think this issue can be closed.

[waleedka]

I believe it's still broken. This code uses Keras 2.0.3 and shows the problem.

import keras
from keras.layers.normalization import BatchNormalization
from keras.models import Sequential
import numpy as np

print("Version: ", keras.__version__)

# Basic model
model = Sequential()
model.add(BatchNormalization(input_shape=(2,)))
model.compile(loss='mse', optimizer='adam')

# Print weights and predictions before training.
X = np.random.normal(size=(1, 2))
print("Prediction before training: ", model.predict(X))
print("Weights before training: ", [[list(w) for w in l.get_weights()] for l in model.layers])


# Train on random output, but set all layers to Trainable=False
Y = np.random.normal(size=(1, 2))
for l in model.layers:
    l.trainable = False
model.fit(X, Y, verbose=0, epochs=10)

print("\n\nPrediction after training: ", model.predict(X))
print("Weights after training: ", [[list(w) for w in l.get_weights()] for l in model.layers])

Output:

Version:  2.0.3
Prediction before training:  [[ 0.99468702 -0.22452217]]
Weights before training:  [[[1.0, 1.0], [0.0, 0.0], [0.0, 0.0], [1.0, 1.0]]]


Prediction after training:  [[ 0.93589312 -0.2035163 ]]
Weights after training:  [[[1.0, 1.0], [-0.0099943792, 0.0099907704], [0.095157444, -0.021479076], [0.90438205, 0.90438205]]]

As this shows, despite setting trainable = False, the weights and the output of the model are changed after training. If I'm missing something, I'd appreciate the hints.

[fchollet]

That's expected behavior.

despite setting trainable = False, the weights and the output of the model are changed after training

Trainable weights do not change. But batchnorm also maintains non-trainable weights, which are updated via layer updates (i.e. not through backprop): the mean and variance vectors.

[waleedka]

That's expected behavior.

@fchollet So what's the correct way to freeze a batch normalization layer? As in, to freeze both the trainable and non-trainable weights?