bgd.nn module

This module contains the bgd.nn.NeuralStack class that represents a linear neural network (LNN).

Any other model of neural nets shall be written down here.

class bgd.nn.NeuralStack

Bases: object

Sequential model that can be viewed as a stack of layers. Backpropagation is simplified because the error gradient with respect to the parameters of any layer is the gradient of a composition of functions (defined by all successor layers) and is decomposed using the chain rule.

layers

List of layers, where layers[0] is the input layer and layers[-1] is the output layer.

Type

list

batch_op

Batching method to use in order to perform one step of the backpropagation algorithm.

Type

bgd.batch.Batching

cost_op

Error metric to use in order to evaluate model performance.

Type

bgd.cost.Cost

optimizer

Optimizer to use in order to update the parameters of the model during backpropagation.

Type

bgd.optimizers.Optimizer

activate_dropout()

Activates the Dropout layers (for training phase).

add(component)

Adds a component to the model: it can be either a layer or a special component like an optimizer. Some components are mandatory to train the model. The architecture of the model is defined by the layers that are provided to this method. Thus, the order is taken into account when adding layers. However, the order has no importance when adding special components like optimizers, error metrics, etc.

Parameters

component (object) – Component to add to the model (Layer or special component).

Raises

WrongComponentTypeError – If the type of component is not recognized.

check_components()

Verifies that all the components are set properly so that training is possible. If either the batch method, the optimizer or the error function is missing, an exception will be raised.

Raises

RequiredComponentError – If any component of the model has not been set.

deactivate_dropout()

Deactivates the Dropout layers (after training phase).

eval(X, start=0, stop=- 1)

Feeds a sample (or batch) to the model linearly from any layer to any layer. By default, X is propagated through the entire stack.

Parameters

• X (np.ndarray) – The sample (or batch of samples) to feed to the model.

• start (int) – Index of the layer where X is to be fed.

• stop (int) – Index of the last layer of propagation (-1 for last layer).

Returns

out:

Output of the propagation of X through each layer of the model.

Return type

np.ndarray

eval_loss(batch_y, predictions, alpha)

Returns the loss value of the output computed by the model with respect to the actual output.

Parameters

• batch_y (np.ndarray) – True labels of the samples.

• predictions (np.ndarray) – Labels predicted by the model.

• alpha (float) – L2 regularization alpha term (0 if no L2).

Returns

loss:

The value of the loss function.

Return type

float

get_accuracy(X, y)

Returns the accuracy of the provided dataset on the model.

Raises

AttributeError –

See bgd.ClassificationCost.accuracy().

train(X, y, epochs=1000, l2_alpha=0.1, print_every=1, validation_fraction=0.1, dataset_normalization=False)

Trains the model on samples X and labels y. Optimize the model parameters so that the loss is minimized on this dataset. The validation fraction must be in [0, 1] (0 to not evaluate the model on a validation set).

Parameters

• X (np.ndarray) – Array of samples. shape == (n_instances, n_features) or shape == (n_instances, n_pixels, n_channels) for images.

• y (np.ndarray) – Array of labels. shape == (n_instances,)

• epochs (int) – Number of times the dataset (X, y) is entirely fed to the model.

• l2_alpha (float) – L2 regularization alpha parameter (0 if no L2).

• print_every (int) – Number of batches between two prints of model state and evaluations on the intermediate validation set (negative number if none is wanted). Defaults to 1.

• validation_fraction (float) – Proportion of samples to be kept for validation.

Returns

errors:

array of the loss of each batch.

Return type

np.ndarray

Raises

• ValueError – see split_train_val().

• RequiredComponentError – see check_components().

Example

>>> X, y = load_digits(return_X_y=True)
>>> n_in, n_hidden, n_out = 64, 32, 10
>>> nn = NeuralStack()
>>> nn.add(FullyConnected(n_in, n_hidden))
>>> nn.add(Activation())
>>> nn.add(FullyConnected(n_hidden, n_out))
>>> nn.add(Activation('softmax'))
>>> nn.add(CrossEntropy())
>>> nn.add(AdamOptimizer())
>>> nn.add(SGDBatching(512))
>>> losses = nn.train(X, y)
Loss at epoch 49 (batch 2): 1.2941039726880612   - Validation accuracy: 83.333%
Loss at epoch 99 (batch 2): 0.764482839362229    - Validation accuracy: 96.111%
Loss at epoch 149 (batch 2): 0.5154527368075816   - Validation accuracy: 97.222%
Loss at epoch 199 (batch 2): 0.3979498104086121   - Validation accuracy: 97.778%
Loss at epoch 249 (batch 2): 0.32569535096131924  - Validation accuracy: 97.778%
Loss at epoch 299 (batch 2): 0.2748402661346476   - Validation accuracy: 97.778%
Loss at epoch 349 (batch 2): 0.2555267910622358   - Validation accuracy: 97.778%
Loss at epoch 399 (batch 2): 0.22863001357754167  - Validation accuracy: 97.778%
Loss at epoch 449 (batch 2): 0.22746067257186584  - Validation accuracy: 97.778%
Loss at epoch 499 (batch 2): 0.22220948073377536  - Validation accuracy: 97.778%
Loss at epoch 549 (batch 2): 0.2089401746378744   - Validation accuracy: 98.333%
Loss at epoch 599 (batch 2): 0.20035077161054704  - Validation accuracy: 98.333%
Loss at epoch 649 (batch 2): 0.1867468456143161   - Validation accuracy: 98.333%
Loss at epoch 699 (batch 2): 0.17347271604108705  - Validation accuracy: 98.333%
Loss at epoch 749 (batch 2): 0.15376433332896908  - Validation accuracy: 98.333%
Loss at epoch 799 (batch 2): 0.15436015140582668  - Validation accuracy: 98.333%
Loss at epoch 849 (batch 2): 0.13860411664942396  - Validation accuracy: 98.889%
Loss at epoch 899 (batch 2): 0.133165375570591    - Validation accuracy: 98.333%
Loss at epoch 949 (batch 2): 0.12890010110436428  - Validation accuracy: 98.333%
Loss at epoch 999 (batch 2): 0.12433454132628034  - Validation accuracy: 98.333%
>>> print('Loss over time:', losses)
Errors: [ 2.50539121  2.28391007  2.40779468 ...,  0.11655055  0.12436938  0.09155006]

bgd.nn.binarize_labels(y)

Transforms a N-valued vector of labels into a binary vector. If N classes are present, they must be 0 to N-1.

Parameters

y (np.ndarray) – Vector of classes.

Returns

binary_y:

Binary matrix of shape (n_instances, n_classes) s.t. binary_y[i,j] == 1 iff y[i] == j.

Return type

np.ndarray

Example

>>> y = np.array([0, 1, 0, 0, 1, 1, 0])
>>> binarize_labels(y)
array([[1, 0],
       [0, 1],
       [1, 0],
       [1, 0],
       [0, 1],
       [0, 1],
       [1, 0]])

bgd.nn.split_train_val(X, y, validation_fraction)

Splits randomly the dataset (X, y) into a training set and a validation set.

Parameters

• X (np.ndarray) – Matrix of samples. shape == (n_instances, n_features).

• y (np.ndarray) – Vector of expected outputs. shape == (n_instances,) or (n_instances, n_classes) if binarized.

• validation_fraction (float) – Proportion of samples to be kept for validation.

Returns

X_train (np.ndarray):

Matrix of training samples. len(X_train) == len(X) * (1 - validation_fraction)

y_train (np.ndarray):

Vector of training expected outputs. len(y_train) == len(y) * (1 - validation_fraction)

X_test (np.ndarray):

Matrix of test samples. len(X_test) == len(X) * validation_fraction

y_test (np.ndarray):

Vector of test expected outputs. len(y_test) == len(y) * validation_fraction

Raises

ValueError – If validation_fraction is not in [0, 1].