Editing Antibody Design by AI (section)

== Training an ensemble of neural networks ==

We used '''six''' different architectures, '''five '''of which were'''convolutional neural networks''' with '''1 or 2 convolutional layers''' with filter size of 1, 3 or 5 residues and stride 1, followed by local max-pooling layer with window size 2 and stride 2. We used 64 and 32 convolutional filters for single convolutional layer networks. In one of the double convolutional layer networks, we used 32 filters with width 5 in first layer and 64 filters with width 5 in the second layer. In the other network, we used 8 convolutional filters with width 1 in first layer to learn an embedding from one-hot to hidden space for each amino acid, and then used 64 filters with width 5 to learn higher level patterns. In each of the convolutional neural network, the output from last convolutional layer was fed into a fully connected layer with 16 hidden units and a dropout layer. It is then connected to the final output layer where the loss function is evaluated. We also designed a 2- layer fully connected neural network with 32 hidden units and dropout in each layer. Table S1 illustrates the detailed setup of each architecture and an estimation of the number of parameters in each architecture.

All the neural networks were implemented using Keras 27. Training was conducted with respect to the binary cross entropy loss function in our classification task and the mean squared error loss in the regression tasks. For the multi-output regression networks, we used a masked mean squared error loss to deal with the missing values during training. Model hyperparameters were tuned by grid search using 10% of the training set as validation set and 5 epochs of quick training. The hyperparameters that yielded best validation loss were chosen for the training phase on the complete training set. We trained each neural network for 20 epochs using RMSProp optimization 28, and retained the learned network parameters from the epoch that produced best performance on the held-out test set.