We consider augmented training of ODE based neural networks, such as ResNets, to increase robustness with respect to adversarial attacks. This is done by adding a first order sensitivity term to the loss function, derived from the corresponding robust optimal control problem. To reduce memory cost in the training process, we take an optimize then discretize approach and compute the gradients via solving the adjoint sensitivity equation, which is of second order due to the modified loss function. The presented work is a collaboration with Enrique Zuazua.