In this talk, I am going to explain several approaches to explain the dynamics of neural networks. First, I will argue, why neural networks should always be viewed within the framework of dynamical systems. Then I am going to prove, how we may be able to derive mean-field differential equations for neural networks even in the intermediate/sparse coupling recurrent neural network regime. This construction involves network dynamics on graphops, which is a framework useful far beyond neural nets. Finally, I am going to indicate how to employ rigorous validated computation to prove dynamics of neural networks rigorously when pencil-and-paper methods fail.