In this talk we characterize local parameter symmetries that occur in machine learning models obtained from Neural Ordinary Differential Equations (Neural ODEs) governed by ReLU Neural Networks. If restricted to a sufficiently well-behaved input space, such a model can be formalized as a realization map from a Euclidean parameter space into an $L^2$ function space. At generic network parameters, this allows for an application of structural statements from functional analysis, like the Constant Rank Theorem, to express the local symmetries in terms of the Fréchet derivative of the realization map. I will demonstrate how, despite the lack of smoothness of the ReLU architecture, methods from subanalytic geometry can be employed to establish structural regularity guarantees, which then yield a precise notion of local parametric redundancy. The obtained results can, for example, be used to justify a minimal dimensional reparametrization of the realization map near generic parameters.