We consider the task of learning dynamical systems from data in a system-agnostic framework. This presentation is divided into two parts. First, we utilize a simulation-based approach to investigate algorithms for learning chaotic Ordinary Differential Equations (ODEs). We demonstrate that for noise-free data and low-dimensional systems, this task is effectively solved, as polynomial regression-based methods can achieve machine-precision forecasts. However, we show that observational noise remains a significant challenge for most algorithms. In the second part, we address this challenge by developing nonparametric statistical theory for learning ODEs from noisy observations. Specifically, we establish minimax optimal error rates for two contrasting observational models: the Stubble model, consisting of many short trajectories, and the Snake model, consisting of a single long trajectory. We conclude by discussing challenges at the intersection of dynamical systems and statistical learning.