Every minute of the day, users share hundreds of thousands of pictures, videos, tweets, reviews, and blog posts. More than ever before, we have access to massive datasets in almost every area of science and engineering, including genomics, robotics, and astronomy. These datasets provide unprecedented opportunities to automatically discover rich statistical structure, from which we can derive new scientific discoveries. Gaussian processes are flexible distributions over functions, which can learn interpretable structure through covariance kernels. In this talk, I introduce a Gaussian process framework which is capable of learning expressive kernel functions on massive datasets. I will show how this framework generalizes a wide family of scalable machine learning approaches, leverages the inductive biases of deep learning models, and allows one to exploit model structure for significant further gains in scalability and accuracy, without requiring severe assumptions. I will then discuss how we can use this framework for reverse engineering human learning biases, crime prediction using point processes, image inpainting, video extrapolation, modelling change points and the impacts of vaccine introduction, and discovering the structure and evolution of stars.