What is Deep Learning?

Most of the popular approaches to machine learning are 'shallow' in the sense that they use a small number of layers of internal representations of their input to achieve their goals. These work great for most problems, and many approaches are theoretically capable of learning any function of their input if they have enough capacity. However, 'enough' capacity usually means a number of parameters that is simply infeasible with current technologies and algorithms. Such shallow models also have difficulty learning multiple patterns of differing spatial and temporal scales. Deep learning attempts to rectify these problems by using models that make use of many layers of representations, allowing them to capture hierarchical features in the input data naturally. Additionally, their deeply layered structure allows them to achieve the expressiveness of a much larger shallow model while using fewer parameters. Deep learning is not without its own challenges though. Models with many layers prove a challenging problem for traditional learning algorithms and require new approaches. How to train these models is still an open problem, although much progress has been made in the past few years.