2012 marked a pivotal milestone in the field of neural networks. The intersection of general purpose computing using Graphics Processing Units (GPUs), labelled big datasets, and very large neural networks (called deep neural networks) enabled a break-through in machine learning that has led to impressive results in many fields and applications, such as self-driving vehicles and real-time language translation. Recently, the advances offered by these techniques have been applied to the areas of music and speech synthesis, which have opened up exciting new areas of applications.