A First Look at Music Composition using LSTM

A First Look at Music Composition using LSTM Recurrent Neural Networks Authors: Douglas Eck Jürgen Schmidhuber By Tuvy Lemberg

Outline • • Introduction Very short background on LSTM Data Representation Training Data Experiment 1 Experiment 2 Discussion

Introduction • Can a computer compose better music than Mozart or Mati Kaspy? • Very likely in the future, but not in this research. • This research shows that by using a Long Short -Term Memory (LSTM) network, it is possible to improvise a form of blues music (bebop blues). • The compositions are convincing and the method is a step forward in music composition using a neural network.

Introduction-background • One straightforward way of composition using neural network is by using a recurrent neural network (RNN) to predict the next notes, at time t+1, using the notes at time t. • The predicted notes will be used as an input to the network at time t+2, etc. • This method is implemented using RNN in several works (Stevens and Wiles, 1994, Mozer, 1994, etc). • The music was lacking thematic structure, minimal phrase structure and rhythmic organization.

Introduction-background • The reason for this failure is linked to the problem of vanishing (and exploding) gradients in RNN. • Making it impossible for the network to deal with long-term dependencies that are vital for producing music (for instance long chords).

Introduction-background • The solution to the problems encountered by the RNN network is using an LSTM network. • LSTM has succeeded in domains where other RNNs have failed, such as timing counting, etc.

Very short background on LSTM

Data Representation • How can we represent music? – The notes are represented by a vector. – One note is on or off, represented by 1 or 0 respectively. – Each vector represents a slice of time. – 12 notes for chords and 13 for melodies.

Training Data • A form of 12 -bar blues, popular among bebop jazz musicians, is used. • 8 time slices per bar. A single song length of 96 time slices. • The chords used did not vary from song to song.

Training Data • Experiment 1 - Only the chords were presented. • Experiment 2 - A single melody line was presented in addition to the chords. The melodies constructed by concatenating segments of music written by the authors. Datasets were constructed by choosing randomly from 4096 unique complete pieces.

Experiment 1 - Learning Chords • Examines LSTM ability to learn to reproduce the musical chord structure (in this case the chords are independent of the melody, and provides a baseline).

Ex 1 - Network Topology and Experimental Parameters • In the network four cell blocks containing 2 cells each are fully connected to each other and to the input layer. The output layer is fully connected to all cells and to the input layer.

Ex 1 - Network Topology and Experimental Parameters • Forget, input and output gates biases are set at -0. 5, -1. 0, -1. 5 and -2. 0. • Learning rate was set at. 00001. • Chord notes were predicted using a decision threshold of 0. 5. • Learning is done by gradient descent method.

Ex 1 - Training and Testing • The network was trained using the cross-entropy loss function. • Learning continued until the network successfully predicted the entire chord sequence.

Ex 1 - Result • The network easily handled this task. • Success is not surprising because LSTM networks known to be successful in such tasks.

Experiment 2 - Learning Melody and Chords • Both melody and chords are learned. • Learning continues until chord sequence is learned and cross-entropy error is low. • Not all training data used (amount of training data is astronomic). • The goal is to see if the network could learn the musical structure for composition.

Ex 2 - Network Topology and Experimental Parameters • Some cell blocks processed chords and other cell blocks processed melodies. Eight cell blocks containing 2 cells each are used. • Four cell blocks are fully connected to the input chords. • Four cell blocks are fully connected to the input melody. • Chord cell blocks have recurrent connections to themselves and to the melody cell blocks. • Melody cell blocks are only recurrently connected to melody cell blocks. • Notes for melody were predicted using softmax. • Other parameters are identical to those described in previous network.

Ex 2 – Results • LSTM composed music in the form of blues. • The network successfully composed music. – Reproducing chord structure while improvising new melodies. • It is remarkably better sounding than a random walk across the pentatonic scale.

Discussion • These experiments were successful. • The study is a cornerstone in DNN music composition. • Later experiments confirmed the ability of the LSTM network to generate different kinds of music.

Discussion • In some sense, the study is not satisfying: • The melodies may be identical to the melodies in the training set. • The musical format is very restricted – In reality many kinds of music are less restricted. – Later experiments used music generators with less restrictions. • No evaluation of musical quality.

Recent studies • Using LSTM Recurrent Neural Networks for Music Generation: – https: //youtu. be/a. Sr 8_QQYp. YM? t=4 m • Wave. Net - generating raw audio (other RNN): – https: //deepmind. com/blog/wavenet-generativemodel-raw-audio/

Questions