RNN Composition of Thematically Diverse Video Game Melodies | The Computer Games Journal
Skip to main content
Springer Nature Link
Log in

RNN Composition of Thematically Diverse Video Game Melodies

  • Published:
The Computer Games Journal

Abstract

Historically, creative action to the extent of composing music was exclusive to humans. However, machine learning is challenging prior paradigms by leading computers to the frontier of human-level intelligence. Integration between computers and the arts began in 1951 when Alan Turing transposed music for a computer to perform (National Computing Laboratory, Manchester, 1951). Recently, considerable progress has been made due to breakthroughs in both the graphics processing unit and deep learning in the early 2000s. Most efforts emphasize melodic and harmonic styles (Hadjeres G, Pachet F, Nielsen F, in Deepbach: a steerable model for bach chorales generation, arXiv preprint arXiv:1612.01010, 2016; Huang A, Raymond W, in Deep learning for music, arXiv preprint arXiv:1606.04930, 2016), but there is little progress in thematic variation within a genre. By utilizing Google’s Magenta, I trained a deep recurrent network to model strongly thematic melodies by using a diverse corpus of video game music. This research challenges the capabilities of recurrent neural networks since throughout a game many themes are presented, ranging from war, defeat, and victory, to love and death. The aesthetics of music are ultimately subjective; nevertheless, my analysis, to mitigate sociocultural biases in music, evaluates generated songs on modulation, strength of rhythm, and repetition. This deep sequence-based model, trained on 2933 video game melodies, recognized unique motifs and composed compelling and distinct themes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. Appendix 4, 2.1.1.

  2. “Video Game Music Archive.” VGMusic—31,684 Game Music MIDI Files, Shane Evans, www.vgmusic.com/. Accessed 3 June 2017.

  3. Adams, Tarn. Procedural Generation in Game Design. Productivity Press, 2017.

  4. Plans, David and Morelli, Davide. (2012). Experience-Driven Procedural Music Generation for Games. Computational Intelligence and AI in Games, IEEE Transactions on. 4. 192–198. https://doi.org/10.1109/tciaig.2012.2212899.

  5. Pythagoras. “Musica Universalis.” Harmony of the Spheres, 6th Century BCE.

  6. Euclid. “Στοιχεῖα Stoicheia.” The Elements, 300 BCE.

  7. Ivakhnenko, Alekseĭ Grigorʹevich, and Valentin Grigorévich Lapa. Cybernetic predicting devices. CCM Information Corporation, 1965.

  8. Eck, Douglas. “Magenta.” Magenta, Google, 2016 www.magenta.tensorflow.org/. Accessed 27 Aug. 2017.

  9. Pachet, François. “AI Music-Making.” Flow Machines, Sony CSL, 19 Sept. 2016, www.flow-machines.com/. Accessed 20 Aug. 2017.

  10. Roberts, Adam. “Tensorflow/Magenta.” GitHub, Google, 9 June 2017, www.github.com/tensorflow/magenta/blob/master/magenta/models/melody_rnn/melody_rnn_model.py. Accessed 29 June 2017.

  11. Appendix 4, 3.3.1.

  12. Appendix 1: Video Game MIDIs.

  13. Hawthorne, Curtis. “Tensorflow/Magenta.” GitHub, Google, 30 Oct. 2017, www.github.com/tensorflow/magenta/blob/master/magenta/scripts/convert_dir_to_note_sequences.py. Accessed 5 Nov. 2017.

  14. Simon, Ian. “Tensorflow/Magenta.” GitHub, Google, 19 June 2017, www.github.com/tensorflow/magenta/blob/master/magenta/models/melody_rnn/melody_rnn_create_datset.py. Accessed 22 June 2017.

  15. Appendix 3, Song 4 (Note Breakdown).

  16. Turner, J. R., Carroll, D. and Courtney, H. (1983), Cardiac and Metabolic Responses to “Space Invaders”: An Instance of Metabolically-Exaggerated Cardiac Adjustment? Psychophysiology, 20: 544–549. https://doi.org/10.1111/j.1469-8986.1983.tb03010.x.

  17. Churchill, Spencer L. “spencerchurchill/ee.” GitHub, 10 Sept. 2017, www.github.com/spencerchurchill/ee/midi. Accessed 4 June 2017.

  18. Appendix 2, MIDI Size Sorter.

  19. Appendix 4, 3.1.1.

  20. Appendix 4, 3.2.1.

  21. Roberts, Adam. “Tensorflow/Magenta.” GitHub, Google, 23 May 2017, www.github.com/tensorflow/magenta/blob/master/magenta/models/melody_rnn/melody_rnn_generate.py. Accessed 22 June 2017.

  22. Mao, Huanru Henry, Taylor Shin, and Garrison Cottrell. "DeepJ: Style-specific music generation." Semantic Computing (ICSC), 2018 IEEE 12th International Conference on. IEEE, 2018.

  23. DeMaria, Rusel, and Johnny L. Wilson. High Score!: the Illustrated History of Electronic Games. 1, McGraw-Hill/Osborne, 2004. ISBN 0-07-222428-2.

  24. Van Den Oord, Aaron, et al. "Wavenet: A generative model for raw audio." arXiv preprint arXiv:1609.03499 (2016).

  25. Simon, Ian. “Tensorflow/Magenta.” GitHub, Google, 17 Nov. 2017, www.github.com/tensorflow/magenta/blob/master/magenta/models/melody_rnn/melody_rnn_train.py. Accessed 5 July 2017.

Author information

Authors and Affiliations

  1. Irvine Valley College, 5500 Irvine Center Drive, Irvine, CA, 92618, USA

    Spencer Lee Churchill

Authors
  1. Spencer Lee Churchill
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Spencer Lee Churchill.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Appendices

Agnello, Anthony John. “The 25 Greatest Video Game Soundtracks of All Time.” GamesRadar+, 29 June 2016, www.gamesradar.com/the-25-greatest-video-game-soundtracks-of-all-time/. Accessed 2 June 2017.

Fitzpatrick, Alex. “Best Video Games of All Time: TIME's Top 50.” Time, Time, 23 Aug. 2016, www.time.com/4458554/best-video-games-all-time/. Accessed 30 May 2017.

Appendix 1: Video Game MIDIs ,

Agnello, Anthony John. “The 25 Greatest Video Game Soundtracks of All Time.” GamesRadar+, 29 June 2016, www.gamesradar.com/the-25-greatest-video-game-soundtracks-of-all-time/. Accessed 2 June 2017.

Fitzpatrick, Alex. “Best Video Games of All Time: TIME's Top 50.” Time, Time, 23 Aug. 2016, www.time.com/4458554/best-video-games-all-time/. Accessed 30 May 2017.

Tetris

Pokémon

Metroid

Mega Man

Kirby

Halo

Mass Effect

Galaga

Fire Emblem

Elder Scrolls

Street Fighter

Final Fantasy

Castlevania

Donkey King

Baldur’s Gate

Kingdom Hearts

Professor Layton

Super Mario Bros

The Legend of Zelda

Metal Gear Solid

Sonic the Hedgehog

Chrono Trigger

Advance Wars

 

Appendix 2: Data Preparation

figure a

Appendix 3: Midi Analysis

3.1 Song 1

Note Breakdown

figure b

Sheet Music ~ Pleasant

figure c

3.2 Song 2

Note Breakdown

figure d

Sheet Music ~ Spooky

figure e

3.3 Song 3

Note Breakdown

figure f

Sheet Music ~ Funky

figure g

3.4 Song 4

Note Breakdown

figure h

Sheet Music ~ Frenetic

figure i

Appendix 4: Outlines

  1. 1.

    Computer Google Virtual Machine (Training Model ~ 6 Days):

    1. 1.1.

      CPU:

      1. 1.1.1.

        (8) Intel Xeon Processor E5-2660

    2. 1.2.

      RAM:

      1. 1.2.1.

        (13) 4 Gigabyte DDR4 RAM

    3. 1.3.

      OS:

      1. 1.3.1.

        Linux − Debian 8

  2. 2.

    Libraries

    1. 2.1.

      Python 2.7:

      1. 2.1.1.

        Python-pip: matplotlib, scipy, bokeh, IPython, pandas

    2. 2.2.

      Google:

      1. 2.2.1.

        Tensorflow

      2. 2.2.2.

        Magenta

      3. 2.2.3.

        Bazel

  3. 3.

    Commands

    1. 3.1.

      Note Sequence:

      1. 3.1.1.

        ./bazel-bin/magenta/scripts/convert_dir_to_note_sequences --input_dir=~/ee/midi/ --output_file=~/ee/res/notesequences.tfrecord --num_threads=64 --log=INFO

    2. 3.2.

      Dataset:

      1. 3.2.1.

        ./bazel-bin/magenta/melody_rnn_create_dataset --config=attention_rnn --input=~/ee/res/notesequences.tfrecord --output_dir=~/ee/res/out/ --eval_ratio=0.01

    3. 3.3.

      TrainFootnote 27:

      1. 3.3.1.

        ./bazel-bin/magenta/train --config=attention_rnn --run_dir=~/ee/res/rundir/ --sequence_example_file=~/ee/res/out/training_melodies.tfrecord --hparams=“batch_size=512,rnn_layer_sizes=[24,13,13,24]”

    4. 3.4.

      Generate:

      1. 3.4.1.

        ./bazel-bin/magenta/generate --config=attention_rnn --run_dir=~/ee/res/rundir/ --output_dir=~/ee/res/generated/ --num_outputs=10 --num_steps=480 --hparams=“batch_size=512,rnn_layer_sizes=[24,13,13,24]” --primer_melody=“[]”

Appendix 5: Sound Files

  1. 1.

    Song 1 (Audio ~ Pleasant)

    1. a.

      www.soundcloud.com/splch/growing?in=splch/sets/magenta-video-game-music

  2. 2.

    Song 2 (Audio ~ Spooky)

    1. a.

      www.soundcloud.com/splch/spooky?in=splch/sets/magenta-video-game-music

  3. 3.

    Song 3 (Audio ~ Funky)

    1. a.

      www.soundcloud.com/splch/technocafe?in=splch/sets/magenta-video-game-music

  4. 4.

    Song 4 (Audio ~ Frenetic)

    1. a.

      www.soundcloud.com/splch/mash?in=splch/sets/magenta-video-game-music

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Churchill, S.L. RNN Composition of Thematically Diverse Video Game Melodies. Comput Game J 8, 41–58 (2019). https://doi.org/10.1007/s40869-018-0063-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40869-018-0063-x

Keywords

Search

Navigation