Loop-Based Graphical Live-Coded Music in the Browser

Jason Gwartz and Nicolas Gold

Computing and programming education in schools has grown in prevalence, necessitating simple and engaging learning tools to teach coding to primary school students. These should be enjoyable to use, provide instantaneous feedback, and be widely available across different platforms (especially targeting low-cost hardware common in schools).

Immediacy of feedback is an important aspect of an educational learning environment, and in this respect, the practice of live coding is highly relevant. Live coding music is an artistic practice that has developed primarily in the last decade or so (see Magnusson’s 2014 survey [5]). Typically, it involves the production of music from algorithms coded live by a performer, often with the code being projected live for the audience to see; the code thus becomes an important part of the artistic outcome in conjunction with the music [5]. Live coding languages are variously graphical and textual (although visualisation is considered important even for textual languages - see McLean et al. [8]).

Several examples exist of textual browser-based music systems such as Gibber [11, 12] and extramuros [9]), despite McKinney’s claim that the direct Web Audio API is itself unwieldy for live coding [7]. The Sonic Pi project has clearly demonstrated the power of textual live coding as a good route to programming education for 10-16 year olds [2], and the success of MIT Scratch [6, 10] suggests that a graphically-driven live-coding environment for education would have potential for this age group and those in earlier years.

To combine a graphical, live-coding approach to programming education with musical motivation and wide, installation-free deployment requires the use of a platform that can offer a standardised interface to the user, network access to obtain the code, and audio production facilities: all characteristics found in modern browsers and the Web Audio API. In addition, web browsers are an attractive route to delivering live coding systems to educational institutions because of their ubiquity, relatively lightweight operational requirements, and universally understood interfaces. However, such environments pose particular challenges in respect of audio timing and scheduling.

This demonstration presents ‘Chopsticks’, a web-based, graphical programming environment for live coding music using phrase-level structures. Intended to teach aspects of programming, Chopsticks enables the creation of polyphonic music in real-time. The environment addresses the technical challenges of browser-based audio programming at the phrase-level, using a multi-clock solution to the problem of call-back scheduling to minimise latency.

Chopsticks encapsulates a programming language and runtime for use in early programming education, introducing some fundamental concepts such as selection (conditionals) and repetition (looping). In order to make the system usable on hardware typically available in schools, e.g. iPad or Chromebook devices, it is developed for and distributed in the browser as a web app, powered by the Web Audio API. The programming language is edited and manipulated largely using graphical symbols, inspired by MIT Scratch [6, 10]. The system produces musical output and draws on popular music’s inherent parallelism, enabling multiple simultaneous code paths to create polyphonic music.

The primary challenge in developing a real-time audio application using the Web Audio API is timing control. Chopsticks addresses this using a ‘mixed-time’ and ‘mixed-clock’ approach to scheduling audio events that utilises both the audio hardware and Javascript clocks. This permits musical-time (bars, beats, phrases) events in the interface to be linked to sufficiently-accurate scheduling of audio-time (samples, timestamps, delays) events. The phrase-based musical input, where the user can manipulate the upcoming phrase until the instant before it begins, leads to a requirement for low computational latency. Combined with the Web Audio API’s ‘write-once’ scheduling functionality, this irregular input creates unique audio scheduling problems.

Input is given in musical language, like ‘beats in a bar’, ‘bars in a phrase’, and ‘tempo’, which is translated for scheduling against the Web Audio API. To make these decisions, Chopsticks maintains a relationship between audio time and musical time. Phrases are recalculated once every four musical bars (eight seconds at a rate of 120 bpm). Any computation delay between the end of the outgoing phrase and beginning of the subsequent phrase must occur within a few milliseconds not to distort the rhythm.

The Chopsticks audio engine considers audio-time to be a first-class singleton entity which monotonically increases from instantiation, a philosophy inherited directly from the Web Audio API and its audio clock. The playback of audio therefore entails scheduling a sound sample for some time in the future, and playing it when that time is reached. The use of a time object in this way has previously been researched for musical or audio processing [3]. Phrase computation happens at a given audio-time, but can schedule and dispatch audio playback for a future time.

The Chopsticks interface uses a declarative wait-until approach, perhaps in the way that a musician might think: on beat 3 -> play. In Chopsticks’ phrase-oriented time structure, the user does not need to think about the sequence of events in terms of starting, stopping, and waiting; this is computed by Chopsticks’ audio processing. The user is thus free to focus explicitly on when in musical-time a sound will occur.

In addition to reasoning with two times (audio and musical), Chopsticks also leverages two clocks: the audio clock for playback scheduling, and the Javascript clock for temporal recursion. This use of temporal recursion is similar to implementations in ChucK [13] and Overtone [1]. In this browser implementation, the engine parses input and schedules audio playback for an audio-time in the future. This function also sets up a future callback against the JavaScript clock to retrigger future input parsing and audio scheduling.

The limitations of the JavaScript clock and its interaction with the audio clock via the Web Audio API make it difficult to guarantee temporal regularity. Wilson [14] illustrated the core issues facing audio scheduling in Web Audio API applications where two clocks exist, identifying that due to garbage collection and asynchronous tasks like HTTP requests, actions scheduled against the JavaScript clock can vary by tens of milliseconds.

During runtime, Chopsticks must periodically, once per musical phrase, recalculate the user input and schedule audio events. This recalculation is scheduling using temporal recursion, where the function that performs the audio dispatch schedules a callback of the same function for some time in the future (just before the next musical phrase). There are two potential problems at this phrase recalculation point: it could start either too late or too early. There are similarly two factors, related to Wilson’s [14] observations, that can cause an audible delay or jump at the turnover point of a musical phrase: the processing time to execute any necessary code for the following phrase could be too long; or the callback of the ‘setTimeout’ dispatch could occur at the wrong time.

Chopsticks solves this issue by setting a long callback time (on the order of seconds or tens of seconds instead of milliseconds), giving the JavaScript runtime extra opportunity to perform background tasks that would otherwise impact audio performance. This lessens the chance that any background tasks will need to occur at the expected callback time, and increases the likelihood that the musical scheduling callback can occur when desired. Delays are not likely to cause a callback to occur early, eliminating the problem of ‘jumps’; if the callback occurs less than 30ms late, it is unlikely to be musically perceived [4].

Additionally, with the Web Audio API, interference issues can occur in sample playback when two samples overlap - specifically when an instance of an OscillatorNode or AudioBufferSourceNode begins playing while another instance of the same source, either an oscillation pitch or a decoded audio file, is still playing. This interference can manifest as either a distortion of the specific duplicated audio, distortion of all playing audio, or an overwhelming interference noise. Chopsticks avoids this audio distortion resulting from such an overlap by inspecting the future state of the audio source at the time when it would be scheduled to start; this state is calculated based on the sum of the most recent previously scheduled playback time and the duration of the sound.

References

[1] S. Aaron and A. F. Blackwell. From sonic pi to overtone: Creative musical experiences with domain-specific and functional languages. In Proceedings of the First ACM SIGPLAN Workshop on Functional Art, Music, Modeling & Design, FARM ’13, pages 35–46, New York, NY, USA, 2013. ACM.

[2] P. Burnard, N. Brown, F. Florack, L. Major, Z. Lavicza, and A. Blackwell. Sonic Pi: Live & coding research report. http://www.sonicpiliveandcoding.com/s/research/_report/_dc/_02.pdf, November 2014. Accessed 30 January 2017.

[3] P. Donat-Bouillud, J.-L. Giavitto, A. Cont, N. Schmidt, and Y. Orlarey. Embedding native audio-processing in a score following system with quasi sample accuracy. In ICMC 2016 - 42th International Computer Music Conference, Utrecht, Netherlands, Sept. 2016.

[4] N. P. Lago and F. Kon. The quest for low latency. In Proceedings of the International Computer Music Conference, Miami, USA, pages 33–36, 2004.

[5] T. Magnusson. Herding cats: Observing live coding in the wild. Computer Music Journal, 38(1):8–16, 2014.

[6] D. J. Malan and H. H. Leitner. Scratch for budding computer scientists. In Proceedings of the 38th SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’07, pages 223–227, New York, NY, USA, 2007. ACM.

[7] C. McKinney. Quick live coding collaboration in the web browser. In Proceedings of the International Conference on New Interfaces for Musical Expression, London, UK, 2014.

[8] A. McLean, D. Griffiths, N. Collins, and G. Wiggins. Visualisation of Live Code. In Proceedings of Electronic Visualisation and the Arts London 2010, pages 26–30, 2010.

[9] D. Ogborn, E. Tsabary, I. Jarvis, A. Cardenas, and A. McLean. Extramuros: making music in a browser-based, language-neutral collaborative live coding environment. In Proceedings of the 1st International Conference on Live Coding, Leeds, UK, 2015.

[10] M. Resnick, J. Maloney, A. Monroy-Hernández, N. Rusk, E. Eastmond, K. Brennan, A. Millner, E. Rosenbaum, J. Silver, B. Silverman, and Y. Kafai. Scratch: Programming for all. Commun. ACM, 52(11):60–67, Nov. 2009.

[11] C. Roberts and J. Kuchera-Morin. Gibber: Live coding audio in the browser. In Proceedings of the International Computer Music Conference, Ljubljana, Slovenia, 2012.

[12] C. Roberts, M. Wright, and J. Kuchera-Morin. Music programming in Gibber. In Proceedings of the International Computer Music Conference, Denton, TX, USA, 2015.

[13] G. Wang, P. R. Cook, et al. Chuck: A concurrent, on-the-fly audio programming language. In Proceedings of International Computer Music Conference, Singapore, pages 219–226, 2003.

[14] C. Wilson. A tale of two clocks - scheduling web audio with precision. http://www.html5rocks.com/en/tutorials/audio/scheduling/. Accessed 26 January 2017.