Automatic Drum Velocity Control Considering the User’s Piano Performance

Automatic Drum Velocity Control Considering the User’s Piano Performance Shinnosuke Seki and Tetsuro Kitahara Nihon University, Japan Twitter: @tetsurokitahara

Introduction Drum machines/rhythm machines are useful Roland TR-808 KORG Volca Beats BOSS DR-3 You can play band music even if you cannot find a drummer But these don’t play expressive drum performance (All pictures taken from Wikipedia)

What are expressive performances? Controlling timing and intensity in drum performances elaborately to avoid machine-like performances Here we focus on intensity (i.e. MIDI velocity) Example No expression Expressive Related works Wright et al. (2006): Learning micro-timing with ML Tidemann et al. (2007): Learning human “groove” with HMM etc. Gillick et al. (2019): Generating expression with LSTM-based seq2seq

Our perspective: Interaction with humans Related works don’t consider mutual interaction with human players velocity Global velocity changes should be shared and synchronized velocity Piano (human) time Drums (system) time

Our system measure m-2 measure m-1 measure m measure m+1 Predict Piano (human) Reflect Drums (system) Predict When human players change the velocity, the system’s velocity also changes accordingly

Demo Sorry, the difference between the sounds with high and low velocity is unclear

Method velocity + mean 0 = actual velocity time Global velocity change Local velocity change measure-wise mean of velocity Deviation from global velocity change measure m-2 measure m-1 measure m measure m+1 Predict Piano (human) ×α Reflect Δv(i)m,n ～ N(μ(i)c, σ(i)c2) Randomly determined following a normal distribution (μ and σ2 are learned with data) Drums (system) ×(1-α) Predict Linear regression

Preliminary investigation Do human players perform correlatedly to each others in velocity? Jazz club students played piano+drums duo Performances were recorded as MIDI data Some had moderate correlation Measure-wise mean of velocity Piano Drums

Experiment We asked an amateur jazz pianist to play the piano with our system Questions Pieces Q1 Was the global velocity change for the drums natural? Q2 Was the local velocity change for the drums natural? Q3 Did you feel that the drums’ velocities follow yours? • Bags Groove • Cantaloupe Islands • Now’s The Time Q4 Were the drums’ velocities following the piano’s velocities natural? Q5 Did you enjoy the performance? Results Scale: 1-7

Recorded video PC’s display

Discussion • Prediction with linear regression • Impossible to predict sudden changes (e.g. those at section boundaries) • If N (the window length for linear regression) is smaller, the drums (system) may follow the piano (human) more quickly • Can we use other cues like gestures? • Our assumption • We assume that global velocity changes should be shared • But is this true? • In bass solos, bass’s intensity is high while other instruments’ one is low

Conclusion • We developed an automatic drumming system in which the velocity automatically changes according to the user’s performance • Basic mechanism: • Global velocity changes + local velocity changes • Global velocity: determined from the piano’s velocity changes predicted with linear regression • Local velocity: randomly determined following a normal distribution • Preliminary experiment: • A participant found the drums’ velocity changes following his own performances • Many issues remain