Pitch prediction with AI in JavaScript

One of my favorite things to do is learning new things all the time, a couple of years ago I decided to lear how to play the violin! It's being a long journey that I've enjoyed so far.

One of the problems beginners have when learning to play the violin, is the fact that you never know if you are playing the right note, the violin doesn't have any kind of mark that you can use as a reference to position your fingers and play the right note.

It'd be really nice if you can get instant feedback on your playing, an app that listens to your playing and lets you know if you played the right note or not.

So, I decided to tackle this problem! And that's how I started Beat Master! A web app that uses the microphone for pitch prediction.

In this tutorial I'll show you how to do that with JavaScript! I was amazed to find out that it was not that difficult using AI.

Reading the microphone

The first thing is to get the input stream, this is what we are going to send to the tensor flow model to get the prediction on the pitch.

This is very straight forward, we can do this by using the navigator object like this.

navigator.mediaDevices
    .getUserMedia({ video: false, audio: true })
    .then((stream) => {
      micStream = stream // We are going to use it later
    })
    .catch((err) => {
      console.error(`you got an error: ${err}`);
    });

We are only asking permission to the mic, but we might want to ask permissions for the camera as well. This is a promise because the user needs to accept the permissions in order for us to have access to these devices.

Pitch prediction

In order to predict a sound, we can use a library called ml5js! This library has a lot of utilities that we can use to make our lives easier! There's one module for sound prediction, exactly what we need!

// Step 1
const audioContext = new AudioContext();
const MicStream = micStream

// Step 2
const pitch = ml5.pitchDetection(
  './model/',
  audioContext,
  MicStream,
  modelLoaded,
);

// Step 3
function modelLoaded() {
  console.log('Model Loaded!');
  getPitch()
}

// Step 4
const scale = ['C', 'C#', 'D', 'D#', 'E', 'F', 'F#', 'G', 'G#', 'A', 'A#', 'B'];
function getPitch() {
  pitch.getPitch(function(err, frequency) {
    if (frequency) {
      let midiNum = freqToMidi(frequency);
      currentNote = scale[midiNum % 12];
    }
    getPitch();
  })
}

In the first step we create the audio context and the stream that we already have, we are going to use these to predict the pitch.

In the step 2, we create the pitch prediction instance of the prediction model, underneath uses tensor flow. This requires the path to where the training data is located, the audio context, the stream and a callback that runs when the model is loaded.

In step 3, we call the pitch prediction function, this will be executed once the tensor flow model is ready to start predicting.

The last step, calls the getPitch method, this accepts a callback that receives the frequency of the sound, it basically returns the closes note to the given sound.

Additionally, the function calls itself again and again to keep predicting sounds.

Rendering the note

The prediction model returns a frequency, but we need to show a note, so people can read.

We need to map those frequencies to a human notation.

function freqToMidi(f) {
  const mathlog2 = Math.log(f / 440) / Math.log(2);
  const m = Math.round(12 * mathlog2) + 69;
  return m;
}

This function receives the frequency and converts it to a midi value, we will use this value to get the actual note.

And that's it! It was not really hard to start prediction pitch.

Beat Master

I've created a product based on this tutorial, people can browser different sheet music, I'll be uploading new music every week.

They will select a song, see the sheet music, start playing and the app will let them know if they are playing the right note or not.

Happy Codding!