Tavim Blog Open Tavim

Tavim · how it works

You recorded it on your phone. Here is how to hear it properly.

Tavim does not clean a bad recording. It takes it apart into what was played and how it was played, and then plays it again.

Everyone has one. A recital filmed from row twelve. A friend at the piano in a bar. Your own practice take, phone propped on the music stand. The playing was good. The recording is a small microphone far from a piano: thin sound, a booming room, a cough in the quiet bar.

Why cleaning does not work

Filters can lower hiss. They cannot bring back what the microphone never caught. The sound of a phone at the back of a room is not noise lying on top of the music. It is the music, as that phone heard it. So we stopped trying to repair recordings and asked a different question: what does a recording contain?

Three layers

  1. What was played. The notes, in their bars, for each hand. The score.
  2. How it was played. When each note really fell, how hard, how long it was held, where the pedal went. The performance.
  3. What it was played on, and where. The piano and the room. The sound.

The test for each is simple. Would it appear in a printed score? Then it is layer one. Would another pianist, reading the same score, do it differently? Layer two. Would it change if the same pianist played on another instrument in another room? Layer three.

A phone ruins the third layer. The first two are still inside that thin recording, complete, if something can hear them.

1 · WHAT WAS PLAYEDThe score 2 · HOW IT WAS PLAYEDThe performance 3 · ON WHAT, AND WHEREThe sound notes in their bars each note: early or late, and how hard the piano and the room
What a recording holds. A phone damages only the third layer.

One model, two answers for every note

Most transcribers answer one question: which note sounded, and at what second. The result is a list of events. It is not a score, because nothing is in a bar. It is not a clean performance either, because nothing says what the player was departing from.

We trained a model to answer two questions about every note at once. Where is this note written: which bar, which beat, which fraction of the beat. When was it played: the moment it sounded, how hard it was struck, when the key came up, where the pedal was. It writes a score in which every note carries its own time stamp.

It listens in 20-second windows and is never told where the beat is. It has to find the beat, lose it in a ritardando, and find it again, as a listener does.

the first page of the score Tavim wrote from that recording.

Keeping the two apart

The hard part is the border. A late note can be a note written late or a note played late. Put it on the wrong side and both layers are spoiled: the score becomes unreadable and the performance becomes flat.

So the two are worked out together. Tavim proposes a grid of bars and beats, then measures how far each note sits from it. If the notes scatter evenly around the grid, early and late, the grid is right and the scatter is the player. If they lean one way, the grid is wrong, and it is drawn again. One guard keeps this honest: of two ways to write a passage, the simpler one wins. A rubato must never be written out as triplets.

What the performance layer holds

For every note: how early or late it fell against its written place, to about 3 thousandths of a second; how hard it was struck; how long it was held against its written length. For the piece: the tempo as it breathes, and every press and release of the pedal.

WRITTEN PLAYED 12 ms early28 ms late6 ms late4 ms early41 ms late how hard
How one bar is kept: the written place of each note, and how far the player was from it. The values are an example.

The proof

Put the two layers back together and you must get the original performance, note for note. We test exactly that. The replay lines up with the recording, and that is also why Tavim's notes can light at the instant you hear them.

And that is how we built a piano

Once Tavim could take any piano recording apart, we ran it across a very large collection of real piano recordings. Each recording gave us three things together: the exact notes, the exact performance, and the real sound of that instrument. Across many different pianos, that is the material a neural piano needs, and it did not exist.

A second model learned the journey in reverse. Give it notes, a performance, and 15 seconds of any piano, and it produces that piano playing that performance. It has no samples. It learned what pianos do from pianos.

What you get

Your recording, played again. The same notes, every ritardando where the player put it, the same touch and pedal, in the sound of the instrument without the phone. Or on another piano: a bright Steinway, a Yamaha concert grand, a dark Baldwin. With it come the score, the MIDI, and a video with every note lit as it sounds.

Try it on your own recording.Free during the beta.
Open Tavim