Piano Transcription With Pivots and Pedals
Piano music is often characterized by many notes, and the pedals can add a great deal to the overall sound of the piece. However, the complexity of piano playing makes it difficult to transcribe. In a new paper, researchers from TikTok developer ByteDance report on a high resolution piano transcription method that focuses on the pedals and pivots of the instrument rather than just the individual notes. This approach outperforms Google’s onsets and frames based system and sets a new state of the art for piano note transcription.
The research uses a sequence-to-sequence Transformer model optimized to directly predict MIDI piano notes. This approach achieves high-quality performance on both the MAPS and MAESTRO piano datasets, surpassing previous SOTA systems on both the onset F1 score and the frame F1 score. Unlike other SOTA methods that use a domain-specific encoder-decoder architecture, this framework can be used to develop general-purpose models that perform multi-task AMT on different musical instruments and datasets.
By using a log mel spectrogram as an input, the transcription model learns to translate from the musical timescale of the spectrogram into a MIDI piano timescale. This model consists of a sequence-to-sequence transformer with a simple semiCRF output layer that predicts a single MIDI piano note event at each frame. The model also leverages a deep reinforcement learning algorithm to adapt its predictions to the specific characteristics of the piano.
https://tartalover.net/nicole-pesce
High Resolution Piano Transcription With Pivots and Pedals
To account for the fact that the pedals can have different durations than the notes they trigger, the system uses a regression-based algorithm to model the short-term dependencies between the onset and offset times of both the note and the corresponding pedal events. This allows the system to find accurate onsets of the notes and pedals even when the onset and offset labels are misaligned, a challenge that has plagued previous approaches.
https://m.youtube.com/@tartalover2682/videos
Lastly, the model tries to find the shortest path between the pedal and the note based on the velocity of the MIDI piano note at each frame. This enables the system to predict the appropriate pedal timing and duration, which is necessary to accurately generate pedal MIDI notes. The resulting system can produce both a MIDI piano roll and a MIDI keyboard, giving users the ability to play the music they heard on their favorite synthesizer or to convert the transcription into sheet music.
This method is especially useful for analyzing how pianists control their pedalling, such as the ability to maintain a consistent pedalling rhythm with a piece of music or the tendency to slur the notes at the end of the improvisational phrase. It can also help train more sophisticated generative models that attempt to emulate the musical style of a particular pianist.
The authors hope that this approach can provide a foundation for future work in automatic piano transcription and other music analysis, including generative models. They have released their code as open source. They have also created a PiJAMA-30 subset of the MAESTRO data that they use for training, which is more suitable for tasks such as pianist identification than the full set of recordings in the original MAESTRO dataset.
