Previously existing solutions for this capability either rely on fixed-size, small samples of sheet music images or expect a machine-readable representation of the score derived with optical music recognition. This research, lead by Florian Henkel of Johannes Kepler University, describes a system capable of tracking a whole page of sheet music at once, interpreting musical performances of any duration from beginning to end. While trackers that only exploit a fixed-size audio input are usually unable to differentiate between repeated notes if they go past a given range, the proposed method has no issue even in scores that cover long stretches of time in the audio, the researchers claim.

Behind the scenes, such systems are usually divided into two core components: Image segmentation and audio interpretation. This framework combines the two into one architecture that is tasked with predicting a segmentation mask (consisting of all the different shapes detected and identified) for the given score image that corresponds to this currently played music. The audio is interpreted using a U-Net model architecture and that, in turn, utilizes a conditioning mechanism that directly modulates the activity of feature detectors that process the score image. This unique approach of tying these two independent pieces together makes the whole system trainable as a unit and much more efficient.

The system is real-time capable due to a constant runtime per step. It compares favorably with existing baselines on synthetic polyphonic piano music and sets the new state of the art for sheet-image-based score following in terms of temporal alignment error. But, there are still problems in generalization for actual piano records. Although the model shows a far more precise synchronization in most cases, we are seeing a decline in performance over different recording conditions. Addressing these shortcomings and more, the team says "future work will also require testing on scanned or photographed sheet images, to gauge generalization capabilities of the system in the visual domain as well". They add that "There is currently no dataset consisting of scanned sheet images with precise notehead to audio alignments, it will be necessary to curate a test set. The next step towards a system with greater capabilities is to either explicitly or implicitly incorporate a mechanism to handle repetitions in the score as well as in the performance."