The system, called Audeo, analyzes top-down videos of someone tickling the ivories to predict which keys are being pressed in each frame. It then produces a transcript of the music, which a synthesizer translates into sound. The researchers trained and tested the AI on footage of pianist Paul Barton playing tunes by famous composers. They then evaluated the accuracy of the Audeo’s compositions by playing them to music-recognition apps, such as Shazam and SoundHound. The apps identified the tune 86% of the time — just 7% less than they recognized the source videos. [Read: How much does it cost to buy, own, and run an EV? It’s not as much as you think] Senior study author Eli Shlizerman, an assistant professor at the University of Washington, said he was surprised by the quality of the AI’s output: You can judge its performances for yourself in the video below: The researchers have also explored using Audeo to change the style of music. Shlizerman said the system could show how music produced by a piano sounds when played through a trumpet. He hopes the research will enable new ways for people to interact with music: You can read the full study paper here.
