Cyborg Drummer and AI Team Create Music That’s Not Humanly Possible
Designing a Robotic Prosthetic for a Professional Drummer
Advancements in technology, from artificial intelligence to 3D printing, have revolutionized prosthetic design. State-of-the-art prosthetics enable their wearers to complete tasks most of us take for granted. And since the goal of prosthetics is to enable their wearers to complete tasks important to their daily lives, it is common to find prosthetics customized for various activities, from running to cycling, and even for dancing.
Prosthetics design requires skills from various disciplines, such as biomedical engineering, robotics, mechanical engineering, computer science—and music. Yes, music. Runners need to run, and drummers need to drum.
Creating a custom robotic prosthetic for drumming proved to be the perfect project for a team from the Georgia Tech Center for Music Technology (GTCMT). And while the advancements in prosthetic design often include muscle- or mind-controlled activation, this project was unique in that part of the design was autonomous: The wearer does not completely control the actions of the prosthetic.
A Drummer’s Drive to Return to Drumming
Jason Barnes is a drummer. He’s played in reggae and progressive metal bands and lists drummers the likes of Matt Garstka, Luke Holland, and Neil Peart among his musical influencers.
At 22, he lost his right arm in a workplace accident. The accident had happened just days before he was scheduled to audition for the Atlanta Institute of Music and Media (AIMM). The aspiring drummer was determined to return to his music. Shortly after being released from the hospital, he created his own basic drumming prosthetic. It was an important step in his emotional recovery.
“It was only a couple of weeks after the accident, and I still had bandages on my residual limb,” he says, “so I taped a drumstick to my bandages and proceeded to play. That was the turning point in my life that let me know it was still possible.”
He developed his own spring-loaded drumming prosthetic, which he used when he reapplied to the school. He was accepted to AIMM in 2013, one year after he was originally scheduled to audition.
New Prosthetic, New Music
During his first private lesson with AIMM teacher Eric Sanders, Barnes described his concept for developing a robotic version of his drumming arm. Sanders showed Barnes some online videos of the robotic musicians from Georgia Tech. Through a chain of introductions, Barnes met Professor Gil Weinberg, the founding director of the university’s Center for Music Technology, who was ready to take on a new project.
Professor Weinberg and his students blend the worlds of music and technology, building musical robotic performers. They focus on the science behind how music is perceived by humans and then use machine learning to build algorithms, so the robots can understand music. The robots learn to comprehend musical elements such as beat, tempo, and syncopation. While they understand music the same way humans do, they can perform with unparalleled speed and virtuosity. They perform music that humans cannot.
“That’s what's interesting about this work,” Weinberg says. “We use musical ideas to push our robotic capabilities. Then the new robotic abilities push new ideas for music.”
Barnes emailed Professor Weinberg, asking if the GTCMT team would build a prosthetic arm so he could regain the capabilities of a wrist. He wanted a muscle-controlled prosthetic that would translate the muscle movements in his upper arm to control a mechanized forearm and wrist.
Professor Weinberg agreed to create a robotic prosthetic but was also interested in linking the design to his existing research on musical improvisation that was used with the robotic musicians.
“I said, ‘Sure, we're interested.’” Weinberg says. “Then I asked him, ‘What if the design allows you to have one stick being controlled by your muscles so you'll have your artificial wrist, and we add a second stick that has a mind of its own? It will improvise, and that will push you to new musical domains.’”
While not his original goal, Barnes readily agreed. Weinberg obtained a grant from the National Science Foundation to fund the development.
The Design Process
The initial design was completed in a mere six months. The drum strokes were modeled using MATLAB and Simulink. The prosthetic has motors that power two drumsticks, each capable of drumming faster than human drummers, at speeds up to 20 beats per second. The first is controlled both physically by Barnes’s arm and electronically using electromyography (EMG) sensors on his bicep. The second is autonomous, listening to the music and improvising a complimentary beat. It is a wearable robot.
There are also differences: The wearable robot enables Barnes to create truly unique music. It uses artificial intelligence (AI) to detect the beat, tempo, and density in his drumming and responds with a beat that complements what it “hears”. It also autonomously listens to the chords played by the guitarist and adjusts its speed based on the chords.
At the start, the team relied heavily on electrical and mechanical engineering skills. With the robotic arm constructed and operating reliably, the team transitioned to a computer science and AI focus. They developed the machine learning and deep learning code that enables the prosthetic and other robots to interact with the musicians.
“We use different techniques of varying complexity, from very basic machine learning techniques such as Markov chains to more complex algorithms such as convolutional neural networks,” says Zach Kondak, a recent graduate of the GTCMT who specialized in robotic musicianship.
Redesigning for Portability
The team would like to create a design that can be made available to other musicians who require prosthetics, but challenges remain. The current design is heavy and requires an AC power supply. It is also tethered to two computers for the signal processing. They are addressing these issues with a redesign.
The planned redesign will continue to work with EMG signals from the arm to control the device but will use embedded processors, so all signal processing will be completed on the device itself. Battery power will replace the AC power supply requirements. It will also be lighter weight, since supporting the weight of the device can become tiring during a long performance.
“We will make the next iteration portable, so Jason can use it on his own,” Weinberg says, “and have a career as a traveling musician.”
For Barnes, his ultimate goal is to tour. “Yeah, just playing music,” he says. “It doesn't get any better than that. That's always been my passion.”