An Instrumental Role

University of Maryland computer science Ph.D. student Snehesh Shrestha seldom turns down a challenge, whether it’s fixing a complicated machine, learning a new song on guitar or running a little faster than his last mile. Now, he’s learning how to play the violin.

But Shrestha is not taking on this challenge alone—he’s building an app that helps violin students, himself included, improve their skills. Though Shrestha has played guitar for 25 years, he finds the fiddle’s four strings to be far more formidable.

“If a beginner practiced guitar for about a month or two, they should be able to play a couple songs. Not in violin,” Shrestha said. “In violin, the beautiful thing—as well as the challenge—is the fact that you have a bow, and with a bow comes tremendous control. My bow strokes have gotten better, but I’m still struggling with the control aspect, and a lot of my skills are a reflection of the technology I’ve built.”

That technology is called VAIolin—pronounced like “violin”—and it’s a joint venture between UMD’s Department of Computer Science, School of Music and the University of Maryland Institute for Advanced Computer Studies (UMIACS). The app uses sound, video and artificial intelligence (AI) to provide personalized feedback to violin players, with the aim of making the violin, one of the most difficult instruments to master, more accessible. The technology could be applied to other instruments in the future. 

This project is part of a longtime mission for Shrestha: to develop technologies that make people’s lives easier and more enjoyable.

From toddler to tinkerer

Growing up in Kathmandu, Nepal, Shrestha was a restless middle child with boundless curiosity. He was fascinated by machines, whether it was the robotic toys he saw on store shelves or the floppy discs in his father’s office that contained (to his astonishment) a library’s worth of information.

As a teenager, Shrestha took a crack at fixing his family’s French-made washing machine. Unable to understand the manual, Shrestha relied on his instincts while taking the machine apart. Ultimately, he was able to diagnose and fix the problem.

“Even though it was tough, it felt like a big puzzle—almost like decoding,” he said. “My parents gave me a lot of freedom and opportunity to tinker—within limits.”

When Shrestha was in high school, he discovered a love of programming and started competing against his brother and his best friend to come up with the coolest ideas. The three boys joined a Southeast Asia software competition and met other students who shared their passion, which later inspired them to launch an intraschool organization of young software developers called the Nepal Society of Programmers.

“We were really curious and wanted to go beyond how we were being challenged in class,” Shrestha said. “I think that was the start of entrepreneurship for me, as well as an introduction to all the different things you could do with a computer.”

Join the club

Shrestha’s love of friendly competition and camaraderie carried over into his college years. He received a scholarship to study computer engineering at Elizabethtown College in Lancaster County, Pennsylvania, where he and his friends started a robotics club.

The members raised enough money to build a 300-pound ultrasonic robot dubbed the “WunderBot.” They entered it into the Intelligent Ground Vehicle Competition (IGVC) in Rochester, Michigan, and placed 12th in the design division. The experience cemented Shrestha’s love of hands-on engineering and digital interfacing.

After graduating from Elizabethtown in 2005, Shrestha worked as a software developer at Two Technologies, Inc. for two years before accepting an engineering role at Qualcomm Inc., one of the world’s largest cell phone chip manufacturing companies. At the time, the company had not yet delved into robotics, so Shrestha once again formed a robotics club. The group attracted hundreds of his coworkers and ultimately built Qualcomm’s first robotics development platform.

Shrestha appreciated the group’s ability to break down barriers and create a bond over a shared interest.

“With robotics clubs, there’s no hierarchy. You have senior vice presidents and people from all walks of life, from the hardware team to the software team,” Shrestha said. “It didn’t matter—we all came together. We were all just excited to build something cool.”

Beginning in 2011, Shrestha served as a judge for Qualcomm’s Innovation Fellowship, which provides funding to Ph.D. students around the world. There, he crossed paths with UMD Computer Science Professor Yiannis Aloimonos and UMIACS Research Scientist Cornelia Fermüller, and the more Shrestha learned about UMD’s computer science program, the more he wanted to be a part of it. Thanks in part to Aloimonos’ encouragement, Shrestha applied and was accepted into the program.

When Shrestha came to UMD in the fall of 2016, he dove headfirst into research, joining the Department of Psychology’s Culture Lab and the Perception and Robotics Group led by Aloimonos and Fermüller. Aloimonos believes that Shrestha brings a unique mindset to UMD and to the VAIolin project specifically.

“When I think of Snehesh, two things stand out: vision and leadership,” Aloimonos said. “He is able to decompose a complex problem like the field of music education, analyzing needs, culture and techniques from many disciplines to achieve a much broader perspective.”

Strings attached

Shrestha was investigating technologies to improve his running posture in December 2020 when he fielded a timely question from UMD Associate Professor of Violin Irina Muresanu: Could computer science be used to improve the posture of violin students and help them play better? 

Inspired by the idea that better posture could help musicians as well as athletes, Shrestha realized an AI platform could help modernize violin lessons, which have gone largely unchanged for centuries. To take this idea forward, he interviewed violinists and studied the physics, music theory and biomechanics of the instrument. 

“When we interviewed teachers and students for this project, we discovered that the way violin is taught today is not all that different from how it was taught 300 years ago,” Shrestha said. “That became obvious during the pandemic when everyone had to teach through Zoom, which is difficult for violin.”

Shrestha, Fermüller, Muresanu and Institute for Systems Research Professor Shihab Shamma
received funding from a 2021 Maryland Innovation Initiative Award, enabling them to make their vision—VAIolin—a reality. Fermüller and Muresanu then received a 2023 UMD Grand Challenges Team Project grant to continue their research.

VAIolin assesses a player’s performance through video and sound. The sounds produced by a violin vary drastically depending on the position of the bow and the amount of pressure applied, so even if a user’s camera is fuzzy, the app can use sound to determine posture. In addition, the app provides feedback on the quality of sound, pitch and rhythm.

To create an inclusive app that works for a variety of students, the team of researchers attached full-body sensors to 15 violinists of different ages, body sizes and skill levels as they played, enabling them to collect data at 100 frames per second from seven different angles. Previously, there was no publicly available data on beginner violinists—information that was needed to train their machine learning model.

Though mastering the violin can take more than 15 years, Shrestha believes the VAIolin app could help students progress more quickly by providing feedback between lessons. This could ultimately help students correct mistakes before they become habits.

“One of the biggest challenges we found is that people would end up practicing wrong and would need to relearn, and relearning is exponentially harder than learning,” Shrestha said. “We want them to learn the right way from the get-go.”

Shrestha plans to graduate with his Ph.D. this year, but that doesn’t mean his work with VAIolin will end. He hopes to help establish a company to launch the VAIolin app and take it to market.      This summer, the research team will conduct focus group trials to determine which of the app’s features are most useful, and which ones could be improved. 
In the meantime, Shrestha plans to keep learning and challenging himself—both as a violinist and as a computer scientist.

“Everything I’ve done at UMD has been related to people interacting with a robot or with technology to solve a problem together,” Shrestha explained. “That vision of the future is very exciting to me.”

Written by Emily Nunez