From “I’m Not a Robot” to Semi-autonomous Mega-machines

Imagine a typical construction site—bulldozers, excavators and other heavy equipment all buzzing with activity. But when you take a closer look, you realize something’s missing—none of these mega-machines have anyone in the operator’s seat. They’re all being controlled remotely by a single human operator miles away from the job site.

It’s not science fiction. It’s Teleo, a groundbreaking startup co-founded by University of Maryland alum Vinay Shet (M.S. ’03, Ph.D. ’07, computer science).

“We are on a mission to revolutionize how we build the world,” Shet explained. “We are converting our customers’ fleets of heavy equipment to semi-autonomous robots. That means they’re able to run jobs with no one in the cab of the machine.”

Since Shet and Romain Clement launched Teleo in 2019, the company has been disrupting the construction and mining industries one super-sized, semi-autonomous robot at a time. Now, after nearly three years of product development, on-site testing and customer feedback, Shet is ready to take Teleo to the next level. The company just marked a major milestone, landing a $12 million Series A investment to accelerate growth and expansion.

“We’re in production with a few customers that are testing it out, and now the goal is for us to start really scaling this thing,” Shet said. “I’m very optimistic that in a few years that vision will become a reality and we’ll start to see many more machines on work sites running the Teleo system.”

Discovering computer vision

Computers have been a part of Shet’s life since his early days growing up in India. He was always interested in technology and as he pursued computer science in college, he found himself fascinated with computer vision, which inspired him to travel from India to Maryland to do his graduate studies at the University of Maryland.

“UMD was then, and still is, one of the top computer vision programs in the country,” Shet noted. “Working with Professor Larry Davis and doing my Ph.D. at Maryland, those were some of the most amazing years of my life. In many ways it set the stage for the rest of my career with computer vision.”  

After earning his Ph.D. in 2007, Shet joined the technology giant Siemens as a research scientist, authoring papers and developing new technologies like advanced video content analytics for Siemens’ video surveillance applications. He quickly realized that using machine learning and computer vision to create new, real-world technologies was no easy task.

“At Siemens, I actually continued the computer vision work I was doing in my Ph.D., and I tried to build a real product around this stuff, and along the way I discovered how hard that was,” Shet explained. “There was really no middle ground where you could take advanced technology and bring it to market.”

The drive to do exactly that pushed Shet toward his next opportunity—at Google.

“In 2013, I went to Google as a product manager,” Shet said. “At the time, Google was expanding the deployment of computer vision technologies at internet scale, and I had the good fortune of working with some really iconic products.”

From Street View to “I’m not a robot” and self-driving cars

At Google, Shet led a team that developed products that are now used by millions. He played a leading role on a project that pioneered the use of machine learning to create map data from aerial imagery and Google Street View.

“With Street View, my team was one of the first to use deep neural networks to build maps from imagery,” Shet explained. “Deep neural networks made computer vision applications actually work and come into the mainstream; it completely revolutionized the field.”

Shet was also one of the creators of Google’s familiar “I’m not a robot” reCAPTCHA system that enables web hosts to distinguish between human and automated access to their websites and protect sites from malicious hackers.  

“I think reCAPTCHA performs a very important function to keep the internet clear of spam,” Shet said. “Like all the Google projects I worked on, it was fun and highly impactful as well.”

From Google, Shet joined Lyft in 2017 as director of product management, leading strategy and development for the company’s self-driving car initiative and working with Clement who would later become his business partner.

“When I joined Lyft, it was just a natural progression for me, I went from building computer technology that passively perceived the world to one that actively operated in it within the context of a self-driving car,” Shet explained. “But even as we were building out maps for the car to see and understand, we were still leveraging a human brain to curate and build maps for the cars to drive.”

It was a huge learning experience that set the stage for what was ahead.

“At Lyft I had a front row seat to see the cutting edge of self-driving cars,” he said. And that experience eventually morphed into what I’m doing today.”

Fulfilling an entrepreneurial dream

For Shet, the launch of Teleo in late 2019 fulfilled a dream that sparked nearly 20 years earlier, when he was a graduate student at UMD.

“In my very first year at the University of Maryland, I took a class on entrepreneurship out of pure interest—it was not required,” Shet recalled. “That class instilled a spark in me that I wanted to go and start a company someday.”

Now with Teleo, Shet and Clement are re-inventing heavy equipment operations on a grand scale. Initially, the idea that hundred-ton machines could be operated semi-autonomously like robots might have sounded nearly impossible to some, but it was a challenge Shet wanted to take on.

“Progress is what happens when you’re working at the edge of what people think is difficult or impossible to do,” he explained. “The reason I think technology has not permeated the construction and mining industries as much is that it is very difficult work, working conditions are harsh and any technology has to survive those conditions. I hope that with Teleo we can make workers’ lives a little bit better.”

With its cutting-edge semi-autonomous control system, Teleo’s mission is to make heavy equipment work more efficient, safer and more employee-friendly, by giving operators the ability to control machines like loaders and bulldozers—even multiple machines at once—from a remote location. Like any new, next-generation technology, it can take some getting used to.

“What is really interesting is the interaction of people with our machines,” Shet explained. “Very early on, we took it to a job site and though there was nobody in the cab, we saw people around the machine signaling to it to come forward or go left, and we were like, ‘Who are you talking to? There’s no one in there.’ But people adapt quickly to the point where they’re saying, ‘Oh yeah, it’s just a robot, they’re part of the job now.’”

For Shet, who has published more than 20 papers and has over 25 issued or pending patents, growing Teleo is about more than building a successful startup. It’s also part of a bigger mission that has inspired his entire career—harnessing the power of technology to change the world.

“I am deeply motivated by the transformational power of technology,” Shet reflected. “I think people don’t fully appreciate this, but technology has fundamentally changed the way we as humans live on planet Earth. There’s good and there’s bad but you want it to take us to a good place. And I really want to be part of that.”

To track Vinay's latest work, follow Teleo on Twitter and LinkedIn.