The Demo Is Not the Product: What Robotics Companies Learn Too Late

Artificial intelligence in robotics can seem too good to be true. Can that software really write that code, identify those objects, or control those movements? Will what works perfectly in the demo stay reliable in real-world conditions at the worksite or in a consumer’s home?

Behind every robotics breakthrough lies a complex set of engineering realities including mechanical design, safety systems, supply chain strategy, manufacturing readiness, product development discipline and new developments in AI enablement.

As part of Boston Robotics Tech Week, FORGE gathered a panel of experts working at the intersection of AI, robotics and manufacturing to answer questions about AI and robotics. FORGE partner M&T Bank hosted the event at their offices in Boston’s Winthrop Center.

We discussed:

• The gap between robotics demos and real-world performance
• The relationship between AI and robotics today
• Scaling robotics
• The future of robotics, AI and automation
• What robotics innovators should know to be successful

Panelists

FORGE Manufacturing Expert in Residence Mark Michalski moderated the panel of carefully chosen experts:

• Fidelity Private Shares Vice President Kristen Craft, for her experience with the startup world, from startup operations world to angel investing
• Massachusetts Technology Collaborative Program Manager Dr. Raagini Rameshwar, for her expertise in economic development at the intersection of AI and robotics
• Brooks Automation Vacuum Automation VP and GM Ulysses Gilchrist, for his practical experience leading product development and engineering for automated and AI-enabled robotics
• rStream CTO Ryan Knopf, for his perspective as founder and CTO of multiple robotics startups

The gap between robotics demos and real-world performance

Robotics demos can make a product look sophisticated, advanced and reliable. But those demos are often limited to a specific environment and set of conditions. The demo might overstate how autonomous the robotics system actually is.

Even a product with a great demo can encounter all manner of issues beyond the prototype: quality inconsistencies that arise once the engineers hand off building robots to a manufacturer, supply chain backlogs that hold back the production schedule, a lack of reliability in a real-world environment. An impressive demo of a product that has issues in deployment isn’t necessarily a sign that an innovator set out to deceive their audience – more often, it’s a sign that the innovator isn’t ready to scale.

Founders often face unexpected barriers on the journey from prototype to production. Rameshwar said, “You have to be testing and manufacturing your robot beyond [a bench prototype]. Getting that infrastructure together can be hard and cost-prohibitive.”

Craft said, “Investors are looking at investment opportunities with more rigor than ever…they ask way more questions versus being content with a flashy demo alone.”

According to Craft, investors are looking for red flags like:

• Lacking connections to potential customers or target industries (or having never worked in the target industry)
• Not understanding a product’s ideal customer profile
• Urgent need for a large amount of funding – and no idea that their expectations are unrealistic

Rameshwar agreed, adding that robotics innovators often don’t consider the practicalities of the end user experience early enough, like who’s going to charge or service the robot.

Knopf said robotics founders often do customer discovery and product-market fit research based on technical metrics and technical performance thresholds, while completely missing operational metrics. He recommended that founders research overall equipment effectiveness (OEE) and define success metrics using OEE principles.

The relationship between AI and robotics today

AI hype is widespread in conversations about robotics, but we wanted to ask the experts how technologists are really using AI…and whether it lives up to the hype.

The biggest robotics challenges Gilchrist sees at Brooks are tuning motion control and servos to minimize vibration. Neural networks have been helpful for auto-tuning, but not helpful enough. In the semiconductor space, too much motion control variability won’t work. Where AI is working for Brooks is in the development process, and in data analysis to predict when robots are likely to fail or break.

For rStream’s Mini-MRF product, AI is an essential part of the software that sorts waste into recyclables and non-recyclables. However, that software only works in conjunction with the Mini-MRF’s hardware system, which has required significant engineering, manufacturing and supply chain knowledge and effort to create.

Knopf said, “The most accurate recycling classification model can’t make a conveyor belt motor have more torque…There are so many other aspects to a hardware system that go into reliable long-term operation and demanding industrial environments.”

On top of the need for advanced hardware to make the relationship between AI and robotics productive in the real world, Rameshwar noted the barrier of user distrust. Robotics customers already worry about reliability. Adding the ambiguity of AI can make customers even more anxious about a robot’s performance.

Investors are also becoming more diligent in their questions to robotics startups about AI. Craft says investors expect founders to have a clear vision of how to leverage AI. This could include using it for:

• Software or hardware development
• A technical aspect of the robot
• Sales outreach and business development
• Marketing and web content

AI can give innovators a competitive advantage. In an environment where investors are constantly seeking to derisk investments, that can make all the difference.

Scaling robotics

The demo is not the product. What happens when you have to build a supply chain, manufacture that product, and deploy it in a factory?

Knopf said robotics founders often underplan for contingencies before starting a pilot. Founders should plan early pilots and installations to have a small “blast radius” (affect the minimum amount of operations onsite if they fail). And they need to have frank conversations with pilot customers about what will happen if their system goes down, such as:

• How long it will take to repair the robot
• Whether they have a spare robot on hand
• Whether losing the robot will completely shut down a production line
• How much an operational interruption would cost the customer

When preparing to scale, innovators also need to prepare for supply chain instability. For example, the “RAMpocalypse”, driven by AI demand for memory, has caused memory shortages and drastic increases in memory costs. When scaling, innovative companies should consult with supply chain professionals and commodity managers to get out in front of supply chain issues.

Craft views scaling from an investor’s point of view: focused on reducing risk. Founders should ask themselves what the biggest risks to their company are and how they can mitigate those risks before they happen.

“Thinking through all these different potential risks and trying to come up with a ‘pre-mortem’ is what I often see among the best companies,” Craft said. “That’s what investors are looking for.”

The future of robotics, AI and automation

Robotics and software have always gone hand in hand. Now, however, increasing amounts of software are developed in part or wholly by AI tools, and AI is a big part of the software in many robotics products. Our panelists weighed in on that change and where they predict it will lead.

Rameshwar pointed out that although there are some regulatory guardrails to protect the environment and consumers’ rights, they have fallen behind current advances in artificial intelligence. “We need to make sure…that we’re not putting out technology that can cause harm, especially in these mission-critical fields,” she said. “It’s time to start those processes around physical AI systems that are going to work right next to people, which could be great but could also cause physical harm.”

Craft observed that technologists will have to meet coming changes in the world. “Our society is not in good shape when it comes to taking care of our aging population,” she said. “Technology and innovation that supports that trend is super important right now.” She meant more than just technology to care for elders. For example, the average age of maritime and shipyard workers is around 41-55 years old, and not enough young people are entering the shipping industry. One solution is for innovators to create more automation in the shipping industry to fill the growing workforce gap.

What robotics innovators should know to be successful

Michalski asked our panelists the one thing they think a robotics company should understand early in product development to be successful.

Rameshwar said, “Think more about the service. You are selling a service that happens to include robotics, not a robot that is doing a cool thing.”

Gilchrist said, “You don’t want to be selling specs and features. You want to be selling values.”

Knopf said, “Is it a complete job? Does the system that you are creating complete a whole unitary process or job in the business or industry it’s in, or are you creating a partial solution that may struggle to integrate with existing business processes?”

Craft said, “Get to know the customer better. It never fails.”

Thank you to our sponsors and hosts for supporting this event and helping bring the robotics, AI and manufacturing communities together at this event for Boston Robotics Tech Week. Special thanks to the Robotics Tech Week organizers, M&T Bank and Fidelity Private Shares, for making tonight possible, as well as to the Richi Foundation for supporting the event.