Children under eight years old outperform ChatGPT and other LLMs on basic innovation tasks.
Between the media prominence large language models (LLMs) have achieved and the speed with which they’ve achieved it, most people have probably wondered whether ChatGPT will be coming for their job anytime soon. The good news for engineers is that we’re still a long way away from any artificial intelligence (AI) being able to perform one of the most essential engineering tasks there is: innovation.
Can LLMs Innovate?
A recent study by developmental psychologists at the University of California, Berkeley found that even young children outperform LLMs on basic innovation tasks. For example, when asked to select a tool to draw a circle in the absence of a compass, given the choice between a ruler and a teapot, children aged three to seven chose the correct answer (the teapot) 85% of the time, while adults chose correctly 95% of the time. In contrast, large language models presented with the same choice performed significantly worse to varying degrees, with five different models giving the correct answers 8% to 75% of the time.
Compare this with model performance on an imitation task, where participants were asked which object would “go best” with another (e.g., a compass and a ruler vs. a compass and a teapot): 88% of children chose correctly, vs 59-85% of LLMs. It’s worth noting that children also outperformed adults (84%) in this task, though the researchers offer no explanation for this result.
The study also found that children could discover how a novel machine worked by experimenting and exploring with it, but LLMs struggled to make the same inferences when presented with descriptions of the evidence the children produced. In this and the other innovation task, the researchers offer the same explanation for the models’ poor performance: the answers were not explicitly included in their training data.
“AI can help transmit information that is already known, but it is not an innovator,” said lead author Eunice You in a press release. “These models can summarize conventional wisdom but they cannot expand, create, change, abandon, evaluate and improve on conventional wisdom in the way a young human can.”
Innovation in Generative AI
The obvious objection to raise in response to these findings is that large language models are the wrong place to look for innovation in AI. Perhaps we need to widen our scope to include other forms of generative artificial intelligence. After all, there are already examples of generative AI designing electronic circuits, which certainly seems beyond the capabilities of a 7-year-old.
Can generative AI innovate?
A team of MIT engineers appears to have answered this question by giving deep generative models (DGMs) a bicycle frame design problem. Initially, the frames the models created mimicked previous designs but failed to achieve the necessary performance requirements. When the engineers gave the same problems to DGMs that were designed to prioritize specific engineering-focused objectives over statistical similarity they saw some improvement, but arguably not enough. While a few of the resulting frames were lighter and stronger, others were physically impossible.
Generative AI for Engineers
What this suggests is that even when generative models are trained with the relevant engineering datasets, they will still be more like tools than users. What kind of tools will they be? Here’s a simple use case from materials science.
Last year, engineers at University of Wisconsin-Madison used ChatGPT-4 to reduce the workload of extracting data from scientific papers by 99%. Generative models trained on materials science data could potentially go beyond mere information gathering to assist with material selection, analysis and even testing through integrations with materials testing software.
We’ve also seen some early attempts at integrating AI into enterprise software, such as in PLM with Siemens Teamcenter and in CAM with SprutCAM X. Generative models trained with the right datasets could potentially offer design assistance, verification and documentation; and it would make it easier to learn design principles and best practices with automated coaching. If it actually works, the gap between design intent and CAD software would be smaller than ever.
The Innovation Problem for Generative AI
With all the myriad tasks that generative AI can handle and the speed with which it appears to be evolving, one might be tempted to dismiss innovation as just another problem that can be solved with the right data set. The trouble is that innovation requires coming up with new ideas, not just synthesizing old ones. As it stands today, AI is wholly dependent on training data. This is an Achilles heel that has led to all sorts of problems with bias, but it’s also the reason innovation will remain out AI’s reach for the foreseeable future.
At least for now, engineers can rest easy that AI won’t be coming for their jobs any time soon.
