Researchers from the Wharton School at the University of Pennsylvania conducted an experiment to determine if ChatGPT can generate better product ideas and do so more quickly than Master of Business Administration (MBA) students.
Large language models (LLMs), like OpenAI’s GPT series, have demonstrated remarkable capabilities in generating coherent and fluent text across various domains.
In this context, Professors Christian Terwiesch and Karl Ulrich from the Wharton School, Lennart Meincke, a graduate student at Wharton, and Professor Karan Girotra from Cornell Tech were part of the research team that conducted the experiment and published it as a technical paper.
The study aimed to compare the product ideation capabilities of ChatGPT-4, a chatbot based on a state-of-the-art language model, with those of students from an elite university.
“I was genuinely impressed by the quality of the results,” said Terwiesch, a professor in the Department of Operations, Information, and Decisions, in an interview with Wharton Business Daily. “I naively thought that creative work would be the last area where humans would be superior in problem-solving… so we organized this race between man and machine,” the researcher highlighted.
Idea Generation Experiment
The academics began with 200 ideas submitted by MBA students enrolled in a product design course at an elite university, class of 2021. The class assignment required them to come up with products for the college student market priced at $50 or less.
The timing of the class is noteworthy because the ideas were generated before the launch of ChatGPT and other generative artificial intelligence tools that are now available for free.
Subsequently, the academics assigned the same task to ChatGPT-4, generating 200 ideas. For the first set of 100 ideas, the program received instructions without context or examples. For the second set of 100 ideas, it was “seeded” with examples of good product ideas.
Prompts Used
The researchers used a prompt to provide contextual information and subsequent user messages to request ideas. The user’s prompt included the additional request that descriptions be between 40 and 80 words.
The system prompt used by the academics for generating product ideas was:
“You are a creative entrepreneur looking to generate new product ideas. The product will target college students in the United States. It should be a physical good, not a service or software. I’d like a product that could be sold at a retail price of less than about USD 50. The ideas are just ideas. The product need not yet exist, nor may it necessarily be clearly feasible. Number all ideas and give them a name. The name and idea are separated by a colon.”
They also used the following user prompt:
“Please generate ten ideas as ten separate paragraphs. The idea should be expressed as a paragraph of 40-80 words.”
ChatGPT Generated Ideas at a Faster Rate
As expected, the chatbot completed the task quickly and generated 200 ideas in approximately 15 minutes. (A single human generates around five ideas in 15 minutes; the pace decreases for groups).
According to the researchers, a professional working with ChatGPT-4 can generate ideas at a rate of nearly 800 ideas per hour. Considering a cost of $500 per hour for human effort, the ideas are generated at a cost of $0.63 each.
But the team didn’t stop there. They wanted to know if the ideas were viable, so they conducted a blind survey of potential consumers to measure their interest in buying the products.
The results were revealing. The average purchase probability for a ChatGPT-generated product was 47%, compared to 40% for the human-generated idea. Preference for “seeded” ideas was even higher, at 49%.
Let AI Be Your Co-Pilot
Terwiesch, who is also the co-director of the Mack Institute for Innovation Management, said there are three major takeaways from the experiment, with the first being “obvious.”
Everyone should use ChatGPT to help them generate ideas.
“It doesn’t matter if you’re in a local business plan competition or working on curing cancer,” he said. “It’s cheap. It’s fast. It’s good. What’s not to like? Worst case, you reject all the ideas and go with your own. But our research clearly speaks to the fact that your set of ideas will be improved.”
Secondly, the academic believes that generative AI can help people move faster through the innovation bottleneck, from idea generation to evaluation. The latter stage is much more difficult and critical for success, he stated. Out of a hundred ideas, only five may be good, and human expertise is needed to discern the winners.
“Thirdly, I believe we shouldn’t reduce the situation to man versus machine,” Terwiesch said. “We need to find ways for generative AI to become your creative co-pilot. Together, they can become a more innovative team.”
The study was funded by the Mack Institute for Innovation Management at the Wharton School of the University of Pennsylvania.
Contact
Christian Terwiescha
Reference (open access)
Girotra Karan, Lennart Meincke, Christian Terwiescha and Karl Ulrich. 2023. Ideas Are Dimes A Dozen: Large Language Models For Idea Generation In Innovation. The Wharton School, University of Pennsylvania Cornell Tech and Johnson College of Business, Cornell University, 14 p.
