The team at Tom’s Guide have been observing and reporting on the recent intelligent chatbot revolution with a subsequent question looming in our minds: Can the likes of ChatGPT replace us when it comes to reviewing technical hardware?
Equal parts concerned and curious, OpenAI’s chatbot prompted us to write review introductions for products we’ve tested over the years. ChatGPT is trained with information until 2021, so we can’t ask it to do a review of the iPhone 14 Pro Max or PSVR 2, but we’ve used some premium devices from the archives as our guinea pigs.
What’s more, we asked our staff to recall the AI-written version compared to the one written by the expert product reviewers in Tom’s Guide—without telling us which one. Watch the video below to see how it went:
Humans vs. AI: Who Writes the Best Product Review?
Product reviews are important to help our readers make informed purchasing decisions. This is especially true when it comes to first generation devices with new technology. Take the Samsung Galaxy Z Flip (2020), for example.
This Samsung Galaxy Z Flip review, written by Managing Editor Philip Michaels, describes the everyday use of an innovative smartphone from the perspective of someone with many years of experience testing mobile devices.
Computing writer Tony Polanco, reading aloud the introduction to Michaels’ review and one penned by ChatGPT (but was not told which one), said the AI version seemed “too dry” to be written by a human. Editor Kate Kozuch got it right that one line from the real review — “The closed Z Flip fits easily in my hand and my claws aren’t quite fleshy” — sounded like something Michaels would say.
We also tried our original Apple Watch review. While the ChatGPT version has accurate specs and a reasonable overview of Apple’s first smartwatch, “It seems too generic to be an actual review,” said Global Editor-in-Chief Mark Spoonauer. Spoonauer happened to write the original review of the first-generation Apple Watch in 2017, so he knew which of the introductions told a true story.
Finally, we compared reviews of the Amazon Echo Spot, the old circular smart display. While Polanco found both reviews “well-read,” Spoonauer compared the AI version to “Amazon’s product catalog.” Editor-in-Chief Mike Prospero easily located his own writing from the original review of the product, too.
Should we worry about our jobs?
Each of Tom’s four guides correctly distinguished reviews written by AI from human reviews. After reading all the rounds of this experiment, Kozoch realized how ChatGPT often used “ultra-generic phrases like unique and elegant” to describe products. not affected.
“Artificial intelligence will not replace the human experience of living with and using the product,” Spoonauer said. “I’m not worried about it coming to our jobs right now, but it’s clear that the industry is using these tools.”
Prospero said that there are “a lot of problematic things in ChatGPT”. While we were able to detect a real review from a chatbot-generated review, not all consumers may see the same cues. And it could get even murkier with the emergence of Bing with ChatGPT and Google Bard as AI-powered destinations for review-based buying advice.
More Tom’s guide
[ad_2]
