Pete Koomen, group partner at startup accelerator Y Combinator, noted a stark difference between how he interacts with AI applications compared to how he utilizes AI itself:

“AI feels like a power tool. It's a lot of fun. Many AI apps don't feel like that. Their AI features feel tacked-on and useless, even counter-productive.”

So why exactly do so many up-and-coming AI products fall flat?

What is a Horseless Carriage?

The image illustrates a black and white drawing of a man riding a large, rectangular carriage with multiple wheels of varying sizes. The man sits at the front of the carriage driving it, and a separate man stands behinds the carriage.

Trevithick's London Steam Carriage of 1803

“Horseless carriages” were early automobiles. There were many reasons they were unsuccessful, but one big issue was that their design relied entirely on the framework of horse-drawn carriages. As Koomen emphasizes:

“Whenever a new technology is invented, the first tools built with it inevitably fail because they mimic the old way of doing things.”

This can be applied to current AI applications that fail to break out of a similar traditional framework.

The Developer-Consumer Split

Historically, the outlook of software design is to let developers design and dictate behavior at a high level, while allowing individual users to customize behaviors to their specific use-cases. Hence, this division of labor led to the AI system prompt and user prompt. In many consumer products, the system prompt is hidden from the user: they can only edit or change the user prompt.

The image depicts two, rectangular text boxes (aligned vertically) containing the description of a “User Prompt” and “System Prompt.” An arrow is aligned in the middle of the image pointing at a third text box containing the description of what an “LLM” does.

The standard convention of LLMs

This split works well for “old world” SaaS (Software-as-a-Service) products because many consumers don’t want to deal with the complexities of code or architecture. However, AI is extremely powerful because of its ability to process and build in natural language. The “system prompt” doesn’t need to be written by an engineer, it can be written by anyone who knows how to communicate through words.

Therefore, the traditional way of building software fails to fully utilize the capabilities of AI. There can easily be a stronger user focus: system prompts aren’t fully representative of users’ diverse voices if they’re being hidden by the builders.

Case Study: Gmail’s AI Assistant

Google’s new email generation feature — built with Gemini — is a prime example of mimicking the “old world” way of building software.