Two Truths and a Lie with ChatGPT: The critical difference between RAG and an LLM alone

Let’s play a game, Two Truths and a Lie.

1. Apple was originally founded as a clothing company before transitioning into the tech industry.

2. Steve Jobs once considered renaming the company "MacMan" before settling on the name "Apple."

3. The first product Apple ever released was a computer called the "Apple I," which sold for $666.66.  

These are the options ChatGPT gave me when I asked it to play. The answer? You guessed it—Apple was not founded as a clothing company.

While it is incredible that LLMs are able to emulate the creativity and complex interactions required to play Two Truths and a Lie, at this points that’s old news. So, why does this matter?

The answer is hidden in the second statement which is, in fact, also a lie.

This is similar to ChatGPT’s Snow White Problem, for which we discussed a more appropriate rules-based AI solution. This time, however, we’ll be using Retrieval Augmented Generation (RAG) and enriching the result by using rules-based AI and a knowledge graph to achieve the best of both worlds.

How to Improve LLM Applications and Why

The second statement given to us by ChatGPT is particularly deceptive because it is almost correct—rather than the company, it was Apple’s original iMac that was almost called ‘MacMan’. Therein lies the problem: Large Language Models (LLMs) can confidently provide falsehoods as truth (known as hallucinations), even going so far as to incorrectly explain why the false statement is true, leaving us with no way to differentiate one from another unless we already know the answer—very unhelpful in real-world situations.

“It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.“

Mark Twain

Our game is just intended as a bit of fun in order to highlight the challenge but there are applications using these techniques today for which there are very real consequences.

• In assisted clinical decision making, a misdiagnosis or inaccurate prescription may be fatal.

• In automated construction and design, misinformation or baseless recommendations could contribute to a major safety incident.

• In regulation and compliance, incorrect advice or missing details could induce heavy fines or legal troubles.

Of course, there are many more applications that aren’t so life-or-death, nevertheless, an incorrect assumption can be costly, with the potential to dissatisfy customers or waste time and resources. While LLMs have changed the world we live in, they’re far from perfect.  

As we mentioned, we previously suggested an alternative AI solution that, in the wake of widespread, well-intentioned but misguided, LLM-eager attitudes, is often more appropriate. However, in the cases where it's needed, there is a way to get the best of both worlds. It’s known as RAG.

What is RAG (Retrieval Augmented Generation)?

Retrieval Augmented Generation, or RAG, is already widely considered an effective solution to this particular problem, safeguarding truth while capitalising on the undoubted benefits of LLMs—providing an intuitive and accessible way for users to interact with an application, while offering game changing functionality at the same time.  

The way we do this is surprisingly simple—we use the LLM as a mediator between us and our data. Instead of asking the LLM for answers directly, we first task the LLM to ask questions to a database, then ask it to use exclusively the data it receives to answer our initial question.

Using this method, we ensure all answers are accurate according to the data we have, not a piece of misinformation spread around the internet. As an added benefit, the LLM is able to explain exactly why it said what it said, referencing the data explicitly rather than giving another unsupported statement.

Beyond truth and auditability, the simplest example of where RAG shines is when a user uses vague and ambiguous language in their input. With RAG, the LLM interprets the user’s input within contextual constraints and points them to the most relevant information in the database, avoiding hallucinations and disinformation. Here, RAG affords the user vast freedom and scope with their input while still guiding them down a known path. The opposite can be seen in my original question to ChatGPT which, like most real inputs, was not truly as precise as it could have been. For one reason or another—we can only guess—it caused ChatGPT to return something that was misleading and untrue.

RAG offers a huge improvement but there is still one piece missing from this puzzle. The persisting problem is that RAG restricts the questions we can ask because all answers are limited by our data. LLMs provide a much-improved user experience with a human-like input method and easy-to-digest output, but one of the main reasons we want to use an LLM in the first instance is for its ability to answer more complex questions which we’ve just managed to neutralise entirely.

How do we get that back? Rules-based AI.

How to Improve RAG with Rules-Based AI

By combining both LLMs and rules-based AI we can run our applications faster, reduce development costs, and critically for RAG, extract greater insights from our data, all while maintaining the truth, consistency, and auditability we’ve worked so hard to achieve.

Rules-based AI, otherwise known as semantic reasoning, adds new information to a knowledge graph database using a rules engine like RDFox. In practice, this means we can automatically enrich our data, creating a wealth of information and insight which allows us to ask much more complex and valuable questions than we ever could have hoped too without.

Since these rules are logical by design, any results that follow are guaranteed to be correct and consistent with the underlying data. This makes reasoning the perfect companion to RAG as it maintains all of the gained accuracy and explainability while amplifying the benefits of LLMs rather than offsetting them.

RDFox supports advanced features such as aggregation and negation that enable even the most complex concept concepts to be encoded in rules. These, couple with automatic incremental updates and unrivalled performance are enabling the most ambitious use cases of our clients, spanning some of the world’s largest banks, manufacturers, retailers, and publishers.

The Evolution of LLMs to RAG to Rules-based AI

With how prevalent they are, it’s easy to forget that the technology behind LLMs is only just leaving its infancy—we're watching it grown into an unwieldy teen before our collective eyes. While this can make it difficult to keep up with the latest best practice, it also presents a golden opportunity for those paying attention.

By introducing LLMs, applications effortlessly inherit functionality and accessibility that once would have been impossible. With RAG, the application becomes practical, able to target the features for real-world use. And by adding rules-based AI, the benefits of both are amplified, creating the most powerful applications available today.

Want to start using RAG yourself? Get in touch for a demo or get started with RDFox for free!