Building a 21st Century knowledge engine

In Neal Stephenson's fifth novel The Diamond Age, a girl in some not-too-distant future is given a stolen copy of a very special educational device – "The Young Lady's Illustrated Primer".

This book is designed to deliver self-directed learning that adapts to the girl's existing knowledge and take her on a journey to become a powerful and independent woman.

“If the item of stolen property had been anything other than a book, it would have been confiscated. But a book is different—it is not just a material possession but the pathway to an enlightened mind, and thence to a well-ordered society.”

If AI language models were a band, the past few months have been one of those "overnight success" moments. Years in the making, indeed two years since the GPT-3 model first emerged, the arrival of the simple ChatGPT user interface has made Open AI's work a revelation.

Interface is everything, though. So it's a perfect time to think about what a just right interface applied to a Large Language Model (LLM) AI can achieve in the years ahead.

There's a lot of talk about its potential to destroy jobs, ruin education, and overwhelm the Internet with misinformation. All this and more is possible. I believe this is a true 'next era' turning point, set to be remembered alongside the arrival of the internet and then smartphones. And like those other technologies, it's the 'when' of significant change that remains the big question.

What excites my noodle so much is how a shift in our search paradigm can create a new 'knowledge engine' experience that has been held back by the existing revenue model at the biggest search company in the world.

Engaging a Knowledge Engine

Today, when we ask a question in the world's biggest search engine we get a list of links. The top few links have paid to be there, and it's up to us as users to scan the titles and snippets to see if a click will lead us to the information we seek.

In some cases, Google also offers a very short answer to our question, drawn from a partner website. These short answers have arguably been bad for the rest of the web. Google saves the user a click, delivering less traffic to the source it was drawn from, but making the user happier on that occasion that they learned what they needed with a minimum of fuss.

But Google also knows that delivering nothing but answers would kill its revenue. It only gets paid when people click on an advertiser link. If Google became a one-stop shop for answers... where would it make its money?

People have been wowed by ChatGPT's ability to generate high school quality essays on seemingly limitless topics. And a focus of this capacity has been on what it means for users to be able to quickly 'cheat' their way to paragraphs of text that they can reuse instead of writing their own from scratch. I'll dig into that theme another day, but it's this assumption about generating copy to be used elsewhere that I want to deflect today.

The ability to reveal useful, readable knowledge in an instant holds amazing potential for quickly cribbing information to build our own understanding of a topic – without sifting through lists of links and websites to hopefully find someone who knows what they're talking about.

This capacity for distilling collective knowledge into succinct, digestible information is what excites me the most. Do you prefer a 1,000 word essay or a three paragraph summary? Would you prefer a list of bullet points or information displayed in a table format? Ask and you will receive.

Of course, right now the scariest problem is ChatGPT's ability to sound incredibly authoritative even as it fabricates lies or gets its facts mixed up. It isn't really a knowledge engine. It's an expert at guessing which words belong near each other when asked about a given idea. But it has no expertise in validating what it's producing.

Not yet.

Learning how to learn

One of the concerns I keep mulling over is on how the next generation of junior staffers in many fields will get their chance to climb the ladder as LLMs start performing a lot of junior-level work. Drafting copy. Outlining plans. Suggesting potential strategies.

But as I come back to this Knowledge Engine idea I think the most important skill in an AI-powered future is to enhance our ability to learn. To use these very same tools to not just generate words but to read what it generates, to keep asking questions and to keep building our own knowledge. If used effectively, we should be able to accelerate what it means to be a 'junior' and raise the baseline for the work we can perform and the creativity we can apply to the elevated basics made possible by a language model.

In The Diamond Age, the book – an incredibly powerful knowledge engine – adapts to the girl's needs to meet her where she is in order to carry her toward the kind of person it can help her to be.

In our world, we know AI is filled with inherent biases and lacks a great deal of nuance and is very unlikely to lead us to where we need to go. But by learning how to ask the right questions, how to verify the knowledge it reveals, and how to go beyond regurgitation of what it said and truly retain the insights offered, there is a path toward knowledge AI is about to make possible that could be so much better than anything we've seen before.

I'd forgotten some of the details of what exactly happened in The Diamond Age, so I asked ChatGPT for a three par summary. Within seconds it triggered the memories I was looking for. Better than scouring a search engine for a page that might have offered something remotely similar.