Building a Better Search Engine

What life beyond Google might look like

While scrolling Twitter as I do, I came across a Tweet from Jason Fried that caught my eye. He brought up the overwhelming freak of not nature that is Google. Why, when I search for the weather, do I get 1.76 billion results?

To take it back a step, when was the last time you ever looked past the first page of Google? If I’m doing that, I almost instantly know I’ve used the wrong search terms.

All the extra results are worse than useless, they are stressful. It’s disorienting to see page after page of results, all cleverly interwoven with ads.

The rest of this builds on Jason Fried’s Tweet to imagine a few alternatives to Google that might better meet our needs. In the end, we decided that the most interesting search engine is one that gives you more control, not less. One that lets you decide if you want to see ads, if you want to see 1.6 billion results or 5.

Read on to see what we came up with.

The Levers

There are two levers in a search engine:

1. Breadth of Search (“The Spirit or the Letter”)
2. Depth of Search (“How Many?”)

These two levers result in 4 different “types” of search engine:

1. Narrow and Deep (Google)
2. Broad and Deep (Chaos)
3. Narrow and Shallow (Giigle)
4. Broad and Shallow (DEVONSphere)

The Google Case: Narrow and Deep Search

Google is problematic because it assumes a narrow search with infinite depth. Which is how I ended up with 1.6 billion results when searching weather.

The Chaos Case: Broad and Deep Search

A “broader” search might still turn up 1.6 billion results, but the focus would be less narrow. I might see news about tornadoes or an article about how to “weather” the storm of life.

This type of search is the least useful — broad and infinitely deep. Neil and I called it “CHAOS”. It’s like throwing a dictionary at someone, even worse than what Google does.

We don’t think either of these first two cases are particularly useful. What we find really interesting are shallow searches.

To talk about these case, let’s start with the narrowest and shallowest case and talk about why it’s interesting, but ultimately useless.

The Hey / Giigle Case: Narrowest and Shallowest

At the end of his Tweet, Jason Fried came up with an interesting alternative to the CHAOS approach:

A search engine that returns a single result.

Unlike us, Jason Fried actually went and made his idea a reality. You can try it out here: https://hey.science/giiggle/.

The Giigle Case is fascinating because it hews more to our intent in searching. We are aiming (most of the time) to find a single thing. Siri knows this and so does Alexa. When we ask them about the weather, they don’t give us 15 different answers, they give us 1.

In the end, Jason Fried was onto something here: We don’t want 1.6 billion results. We want 1 good one.

Our Case: Narrow and Shallow

While I find Jason Fried’s narrowest and shallowest case interesting, I find the narrow and shallow case more useful. To get a picture of the narrow and shallow case, imagine a Google search that pulls up 5 rather than 1.6 billion results.

The main benefit of this type of search is that is calmer. The 1.6 billion results from Google is impressive, but stressful. The simple fact is we don’t need 1.6 billion results. We just need 1 good one.

A more murky question is whether this scaled down approach would be faster. If it were done on Google’s own servers, the easy answer is yes. But, will it be faster for a company with limited resources? The reality is that a narrow and shallow search would either be slower or similar in speed to Google’s current approach.

The Last Case: Broad and Shallow

The narrow and shallow case is great for defined searches, but what if you aren’t sure what you want? Or better yet, what if you are open to finding something unexpected?

These unusual cases call for an unusual solution: a broad and shallow search. In this case, you cast your net wide (loosely related results) but shallow (5–10 results). Google gives them impression of a broad search range, but it is simply a narrow search with tons of results. A truly broad search must be “loose” like DEVONSphere. Whereas DEVONSphere does loose, broad searches on data already on your computer to surface unseen connections, a broad and shallow search engine would do that over the whole internet, surfacing unseen connections from across the web.

Let’s take an example here. Let’s say you want to learn more about islands. The broad and shallow case would lead to 5 results like so:

1. Island Wikipedia Page
2. No Man is an Island by Thomas Merton
3. Etymology of the word Island
4. Islands for Sale
5. Darwin’s Origin of Species

A Google search, in contrast, brings up the following 5 results:

1. Island Wikipedia Page
2. Island Britannica Definition
3. 17 Most Beautiful Islands in the World
4. Mackinac Island Tourism Bureau
5. Governors Island

Plus 2.7 billion more results as well. While this Google search can help you learn about Island’s, it’s unlikely you will create any new connections or explore new avenues. Part of this is because of the ads, which dominate, and part of this is the narrowness of the search.

The best example of a broad and loose search out there today would be a Wikipedia article. Each article is shallow (short) yet contains many “loose” connections in the form of links to other sources.

This use of Wikipedia as proof of concept shows the potential utility of a broad, shallow search. Where do you go to learn new things? A Wikipedia Wormhole of course!

A Rethink of the Search Engine

In light of all these ideas about the search engine, we now have a couple interesting ideas to bounce around for a new search engine:

1. Breadth and depth controls
2. Choice of ads

Breath and Depth Controls

First, it would be fascinating to have a search engine that allowed you to choose what type of search you wanted. Google has done a little bit of this with it’s “I’m feeling lucky button”. I’d love to build on that. Both Hey and Google have built narrow, shallow searches because in certain cases they can be useful (despite my earlier claim). Imagine a search engine that can give you one result, or many.

Then imagine you can tell the search engine what you are looking for. Are you trying to find something specific, like a restaurant? Or is your search more general, like when researching for a school paper? With our “new” search engine, you’d be able to tune the breadth of the search engine.

Overwhelmed by hundreds of millions of results? You can also tune the depth. As mentioned before, you can ask to return a single result, but why not also have the ability to tune things further. Why not ask for 5? Or 10? Or 100?

All these tuning options could be done in an “advanced search” tab that keeps all the dirty details out of your hair unless you really need them. The big difference is that by default you’d get simple search, unlike Google’s giant overwhelming tangle.

As I pondered this idea, I realized it might be possible to get all this with a well designed Chrome Extension or clever website that used a service like Google as it’s backend and covered it with a wrapper. While we won’t be building anything like this soon, I will say I find it interesting to give it a shot at a later date.

Choice of Ads

Giving users all these new search control gave me yet another idea, why not allow the user to choose whether or not they want ads clogging their results? There’s a few ways this option could be presented. You could tune your search in advanced settings for a fee on a per search basis for a low cost. Or you could pay a subscription to keep the ads at bay.

The challenge of the internet is that we have become sensitized to free services. I cannot imagine paying for Google, yet the tricky thing is that I do pay for Google. I give them all of my data in exchange for free searches.

If a product is free, you are the product.

Projects like Hey by Basecamp are flipping this idea on its head for those who don’t want to be a product. If Hey can charge for email, could someone else charge for search?

What do you think? Would you pay for a search engine?