Google hides search result URLs behind an opaque goto redirect

I was suspicious when they started obfuscating URLs in their own browser, then on their SERPs, and now this...

For many years, I had my filtering proxy rewrite the URLs in the way mentioned in the article.

Almost exactly a year ago, Google stopped working without JS. I stopped using Google.

Now they're upping the game, and as the article (which is a bit of marketing itself) admits, those who have the resources can still blast through these obstacles while those who don't are locked out.

Since the article brings up "AI scrapers", I'll just point it out as being the latest scare-tactic for coercing people to give up the privacy, anonymity, and (browser) freedom of an open interoperable Internet.

I did an interview with Google around 20 years ago, where they posed a challenge involving tracking which specific search results people click. It's obvious in hindsight the solution required rewriting all the urls to redirect through their servers. Note this was in the days before they already did so as a matter of course.

I failed to gain traction on the problem, because to me the very idea of doing such a thing was too reprehensible to seriously consider. It broke an unwritten contract between the company and the user's expectation of how websites worked. You expect to be able to do things like right-click a link and copy the authentic URL, or hover to see where it wants to take you. The notion of obfuscating the link beyond easy recognition and polluting it with tracking markers felt misleading and, well, evil. A move that would mainly only benefit Google, and not it's users. I (quite mistakenly) presumed this opinion would be obvious and self-evident to anyone who spent enough time around the early web to understand its norms.

I explored other ways of achieving the goal, but it clearly wasn't the answer the interviewer sought.

I'm more seasoned now, and experienced enough to say with confidence the approach was wrong. This may seem like a small thing, but a series of misteps and chronic failure to adequately advocate for users is what has led us to the toxic waste dump that so much of the Internet has become today.

I'm really glad to have fresh alternatives (like […]

While a lot of people are concerned with local model performance, I wonder how feasible is it now to run a local indexed web search? Surely running an old school Google is possible with the beefy AI rigs today. I know the problem will be crawling which would be bottlenecked by the ISP but I use Google to search SO, Wikipedia, programming language docs, Github issues, and AWS docs. I think a feasible workflow would be to build a set of sites of most interest to you and then prioritize those in crawling.

While typing this out I remembered https://en.wikipedia.org/wiki/Google_Search_Appliance which I never personally used but shows feasibility for the idea. I'm pretty sure one of the newly-announced Macbooks is more than up to the task of matching GSA's offering.

Direct URLs in Google search results have been replaced with redirect URLs in the form of www.google.com/goto?url=<opaque base64 string>.

The base64 data appears to consist of a very basic protobuf structure, containing a long string of bytes in field 2 which presumably identify the URL.

Sometimes, these redirect URLs take a perceivable amount of time to load, which is very irritating.

So why are we angry about that ? I mean the end result for the users are exactly the same, it matters only for bots.

Google have such a (justified) bad reputation that whatever they do, people assume it’s entishification. I don’t believed it is on that matter.

I've been using DuckDuckGo for years now, ever since it became noticeable that two different people searching for the same search term would get two different results back from Google. Meaning they were no longer completely reliable: they might show one person a result that they hide from the other person by burying it on page 3 where few people ever look.

DDG's search results have been poorer recently than they used to — I often see completely unrelated results (to the point of my saying "Why in the world did that come back as a search result??!?") starting from page 2. And yet, I still use them, simply because they aren't Google.

At this point, we can all just drop SEO [0]. We are writing content for a robot that hides the source of information.

[0]: https://news.ycombinator.com/item?id=49665572

It's sad that instead of searching things other people put up, we're basically asking sam or dario oracle to tell us the truth. The people should be furious. But we've internalized this idea that they are somehow better.

When Google stopped paid API search a few months ago, I looked for an alternative for my agents that I felt would be sustainable (one-time setup, then out of my mind). I quickly excluded SERP as I feard Google would pull exactly this type of shenanigans to cut them off.

I somehow found Mojeek and settled on it. I had never heard of them. Unlike Kagi, their business model is ads (so they hold no particular moral high ground). But they have a cheap, working paid API.

What I was astonished by is the quality of the results. For my uses, it's undistinguishable from Google. The conventional wisdom is that web search is a Google-sized problem. How did those obscure Brits pull it off?