In 1998, a young developer named Jim Gilliam was hired at Lycos after he impressed management by finding bugs in their site. He took on the task of improving their search results to find a way to beat their biggest competitor, Yahoo.
It was exhilarating to be back in the game again, a 20-year-old college dropout with stock options, working at the center of the internet revolution. But I was overwhelmed. I’d only ever worked on a team with a half-dozen people, and Lycos was a huge company with hundreds of employees. All the developers seemed much smarter and more experienced than I was, and I was struggling to understand all the different proprietary technologies Lycos had created.
As I dug in, I realised that the scope of the problems was immense. I was paralysed. I didn’t know where to start. At the end of my first week I passed Lycos’s VP of development Dave Andre’s office on my way out for the night. No one else was around and he waved me in. He asked how everything was going, so I was honest and told him what I was feeling. With no hesitation, he dropped the most influential piece of advice I’ve ever received. He said, “Jim, you can code. You have all the power. Just go do it.” So I did.
Lycos was a search engine, and like all search engines at the time, it was trying to figure out how to make money. The key was to make our search engine into something that would appeal to advertisers. Like Excite and Yahoo, Lycos paid the browser, Netscape, to send traffic our way, and we were all trying to keep people on our sites longer, because the longer people were on our sites, the more ads they saw. Lycos’s CEO, Bob Davis, was a sales guy, and his strategy was to cut deals with new, venture-funded dot-coms and split the revenue on all the ads that we sold. We would increase our ad inventory, help the startups, and the Lycos logo would be all over the web.
I didn’t really care about all that. I cared about our search results, which seemed to be the one thing that no one was paying attention to. We were a search engine, but our results sucked, mainly because it took between six and nine months to refresh the search catalogue. This meant that even our partner sites didn’t show up in our search results, making the entire sales strategy pointless. If I could fix this, our search would be better and we’d actually sell more ads.
So I created an internal web-based tool called LINK to manage all the information from our partner sites and publish out a new, much smaller, search database every day that could be merged into the main search results. If someone searched for Los Angeles weather, we would show a picture with the current weather for LA. If the user clicked it, he’d be routed to Lycos Weather, powered by one of our partners. If she searched for Boston Red Sox, we would show her the previous night’s box scores.
The people cutting the partner deals were thrilled. My boss Dennis Doughty, LINK, and I were heroes, and our little team was promoted to be in charge of Lycos’s most important page: Search. It was now officially our responsibility to make sure that we were pushing out the best search results on the internet. Which meant that we had to take on Yahoo.
The “Yahoo Killer”
Yahoo was unquestionably the most popular site on the internet, due to its huge directory of websites curated by several hundred full-time editors. So, for roughly $US40 million, Lycos purchased a company called WiseWire, which trained machines to categorise webpages. It was the key element in Lycos’s plan to beat Yahoo. There was just one small problem: WiseWire sucked.
A few months later, Netscape quietly acquired a small company called NewHoo, which intended to compete with Yahoo by having volunteers curate the links instead of paid staffers. It was genius. Getting paid to categorise websites all day gets boring fast, but when you’re doing it in your free time on a topic that you know a lot about, it’s fun. Netscape had outflanked everyone. Yahoo was using paid editors, Lycos was using machines, and Netscape was using volunteers. I knew immediately Netscape was going to win.
Then Netscape upped the ante by opensourcing NewHoo and renaming it the Open Directory Project (ODP). A fundamentally open project with volunteers loosely coordinated around a technology platform was a revolutionary new model for getting things done on the internet. This could completely undercut Yahoo if enough volunteers signed up. But Netscape couldn’t really do much to promote it because they made millions of dollars from redirecting searches from the Netscape browser to search engines like Lycos. They would be pillaging their own revenue stream. But I realised that if Lycos adopted the ODP, people would flock to volunteer, because their work would be seen by millions, which, in turn, would feed the Lycos search engine. If Lycos relied on the internet and volunteers and chose to put their full weight behind the ODP, it would become the biggest thing out there. A directory that was broad, authoritative, and free? Yahoo would never be able to hire people fast enough to compete. And that’s how we’d beat them.
The idea caught on internally — earning me the moniker “Yahoo Killer” — but it was highly controversial because it meant we didn’t need the company we’d just paid millions for. The WiseWire folks argued against the idea because Lycos wouldn’t own the ODP. But Lycos had just acquired another company, Wired Digital, and when they got wind of my plan, they wanted their search engine, Hotbot, to use it, too. A few months later, in April 1999, Lycos stopped using WiseWire, and flipped both Lycos and Hotbot — two of the biggest search engines on the internet — to the Open Directory Project.
The press was stunned. Even Netscape was surprised. The ODP, invented by Chris Tolles and Rich Skrenta, was given rightful attention for being what it was, a revolutionary industry first. Lycos’s participation took it to the next level and the ODP started to explode. As predicted, there was a huge incentive for people to participate because, through our distribution channels, there was a massive audience for their work. With over 100,000 categories, ODP quickly dwarfed the Yahoo directory. All the information fed into LINK, which we supplemented with sites in our ad network, before pushing everything back into our search results. Having your listing on the ODP meant power and visibility. It became the most authoritative directory on the internet, and it ultimately grew to over four million sites organised by 80,000 editors all over the world.
I received a really big award at Lycos’s all-company retreat that summer.
Monetizing Search Results
A few months later, our team made a huge discovery. In our ongoing efforts to make search results better, Dennis set up an eye-tracking lab and began scientific testing of how people used search. We watched where people looked on the pages and noticed something shocking: people didn’t look at the ads. Not only that, but the more we tried to make the ads stand out, the less people looked at them. Our entire advertising philosophy was based on making ads flashy so people would notice them. But we saw, quite counterintuitively, that people instinctively knew that the good stuff was on the boring part of the page, and that they ignored the parts of the page that we — and the advertisers — wanted them to click on.
This discovery would give us an edge over everyone in the industry. All we had to do was make the ads look less like ads and more like text. But that was not what the ad people wanted, and the ad people ran Lycos. The advertiser was seen as our true customer, since advertising was where our revenue came from. Our team argued that our customers were also the people searching, and without them, we’d lose the advertisers. The eye-tracking revelation wasn’t enough to convince them, so we tried another tack.
In the ultracompetitive world of search engines, the biggest factor aside from the quality of the results was how fast they loaded. We were constantly trying to take things out of the pages to make them load faster. So I created a program that took queries coming into our site and ran them on all the major search engines, ranking them in order of speed. Yahoo was the fastest and Lycos was down near the bottom. In order to get the company’s attention, Dennis set up a computer in a main hallway to show the results in real time. We started pushing to make the pages lighter, and specifically, to make the ads text — or at least make the images in the ads much smaller. We pushed and pushed but the executives kept pushing back, telling us that that was not how the advertisers wanted it. We knew that what the advertisers wanted was bad for them and for us, but no one would listen.
Part of the problem was the treadmill of quarterly earnings reports. Lycos was the fastest company to ever go public. We were a publicly traded company in which all the employees had stock options, so the mood of the company rose and fell along with the stock price. Every quarter you tell shareholders how much profit you made. If it’s less than they think you should have made, the stock price goes down, employee morale goes in the toilet, some employees leave for other companies, and the company descends into a death spiral. So we had to hit not our own quarterly revenue targets, but the press’s quarterly revenue targets — no matter what. Ideas that could help the company in the long term but that might have a negative impact on quarterly earnings were shot down. Which meant that we had millions of people coming to our website every day and telling us what they wanted, but we were giving them something else. It was only a matter of time before the treadmill killed us.
Then a new search engine came on the scene that was getting a lot of buzz. Less than a year later, Google released its ad product — and it was all text. They had seen the same thing our team had seen in the eye-tracking lab, and because Google was data and engineering driven, they made the right decision. While we were focused on giving advertisers what they wanted, Google told advertisers what they needed. Google knew that people would be more likely to click on ads relevant to their original search queries. If someone searches for “Red Sox” they might click on ads for tickets or scores, but they won’t click on a beer ad, no matter how flashy it is, because it’s not what they want right at that moment.
This was the exact opposite of what the advertising world thought, because they were still operating with a television mindset. But Google was willing to trade the short-term for the long-term, and they owned the industry as a result. They had figured out how to align the interests of the user and the advertiser. Advertisers weren’t allowed to buy ads that weren’t useful on searches, which was great for users, and advertisers no longer had to pay for ads no one was clicking on. When Google launched its ad program, it was the beginning of the end for Lycos.
This is an excerpt from Jim Gilliam’s new book, The Internet is My Religion, which you can download here for free.
Jim Gilliam is the author of The Internet is My Religion, and the founder and CEO of NationBuilder, the software platform for leaders.