Who Killed Gopher?

An Extensible Murder Mystery

By Rohit Khare // December 23, 1998

As best our forensics team can reconstruct, a serial killer first surfaced in the beginning of 1991, born of an academic's midnight hack gone awry. NeXTstep users, you see -- precisely the kind of object-oriented revolutionaries that would think they could get away with inventing a brand new protocol for a mature Internet.

The first victim was the campus phonebook: it dismembered the whole corpus into tiny bits of `hypertext', left hanging together by search strings alone. Here we see its modus operandi: not content to slice information into neat lists or trees, it left behind completely unstructured graphs littered with redirects hither and yon across the Internet to co-conspirators' servers.

Hegemonic fantasies drove it to assimilate more and more multimedia formats at each node. Crude at first, shoveling any old file through an 8-bit clean channel, it grew more explicit until it could label text, images, audio, even compound documents. By now, it can even masquerade behind several renderings, negotiating whatever face -- or language! -- pleases the victim.

As a protocol, it was a blunt weapon: aim a socket at the target, send a one-line demand, and slurp down the response until the connection died. The crazy way it grew from there is proof itself that the killer dropped out of Application Layer Protocol design school at an early age it shows no understanding of the basics of TCP, botching handshakes, slow-start, server-sends-first-byte, even tripping over the Nagle timer. Instead, in a stroke of streetwise punk genius, it stayed 'simple' enough that any two-bit Telnet client could take a hit.

And what an addictive product it was! Workgroupies were seduced by the ease of installing new servers and authoring new content -- and pushing free software helped the lock-in cycle along. The killer preyed on individual users' self-esteem by promising instant fame*, in the form of a backlink from the master list of servers.

*Anyone have firm attribution for the famous quote, "On the Internet, everyone will be famous for 15 people"?

Soon, it was assimilating more than bags of bits: it was a gateway drug to harder services. Scripts, database lookups, even long-established news, file, and email access transactions were reduced to mere documents in a menu. The address sent in the 'demand' grew into a miniature RPC.

All the while, with every new user, every new server, every nugget of information it wrested from 'legacy' protocols, the killer bled off a little more of the Internet's bandwidth -- getting away with murder with every handshake!

Such profligacy is no idle threat: for a while, the killer was the largest share of application packet traffic on the national NSFNET backbone.

Or was that the victim?

Did the killer come knocking on port 70 or port 80? From boombox.micro.umn.edu or info.cern.ch? Bearing the imprimatur of campus administrators or physicists? Are we hunting gophers or spiders?

Dial 'H' For Murder

In fact, the profile in Table 1 fits both suspects to a 'T'. In the early '90s, an interesting drama played out between two contenders for the first integrated, interactive internet information retrieval protocols -- neither of which, arguably, had any technical reason to exist.

That's not to say the systems weren't inevitable: everyone suspected there'd have to be an easier way to use the Internet in the '90s than UNIX shell experience. That encompasses innovations in user interface, authoring formats, and above all, resource identification. It does not, however, motivate the original editions of the Gopher and HyperText Transfer Protocols. They were painfully trivial hacks that did nothing FTP didn't already handle.

So here begins our mysterious tale: why either protocol ever rose to prominence in the first place; and how the fratricidial drama eventually played out. Understanding how HTTP killed gopher may lead us to the forces which may, in turn, topple HTTP.

Science proves a blind alley, though. Their fates were decided not on technical merits, but on economic and psychological advantages. The 'Postellian' school of protocol design focused on engineering 'right' solutions for core applications (batch file transfer, interactive terminals, mail and news relays) anchored in unique transport layer adaptations (slow-start, Nagle timers, and routing as respective examples). Our two specimens are 'post-Postel', in their details and in their adoption dynamics. They are stateless; they don't have (Gopher) or dilute (HTTP) the theory of reply codes; they scale poorly, imperiling the health of the Internet; and they are 'luxuries' for publishing discretionary information, not Host Requirements which must be compiled into every node.

Burrowing into Gopher

While the Web was arguably invented a few months before Gopher, the latter was the first to garner public notice. By the fall of 1994, RFC 1689's census of information retrieval tools estimated there were 4,800 Gophers, and 1,200 anonymous FTP archives, but only 600 Webs. Early and "explosive" adoption had a lot to do with Gopher's "fiercely simple" design, and hence its wide availabilty on a number of students' (DOS, Windows, Mac, NeXT) and campus IS departments' (UNIXes, VMS, VM/CMS, MVS) platforms.

It was designed by the microcomputer support folks at the University of Minnesota for a teletext-like Campus-Wide Information System (CWIS). The original prototype was a multicolumn tree-browser pioneered for the NeXTstep file browser. Each scrolling column had a list of titles; clicking on an entry would either display that file in the pane below, or load the child directory listing in the column to the right.

Naturally, the messages on the wire follow directly: open a connection to the target server on port 70; send a one-line 'selector' string; get back either the file itself, or a list of further selector lines, each prefixed by a character indicating the type of file it points to. Table 2 lists the rudimentary type system Gopher shipped with, illustrating on one hand its ambitions to assimilate new media types and on the other what a thin skein it was around protocol- and platform-specific data formats.

The flip side of 'thin skein' is 'ease of implementation', though. In effect, Gopher was a (very!) poor man's version of FTP, sans authentication, navigation, authoring, or bulk/restartable transfers. It only used TCP as a half-duplex fopen() call. It just happened that some of the "files" it transferred were menus, in a Gopher-specific, tab-delimited, US-ASCII format. So a Gopher server did little more than read files out of a specified directory tree; and clients could be hacked out of spare Telnet client code.

A little further up the evolutionary tree, servers allowed scripts and other programs to be served as well. Rather than copying a file back across the socket, it connected it to some process's output. Similarly, multiprotocol clients could recognize a non-Gopher selector and switch to another language (as for phonebook queries). "Gateways exist to seamlessly access a variety of non-Gopher services such as ftp, WAIS, USENET news, Archie, Z39.50 (1992 rev), X.500 directories, Sybase and Oracle SQL servers, etc."

The base protocol didn't leverage TCP well, though. For example, TCP's three-way handshake ends with a packet from the server to the client -- so even when the client opens the connection, the server can send data first for free. Gopher did not have a server-hello message, and thus no version number to key upgrades off of. To signal the end of a particular transmission, Gopher borrowed the SMTP/NNTP convention of <CR><LF>.<CR><LF>. This only worked for its own menu format, though -- as soon as an actual file was transferred, TCP FIN had to be used as the EOF.

This left lots of socket buffers in the FIN_WAIT state at the server side, just like the Web. At the same time, the Gopher Team was planning to scale to Internet-wide use. They did plan for redundant backup servers: a '+' line was an alternate for the selector on the previous line. They maintained a master list of publically-accessible servers, which was in turn indexed regularly by Veronica (Very Easy Rodent-Oriented Net-wide Index to Computerized Archives, by analogy to the Archie FTP server index).

In the long-run, though, no one ever plastered Gopher selectors on shirts and lunch trucks and golf tees...

Spinning the Web

"The key insight I credit Tim Berners-Lee with, is the URL: the idea that there's a Uniform Resource Locator that says I can point at any bit of information on the Internet." -- some obscure Web scholar in Stephen Segaller's new book Nerds 2.01: A Brief History of the Internet

If anything, Tim's original Web manifesto was even more ambitious than Gopher's. Its hook for assimilating "the entire universe of network-accessible information" was the URI, which is able to subsume entire namespaces as merely one more scheme (telnet:, mail:, and so on). URIs reduced operational instructions on how to access a resource to a declarative address. For example, contrast the one-line ftp: URL with the MIME message/external-body part for FTP access -- logins, passwords, alternate sites, alternate directories, alternate formats. As RFC 1630 intended "URIs may, if necessary, be passed using pen and ink."

But while URIs were the innovative essence of the Web, the other two legs of the triad were less steady. HTML is merely a markup language which makes it natural to embed URIs as hyperlinks. URIs could just as well have been popularized as a pointer format within Rich Text Format, PostScript, LaTeX, or any other language. HTTP is merely a lookup protocol which makes it natural to resolve http: URLs or gateway to other schemes. URIs could just as well have been popularized as a naming interface to existing clients for FTP, Gopher, News, etc.

Somehow, though, HTTP/0.9 took root. It is a famously 'simple' protocol, arguably even less useful than Gopher. Here is its entire grammar:

Simple-Request = "GET" SP Request-URI CRLF
Simple-Response = [ Entity-Body ]

That's right: it only supported one method, GET, and required a new TCP connection for every transaction. Even the 1.0 edition, which eventually added authentication and navigation and simultaneous transfers of several files for a single "page", had no excuse to be born when FTP was on the shelf. Except that HTTP had broken authentication, nonstandard conventions for navigating directories and welcome/index/home pages, and ran spectacularly afoul of slow-start for each of its typically-small file transfers.

If HTTP/0.9 was really shipped for no better reason than that programming separate FTP-Control and FTP-Data channels seemed too hard, why did it survive? Many of the earliest "web" sites were, in fact, just ftp: URLs. The saving grace was 1.0's new metadata headers. It augmented the one-line request with additional fields for authentication, arguments modifying the method, and information about the user-agent's preferred languages and formats. The entity bodies it returned were prefixed by MIME Content-Types, last-modified dates, base-URI, and more. This information, finally, was what took HTTP beyond the realm of file transfer to hypertext transfer.

The headers are how HTTP's adoption cycle diverged from Gopher's. Both servers were simple to install and author content for -- there was even a hybrid gn server which could vend files over both port 70 and 80. There were kudos for new sites, in the form of the Web Virtual Library project at CERN, and later W3C, and early search engines like Brian Pinkerton's NeXTstep DigitalLibrarian-derived WebCrawler. The client was the defining wedge. Graphical web browsers leveraged rich HTML text with MIME media type headers to display graphical images, to launch helper applications, and present multilingual content.

Extensibility: the fingerprint of a killer

The straw that broke the Gopher's back wasn't even a feature of HTTP per se. The synergy of HTML and HTTP in the graphical browser began with FORMs, through refresh META tags, up through scripting and applet OBJECTs. While one protocol stayed lean, the other opened up so much headroom that it grew into a beast so complex even today's base 1.1 specification takes 160+ pages.

An avalanche of new headers modified GET to only retrieve the full entity if-modified-since an existing copy, or only a specified byte-range; enhanced security with keyed-digest passwords; even reverse-engineered state back into HTTP using 'cookies.' HTTP reply codes were added for payments, for cache validation, for upgrading to new protocols.

Entirely new functionality could be grafted on HTTP using new methods. WebDAV added LOCK and PUT; collections which allowed MOVE and COPY on entire swaths of URLspace; and even went beyond new headers to encoded additional parameters in XML. Conversely, HTTP was borrowed wholesale for new protocols for printing (IPP), multiparty call setup (SIP), and event notification (RVP).

Politically, any server or client could inaugurate a new method; third-parties could even thread themselves into the proxy chain and filter messages in transit. Proxies exist to add public annotations (Crit-Link Mediator), anonymize access (Lucent Personal Web Assistant), convert file formats (ProxiNet for PalmIIIs), language translation (AltaVista's BabelFish can be so hacked), and content filtering (PICS). Decentralized extensiblity was key to HTTP's adoption. Rather than waiting for the IETF standards process to revise a canonical table of one-letter content codes, for example, MIME headers could deploy new content-types on the spot.

At the same time, uncontrolled divergence reduces the value of the Web platform as the probability of compatible extensions matching up declines. New versions of HTTP would appear to be one solution, but there is little political will to empanel a standing working group to review a grab-bag of extensions for 1.2, 1.3, ... 39.7 and so on. What is neccessary? Some hook for identifying extensions, the level of support required, and which parts of URLspace are so extended.

W3C has been promoting such mechanisms for years -- three calendar years! -- to better "Lead the [Controlled] Evolution of the World Wide Web." At first, the motivating scenarios were complex economic and social negotiations (payments, content selection, privacy, cryptographic algorithms, &c). Early revisions of PEP advertised preference lists of alternative extensions, applied in pipelined sequence. PEP could also transport metadata about which extensions applied to other resources and required orders.

Though PEP remained a W3C experiment, the basic ideas of 1) packaging an entire extension -- methods, headers, encodings and all -- behind a single name and 2) indicating clearly if an extensions was required or optional for 3) the next hop or end-to-end were reincarnated as the Mandatory proposal. The current draft defines four header fields enumerating the URIs of the extensions which apply (Man: and Opt: for client and server; C-Man: and C-Opt: for the next hop). To interoperate with the installed base, any request with mandatory extensions prefixes the method name with 'M-', which forces 1.0 and 1.1 servers to fail with "501 Not Implemented."

Transport: the Achilles' heel

By April 1995, Web usage passed Gopher's share as well as every other Internet application protocol as a fraction of NSFnet backbone usage. Even patching its bandwidth leaks with 1.1's persistent connections can't mask HTTP's inefficient waste of Internet resources. Its content base has grown so immense that caching alone can't reduce bandwidth demand by a significant margin for the average user or ISP. Former strengths have now become liabilities: full 'English' headers, stateless transactions, full enumeration of client capabilities once critical for debugging now inflate request packets while progressive rendering of composite web pages places a premium on simultaneous downloads (aggravating the share of packets wasted on handshakes and delayed by slow-start).

Individually, these phenomena have appeared in application protocols before -- and were nipped in the bud by active involvement by transport protocol folks, or coevolved with TCP. When Telnet generated a packet per keystroke, even on links where you could type faster than the ACKs returned, the Nagle timer throttled TCP down to batching keystrokes into a single packet per round-trip. When file transfers immediately swamped LANs, slow-start eased up to limit congestion.

So when HTTP presents its demand for many small transfers in parallel immediately, there are solutions on the shelf. Some vendors are already marketing "accelerators" which rely on matched encoders at server and client, like Sitara's SpeedServer. Unfortunately, these approaches aren't backward compatible with HTTP as-is -- nor as simple to code.

W3C's HTTP-NG (Next Generation) project sees it as a three-layer problem: message transport, remote invocation, and "The Classic Web Application" (TCWA). At the bottom, they propose a multiplexing protocol which combines all the traffic destined for a single web server into a single TCP connection. This allows the conversations regarding individual transfers to be diced up independently: an intelligent server might respond with the first few critical bytes of all the images in the first packet. A fully specified W3Mux layer still needs to address flow control (gating the relative priority of the subchannels without starving them), simultaneous open (which TCP does allow), and lots of interoperability testing.

The middle layer 'compresses' the data to be transferred by marshalling arguments in native data types rather than English. So numbers would be sent as integers; dates as binary fields; and strings as reusable tokens. The tip of their proposal (reaching into the cloudy heights of improbability, some critics might say) is reengineering the Web as a distributed object system, modeled as operations on data structures rather than documents.

Building the Perfect Beast

Having slain Gopher (and a host of other competitors), the Web appears to be expanding in all directions today. Just as the SMTP state machine defined a style or protocol design for a generation, every new application these days seems to ape HTTP's state-less style -- if only to piggyback a free ride through firewalls at port 80. The Internet Architecture Board is on the verge of issuing guidelines to rein in this tendency. First, the HTTP port should be reserved for protocols which manipulate the kind of information already in web servers. WebDAV is OK, since it extends documents; IPP would not, since it uses POST as a barn door for its printing procedure calls. Second, HTTP should not be referenced by-copy. Lots of other efforts want to 'borrow' a few headers for authentication or navigation and end up copying text into other specs, raising the scepter of divergent versions. Third, HTTP is not a valid precedent for MIME users -- it plays fast and loose with certain rules which enable absolute interoperablity through mail gateways. HTTP allows bare <CR> and <LF> as well as inventing a new Content-Encoding: process.

The IAB guidelines are a reminder that HTTP still occupies a limited niche in the space of possible Transfer Protocols. True, its message format can encode any MIME entity, even live data streams -- but only from the server to the client. True, its address format can point at anything -- but even then it can't encode whether the resource is expected to be secured, paid for, or otherwise reprocessed. And it is most certainly restricted to one-to-one, synchronous, request-response (client-pull) interaction. All the various hacks to 'push' data to groups of subscribers are, well, hacks: automatic polling, shared caches, and "we will mail your response within two working days".

The moral, then, for would-be sucessors to HTTP is to pick another niche and follow the network effects. Internet-scale event notification, for example, requires true push and subscriber management. Current efforts to shove presence information through HTTP may succeed, but any larger shift to event-based integration could catalyze a new, symmetric protocol which allows servers to initiate connections back to clients. Simply compressing existing HTTP traffic without adding new functionality faces a powerful entrenched competitor. Adding programmability, whether betting on HTTP-NG or the Mandatory scheme, is an attempt to harness network effects between all the communities that have been angling to extend HTTP to their application. Yet, the rewards for cooperation are spread thin: only the server which simultaneously supports HTTP, WebDAV, and Internet Printing Protocol would gain from a common syntax for those extensions.

And now we are left back where we started: the mystery of protocol adoption. Reuse, extensiblity, simplicity are all tricks to reduce the cost of implementing a new protocol -- but that doesn't mean the market as a whole will make the most efficient decision. If you're a Gopher, it may just bury you :-)

The morbid conceit of this article was inspired by the first-rate book Aramis by French historian of technology Bruno Latour. It told the tale of two decades of R&D on a Parisian peoplemover system in the first person -- the train itself speaking of its struggle to become, and its death.


Table 1. Fingerprints of a murderer -- Spider or Gopher?

Vitals

Academic NeXTstep hack, about 8 years old; birthmark at port 70 or 80

First Victim

Notorious primary use was for mere phonebooks; gatewayed queries to existing campus database and represented results as hypertext nodes

Disguise

Began without any meaningful type system at all, but soon grew past file extensions to vend multimedia information, even negotiating formats

Modus Operandi

Open a new connection, send a one-line demand for the information at a given address, then receive the response until connection dies

Come-on

Infiltrate small workgroups with small, easy, and free tools, promising acceptance with a simple `backlink' from a master list of servers

Serial Victims

Reaching beyond the degenerate case of static information, its addresses became entry points to scripts, databases, and other `stateless' servers

Motive

Even its first manifesto laid out hegemonic plans: to incorporate, by reference or by proxy, every other Internet information service.


Table 2. Gopher's rudimentary type system used character codes to prefix selector lines. Gopher+ later added content negotiation.

0

File

1

Directory

2

CSO phone-book server

3

Error

4

BinHexed Mac file

5

DOS binary file

6

Uuencoded Unix file

7

Index-Search server

8

Text-based telnet session

9

Binary file

+

Redundant server

T

Text-based tn3270 session

g

GIF image file

I

Any image file


Table 3. RFCs and Internet-Drafts discussed in this issue.

RFC

Date

Title and Comments

1436

Mar 1993

The Internet Gopher Protocol
Described a `fiercely simple' request-response protocol and menu format

1630

June 1994

Universal Resource Identifiers
Raised the stakes from Gopher's own selectors to any namespace

1689

Aug 1994

Status Report on Networked Information Retrieval: Tools and Groups
A wonderful archaeological census of competing protocols, many now extinct

1945

May 1996

HyperText Transfer Protocol 1.0
Also defined the more primitive 0.9 GET. Even 1.0 only added HEAD and POST

2068

Jan 1997

HyperText Transfer Protocol 1.1
A vastly expanded contender with better caching, security, and extensibility

2227
P. Std.

Oct 1997

Simple Hit-Metering and Usage-Limiting for HTTP
Entire RFC defined a single Meter:

2396
D. Std.

Aug 1998

Uniform Resource Identifiers: Generic Syntax
The deceptively small definitive grammar took years to hammer out

TBA
P. Std

Dec 1998

Extensions to HTTP for Distributed Authoring
Adding locks, collection-resources, and namespace operations took major reengineering

I-Draft

Nov 1997

PEP -- an Extension Mechanism for HTTP
`Protocol Extension Protocol' had negotiation policies for pipelining message processors

I-Draft

July 1998

HTTP State Management Mechanism
Cookies took three years from a quick, risky hack to a reasonably secure mechanism

I-Draft

Aug 1998

On the use of HTTP as a Substrate for Other Protocols
Official -- conservative -- IESG and IAB guidance on port assignment, URI schemes, and HTTP security

I-Draft

Sep 1998

HTTP Authentication: Basic and Digest
A companion specification for nonexistent (UUencoded passwords!) and new security.

I-Draft

Nov 1998

HTTP Extension Framework
Pared down to headers which label `Mandatory' and optional extensions of each message

I-Draft

Nov 1998

Upgrading to TLS Within HTTP/1.1
Using the Upgrade: for Transport Layer Security, in lieu of a dedicated port like https: at 443