Empire Building on the Internet

So you want to build yourself an empire on the Internet.  You have a great idea for a new product, you scrape together some money, get a web site built, and open your first Internet storefront.  Time to conquer the competition.  Things start off well.  Your site makes a splash, people love your product, and they start coming in droves.  One problem: your site can't handle the load.

Waiting time for a queuing system (Brian Tung)

You've just encountered the harsh realities of queuing theory. Imagine you own a bakery, and you employ one clerk to staff the store.  The clerk is fairly efficient and takes about a minute to handle a customer.  Some customers will take longer, because they have a special order.  If two customers arrive within 30 seconds of each other, clearly the second will have to wait until the first one is finished.  If customers keep arriving quickly, a line will form.  Queuing theory says that if customers start arriving close to the capacity of the store (1 customer per minute), delays will quickly grow and waiting time will become infinite.  The remarkable thing is that this happens before the store reaches its capacity, because some customers take longer to service than one minute, and because sometimes customers arrive sooner than the average.  The same situation holds whether you run a bakery, a web server, or a highway.  (See Brian Tung's excellent blog entry explaining why this happens for freeway traffic.)

Internet caching

Clearly, if you want to build an Internet empire, you need to be able to create a scalable system, one that can handle significantly large increases in demand.  The general rule of thumb is that scaling problems are best handled using a hierarchy.  If you are the king of an empire, you can't possibly settle the requests of every commoner in person.  You need a hierarchy of lords and nobles to handle these matters for you, and as your empire grows you'll need trusted governors who can manage the far-flung outposts of your realm.  On the Internet, this translates into creating a hierarchy of caches.  The server running a large web site will often push its data into caches distributed around the Internet; each cache holds a copy of the same information the web site would normally provide.  When a customer needs to access the site, it instead uses a cache, spreading the demand across all the caches.  (Note: The same thing also happens in reverse.  Web browsers access the Internet through a cache run by their ISP, so typically a customer's cache is talking to the server's cache.)

Hierarchies of caches are everywhere in computing, because scalability is such an important trait. What role did hierarchy play as civilizations tried to "scale up" to become empires?  How did the evolution of communication affect the feasibility of these structures?