Edited directories

by Peter Small

---

As the web continues to grow at an uncontrollable pace, the algorithms designed to automate the processes of indexing and categorization pages on the Web have become less and less accurate and reliable. However sophisticated the technology, these algorithmic techniques are woefully inefficient for many types of queries put to search engines.

Similarly, a reliance on Web site submissions - to provide search engine directories with indexing and classification information - is equally flawed. The editorial staff find it difficult to keep up with the submissions, let alone be able to check their accuracy and relevance.

For this reason, the major search engines are becoming increasingly reliant upon niche directories edited by humans. By utilizing existing, categorized sources of information, far better results can be achieved than by relying solely upon methods of indexing and classification.

You can see for yourself how easy it is for the major search engines to access information in a pre indexed form by going to any of the major search engines (i.e., AltaVista; Google; USENET; HotBot; Lycos; Northern Light; Yahoo ) and typing in "Directories". These are the sources that the search engines themselves use, when they answer your queries.

Figure 1 is a graphical illustration of this situation.

Directories produced as a result of stigmergy

Directories are the most efficient sources of information because they are usually compiled as a result of extensive human effort and intelligence. However, they can be very expensive to create and maintain, unless a self organizing, stigmergic system is used to coordinate the activities of many people who are motivated to contribute their services freely.

Two of the most successful of these stigmergic directories are RingWeb and The Open Source Directory. Take quick look at these now, to get an idea of how they work, before reading on.

Each of these stigmergic directories provides an infrastructure for individuals to take part in a communal activity to help each other locate useful information on the Web. Individuals can propose subject areas - usually as a subset of another subject within an expanding hierarchy - where they can organize collections of links to Web sites relevant to their chosen area of specialty.

This kind of organization is totally foreign to the planned business models that dominate the thinking of conventional business strategies. Yet, look at the statistics of these two successful directories:

The Open Source Directory has become a self organizing directory with 460,000 categories, pointing to over 3.8 million sites and involving the activity of 51,891 people (figures from the Open Source Directory Web site as of 17th September 2002).

Taking the published statistics for August 2002: WebRing had 63,000 rings operating in that month, pointing to 1.05 million active sites. There were 1,500 contributing members and over 750,000 registered users. The site had 3.12 million unique visitors in the month, producing 27 million hits.

These figures provide ample evidence of the effectiveness of self organizing, stigmergic systems. Not only are they effective and efficient, they are also very cheap to run and maintain. This allows these services to be offered at no cost to the user.

Differences

Although similar, these two stigmergic systems use different rules to regulate their growth. WebRing is freer and more organic. People are able to start new rings (subject categories) where they like, on any subject they like and are free to choose and regulate the links that appear in their own ring without recourse to others.

WebRing describes their system as follows:

"WebRing is one of the best ways to find great sites. Similar sites are grouped together in rings and each site is linked to another by a simple navigation bar. Rings form a concentration of sites, allowing visitors to quickly find what they are looking for. Each Ring is created and maintained by an individual web site owner called the RingMaster. RingMasters determine the look and feel of the Ring, approve and manage member sites, and encourage other sites to join. Ringmasters help to develop virtual communities based on the Ring topic. WebRing is free and easy."

The Open Source Directory on the other hand is designed to be much more regulated. Each subject area and each of its branching sub categories can attract not only submissions for Web site inclusions but also for editors to supervise and monitor the inclusions.

The editors are themselves supervised and monitored, by editors further up the hierarchy. In this way, a self organizing, twin hierarchical system evolves; one hierarchy providing organization of the subject areas, the other hierarchy providing a system of quality control. (Note: full details of the rules and mechanics of this system are provided on the Open Source Directory Web site).

Both of these systems have advantages and disadvantages, which you can ascertain by comparing the contents of similar categories. However, they both work and are nicely complementary to each other.

Clearly, these system have overcome the problems of achieving critical mass and, through the unpaid collaboration of thousands of individuals, have created valuable resources of selected and indexed information which can be universally accessed, completely free of charge.

Stigmergic genes

Ants are able to build nests and lay message trails because they have genes that prompt them to react to what they see or detect in their environment. This self-organization - or stigmergy - occurs as a direct consequence of genetic programming. It is a phenomenon that arises spontaneously out of inborn instincts.

Exactly the same phenomonon occurs with humans. This is evident whenever people come together in a group, there will always be a natural tendency for them to communicate, to work together and self organize, especially if there is a common problem or threat.

This is not the kind of planned organization that is the hallmark of corporate business - i.e., based upon intelligently predicting or calculating a future. Stigmergic organization is more spontaneous; it is about organizing the present and assumes no knowledge of the future.

Nowhere is this tendency more obvious than in the environment of the Internet, where the existence, the evidence, the power and effectiveness of stigmergy and stigmergic systems are staring everyone in the face. The popularity and ubiquity of news groups and Internet discussion forums are clear examples of the effectiveness of stigmergy to create spontaneous, self regulating organization.

The various open source movements are prime examples of the way in which humans will independently respond to and change an environment to bring about increased order and complexity. This is what stigmergy is about. Stigmergic systems harness, motivate and give direction to basic instincts.

It is also the phenomenon that generated the World Wide Web itself. Just think about the World Wide Web for a moment. This was not centrally planned or organized. It evolved from nothing. It came into existence as a result of millions of people following their instincts to communicate, cooperate and collaborate. Most of the successful organizations on the Web - such as AOl, Amazon and Ebay - have succeeded as a result of stigmercic induced participation.

The concepts used to describe conventional business models have no way of explaining stigmergic systems. The only way to understand them is to see them in action. Two of the best examples to study for this purpose are to be found in the world of edited directories: RingWeb and The Open Source Directory.

By examining and analyzing these two stigmergic systems - to see how they produce results superior to any planned technological solutions - you will be able to understand and appreciate the potential of stigmergy.

Stigmergic genes

Ants are able to build nests and lay message trails because they have genes that prompt them to react to what they see or detect in their environment. This self-organization - or stigmergy - occurs as a direct consequence of genetic programming. It is a phenomenon that arises spontaneously out of inborn instincts.

Exactly the same phenomonon occurs with humans. This is evident whenever people come together in a group, there will always be a natural tendency for them to communicate, to work together and self organize, especially if there is a common problem or threat.

This is not the kind of planned organization that is the hallmark of corporate business - i.e., based upon intelligently predicting or calculating a future. Stigmergic organization is more spontaneous; it is about organizing the present and assumes no knowledge of the future.

Nowhere is this tendency more obvious than in the environment of the Internet, where the existence, the evidence, the power and effectiveness of stigmergy and stigmergic systems are staring everyone in the face. The popularity and ubiquity of news groups and Internet discussion forums are clear examples of the effectiveness of stigmergy to create spontaneous, self regulating organization.

The various open source movements are prime examples of the way in which humans will independently respond to and change an environment to bring about increased order and complexity. This is what stigmergy is about. Stigmergic systems harness, motivate and give direction to basic instincts.

It is also the phenomenon that generated the World Wide Web itself. Just think about the World Wide Web for a moment. This was not centrally planned or organized. It evolved from nothing. It came into existence as a result of millions of people following their instincts to communicate, cooperate and collaborate. Most of the successful organizations on the Web - such as AOl, Amazon and Ebay - have succeeded as a result of stigmercic induced participation.

The concepts used to describe conventional business models have no way of explaining stigmergic systems. The only way to understand them is to see them in action. Two of the best examples to study for this purpose are to be found in the world of edited directories: RingWeb and The Open Source Directory.

By examining and analyzing these two stigmergic systems - to see how they produce results superior to any planned technological solutions - you will be able to understand and appreciate the potential of stigmergy.

Stigmergic genes

Ants are able to build nests and lay message trails because they have genes that prompt them to react to what they see or detect in their environment. This self-organization - or stigmergy - occurs as a direct consequence of genetic programming. It is a phenomenon that arises spontaneously out of inborn instincts.

Exactly the same phenomonon occurs with humans. This is evident whenever people come together in a group, there will always be a natural tendency for them to communicate, to work together and self organize, especially if there is a common problem or threat.

This is not the kind of planned organization that is the hallmark of corporate business - i.e., based upon intelligently predicting or calculating a future. Stigmergic organization is more spontaneous; it is about organizing the present and assumes no knowledge of the future.

Nowhere is this tendency more obvious than in the environment of the Internet, where the existence, the evidence, the power and effectiveness of stigmergy and stigmergic systems are staring everyone in the face. The popularity and ubiquity of news groups and Internet discussion forums are clear examples of the effectiveness of stigmergy to create spontaneous, self regulating organization.

The various open source movements are prime examples of the way in which humans will independently respond to and change an environment to bring about increased order and complexity. This is what stigmergy is about. Stigmergic systems harness, motivate and give direction to basic instincts.

It is also the phenomenon that generated the World Wide Web itself. Just think about the World Wide Web for a moment. This was not centrally planned or organized. It evolved from nothing. It came into existence as a result of millions of people following their instincts to communicate, cooperate and collaborate. Most of the successful organizations on the Web - such as AOl, Amazon and Ebay - have succeeded as a result of stigmercic induced participation.

The concepts used to describe conventional business models have no way of explaining stigmergic systems. The only way to understand them is to see them in action. Two of the best examples to study for this purpose are to be found in the world of edited directories: RingWeb and The Open Source Directory.

By examining and analyzing these two stigmergic systems - to see how they produce results superior to any planned technological solutions - you will be able to understand and appreciate the potential of stigmergy.