A metaphor

It may be best to start with some kind of mental picture - a metaphor - of what it is we are trying to do with stigmergy.

Imagine all the information in the world, including copies of all documents on the Web, being sent to a huge library, where it is placed randomly upon shelves. Imagine the staff at this library struggling unsuccessfully to catalogue the ever increasing mounds of information, managing to deal with only a small percentage of the documents and having no time to check the quality or see if any of the information is out of date. The following illustration is a diagram of this situation.

The optimum strategy

What would be the best strategy to search for information in this library?

You'd probably look at the cataloguing that had been done. You might walk around and search randomly in different places. Fairly soon, you'd come to the conclusion that this was a hopeless task as you'd never be able to search through all of this information in a single lifetime.

The thought might then occur to you that everyone else would be in this same position. There would be many others looking for similar information to yourself. You could perhaps collaborate with them, to share what knowledge you had between you.? This might prompt a visit to the cafe in the library, hoping to meet someone there with the same interests as yourself.

Now imagine going to this library cafe and finding its size is on the same scale as the library, a gargantuan room, thronging with thousands and thousands of people. It would be as hopeless here, for finding information, as it was in the library - because you can't go around asking everyone in the room, if they were interested in the same subject matter as yourself. If only you could find some way of getting all the people interested in the same subject matter as yourself into one corner. Then you could all get together to exchange information as to where useful and relevant information could be found.

Now, think of the scene - thousands and thousands of people wandering randomly around the room looking for others who'd found information that would be of interest to them. Doesn't it call to mind a swarm of ants, randomly searching a landscape looking for sources of food? The difference is that ants have solved the problem of how to communicate in a way that allows them to share their knowledge.

By being able to understand how ants pull off this clever trick, we can use the same principles in a system to help people collaborate in finding information in niche subject areas.

The hidden constructor

What is it, most people who play around with artificial ants seem to forget when they build their systems?

Step back a little from the observed activities of a real ant colony. Apply a little Zen like thinking and ask yourself what it is you are not seeing. What is behind the activities of the ants? How did they get created to be there and do the things they do? What is the engine that gets it up and running?

Then, in true, Zen like fashion your mind focuses on something you can't see: the queen ant. This is the constructor of the system. The worker ants are simply instances that she has created to perform functions to aid the colony's survival.

It is by emulating the functions of the queen ant, rather than those of the worker ants, that leads to the harnessing of the powerful potential of stigmergy.

Do ants use databases?

Imagine an ant colony using a database to record the whereabouts of food sources within the landscape of its environment. It would have to be located centrally to the system, so, logically it might be a functional part of the queen ant's responsibilities.

Even if the ants were provided with the equivalent of an internet connection to the queen, it is quite clear that there would need to be some very sophisticated communication, storage and information processing facilities employed. As different queens will be working in different landscapes, they would need to be able to build up a picture of their local landscape from scratch.

Not only would a queen have to be able to map the sources of food within her environment, this information would have to be continuously updated as food sources appear and disappear. Thus, she would need to be able to handle a continuous flow of messages from thousands of worker ants in the field.

Fortunately for the ants, nature has evolved a far more efficient way of achieving the same results.

The ingenious solution

The queen ant dispenses with all the bother of having to maintain a complicated and time consuming database by simply providing each ant with genes that enable them to read and write pheromone trails.

In this way, the worker ants can write their food searching routes into the landscape, enabling other ants to follow these trails to any food sources they find. Trails to food sources are identified from stronger markings and the updating process occurs automatically as a result of evaporation. Simple, efficient and effective.

Four essential features

Four essential features of this system are important. Firstly, the organization and methods emanate from the queen. She provides the functional tools for the ants. Secondly, the ants have no need to communicate with each other directly. Thirdly the activities of the ants are not controlled or restrained. Fourthly, the landscape acts as a database, carrying the information that allows individual ants to transfer knowledge about the location of sources of food.

The trick now is to abstract the essential elements of this clever system and use them to create an efficient system for humans to work in the same way to tell each other about the location of information sources.

Treating ants as black boxes

To abstract the essence of the ant system, it is necessary to remove the ants. This is done by treating the worker ants as black boxes.

These black boxes can then represent anything that can accommodate genes. In this way, you can ignore everything about ants except for the genes - which are providing an unknown entity with the ability to read and write pheromone trails. This unknown entity can then be any entity you choose: insect; bacterium; dog; cat; elephant or even a human.

This may seem odd, but it is a common experiment in biological research to switch genes between different kinds of organisms. Most well known are the many experiments being carried out with genetically modified food crops. A more bizarre example is where the genes of a jellyfish were transferred to mice embryos. When the mice were born, they carried the jellyfish gene in their own genes. Under fluorescent light, all their major tissues and organs - including skin, bones, muscles, lungs, liver, kidney, stomach, brain, and retina - emitted a green glow. The trait became a permanent feature of the mice's genome and was passed along to many of their offspring (Note: there is an article in the 'References' section that describes this experiment).

Not so spectacular, but very similar, are the very many medical treatments that involve inserting genes to compensate for genetic deficiencies. Bacteria also regularly exchange genes, as an evolutionary strategy. So, the concept is not so strange as it may at first appear.

This simple abstraction allows you to think of the system of ants as an object (the queen) that adds genes to black boxes.

A further abstraction

To be able to make use of the stigmergic system used by the ants, we need to take the abstraction further. We need to remove the queen and the genes and substitute abstract equivalents.

The queen we can represent as a provider (of functions) and the genes as modules that allow the black boxes to perform functions. In this way, we can describe the stigmergic system as a provider that provides individual entities with the wherewithal to perform functions that facilitate the exchange of information.

This description gives us a generic system that can be applied to a variety of different situations. For an ant colony, the provider is the queen, who provides ants with genes that allow them to read and write pheromone trails to lead each other to sources of food. We now have to use the abstraction to create a system for humans, whereby humans are provided with functions that can lay trails to guide each other to sources of information.

Applying the abstraction to a system of people

From the abstraction, we can apply the system used by ants, to people, by using computer programs and modules of code.

The provider takes the form of some kind of application program that creates functional modules people can use to read and lay trails to sources of information.

Transferring genes to people

Trying to emulate a queen ant's method of equipping worker ants at birth with the necessary genes to lay pheromone trails may seem to be beyond our ability. However, it is not beyond our ability to provide functions to humans when they access a Web site.

Humans already employ the equivalent of extra genes, simply by using a Web browser. The browser provides them with extra genes - in the form of code modules - enabling them to read information on a variety of Web sites and display it on their computer screens. The code modules, incorporated in the browser, can easily be extended: by adding new code modules embedded in a Web page.

Effectively, this is like extending the genes of a human to be able to lay trails in an Internet environment - in the same way as a queen ant extends the genes of a worker ant for it to be able to lay trails in a geographic landscape.

Using the Web to deliver functionality

The diagram below shows how a computer program can deliver modules of code - for reading and writing trails to information sources - to thousands of people via a Web page.

Notice there is no database involved here; the program creates a Web page, which has code modules embedded into it. These code modules are transparently transferred to a user's browser whenever a user accesses the page. Just as a queen ant is not connected directly to the worker ants, the program that supplies the functional code to the people has no direct connection to them.

The significance of this lack of a direct connection is that it allows people to do whatever they like with the functions they are provided with. In other words, there is no procedural control imposed upon the users; they can use the functions in any way they choose.

This may seem a rather strange idea to most people used to designing conventional computer systems, because it is customary to remove the uncertainty about how a system might be used (by guiding user actions and limiting their options). In stigmergic systems, these kind of restraints are kept to a minimum.

The paradigm shift

Like so many aspects of biological systems, the concept of stigmergy is not intuitive. It is about the interaction of individuals with the environment or landscape. It is NOT about the interaction of individuals with each other.

The subtlety and power of stigmergy is that the complexity, in the form of the sum of all individual messages, is in the environment (or landscape), NOT in the individuals.

It is difficult to get your head round this at first. It is counter intuitive and most people are fooled into thinking that a stigmergic system should involve the bestowing of intelligence or processing power into the individuals of the system. It is not. The key to the design of a stigmergic system lies in putting the intelligence and processing power into the environment. This requires a very large paradigm shift: a switch into a new and unfamiliar conceptual area.

Providing a landscape

When it becomes clear that the environment, or landscape, is an essential part of a stigmergic system, a means must be found to provide one. Ants don't have this problem because they can use an existing, physical environment in which to lay their trails. In a computer environment, a landscape must be provided by the program.

The advantage of having a computer generated landscape is that it can be programmed to contain all kinds features that will facilitate more sophisticated exchanges of information. It can be formatted into areas or categories. It can be given features that channel feedback from the users back to the program. It can be used to monitor how the system is being used, enabling the program to react to the overall performance of the system ( by making changes to the Web page code modules and the design of the landscape as and when appropriate).

In other words, the program, the Web page code modules, the landscape design and the user activity can all be integrated into a single interactive dynamic structure that adapts and changes according to the way in which the system is being used.

The stigmergic system can then be thought of as a dynamic, complex system, where the functions and the environment are changing and adapting independently, as they respond to the unrestrained activity of the users as they use the system in ways that suit their individual purposes. This requires conceptual models that extend far beyond those normally used in the design of conventional Web sites.

Note: the programmable landscape is created by the stigmergic system program to be a changeable part of the Web site. It is not part of a backend database. The reason for this will become clear later in this tutorial.

Creating a system

The diagram below outlines the main components of a stigmergic system. It includes the landscape as an interacting component of the design. Take a close look and try to figure out how it will work.

Notice individuals do not interact with each other, but with the landscape. They make changes to the landscape and these changes influence the way they react to it. Feedback from the landscape influences (through the program) the function modules that are delivered to individuals and also the structure of the landscape itself. All the components in this system are interdependent, where changes by any one can influence not only all other parts of the system but itself as well. This is what is called a complex dynamic system.

Conventional Web site design cannot cope with this kind of system. It takes a special kind of thinking. The system cannot be pre-designed, it has to be "grown" as a result of the program responding to user activity as they react to different rules and organizational structures that are introduced into the environment.

This is the non intuitive step: the program is able to introduce rules and structure after the program is designed. This is totally different from a conventional program, where the rules and structures are predetermined and form the underlying, fundamental basis of the system. Here, the rules and structures are simply spurious components, which can be introduced, changed or discarded at will.

As no predetermined restraints are imposed on the users, they are free to use their own initiatives, to shape the system in ways that provide them with maximum personal benefits. This contrasts sharply with robotic simulations of stigmergic systems where the robots (or cognitive agents) are confined to using only the limited number of functions they are provided with by their designers.