Written in 2004

Note: This page comes from the original site, written in 2004, and reflects the history as it was understood at that time.

Interestingly, it mentions a pause in the project development due to new insights and emerging technologies. At the time, I had no idea that the ideas and explorations recorded here were already moving unintentionally in the same direction as major technological trends that would emerge in the years ahead. Over the next two decades, as new knowledge and innovations unfolded, particularly in the field of AI, these early efforts provided a foundation of understanding and experience that would allow me to engage meaningfully with the changes that AI has now brought about.

History and background to the site and projects

Origins

The seeds for the ideas behind alter ego agents, organic portals and stigmergic systems were sown in 1995. It began as a fun experiment on the Direct-L mailing list (an email discussion forum for users of Macromedia's multimedia authoring package: Director). This was at the time of writing the book "Lingo Sorcery".

With the introduction of the new version of Director (version 4), there arose a need to explain and demonstrate the concept of objects and object-oriented programming. The idea came up of a program called "SpaceCruiser", which was controlled by two objects named "captainKirk" and "mrSpock" (it took the form of a Director movie programmed in Lingo).

This "SpaceCruiser" was designed to travel over the Internet (as an email attachment) and, when activated by the recipient, would seek out a prepared floppy disk named "Uranus" to find another spaceship that had a "holodeck". The "SpaceCruiser" would then copy the movie running in the "holodeck" and return back to base (again as an email attachment) where the movie could be played on the computer that had originally sent out the "Spacecruiser". Although very simplistic by today's standards, at the time it was quite novel.

This experiment inspired the idea of creating personal agents that could represent their owners and be sent out over the Internet to interact with other agents at meeting places situated on websites. This led to the writing of "Magical A-Life Avatars" - a book that explored a whole range of possibilities based upon this idea.

Using agents to insert extra genes

An important conclusion coming out of the "Magical A-Life Avatars" book was that it is prohibitively complicated and expensive to get autonomous agents to exhibit any real intelligence. This view was endorsed by the lack of evidence that any significant progress had been made over several decades by researchers working in the field of Artificial Intelligence.

However, it was realised that agents could be used as vectors to enhance human capabilities, particularly in dealing with a computer environment. This works very similarly to the way biological organisms are given behaviours by means of genes.

In effect, agents can be thought of as devices that add genes to the human genome to give people special abilities to interact with a computer system. This is a very useful metaphor, allowing the role of agents to be viewed in a more productive light.

Note: This technique can be likened to the way in which queen ants empower ants to read and write pheromone trails by giving them the necessary genes (see the tutorial "Harnessing the power of stigmergy" for a detailed description of this technique).

Possibilities for serious application

Although this had started out as a fun exercise for trivial game playing, it soon became apparent that there was much potential for serious applications. In particular, it could be used as an alternative to databases and search engines in areas where information is ambiguous, hard to define or of dubious quality. In these areas, indexing and sort algorithms are inefficient; only humans are capable of dealing with this kind of complexity. By using personal agents as intermediaries, human selection and filtering can be brought into play.

Dealing with uncertainty, incomplete knowledge and unpredictable competition was the central theme of the trilogy of books - The Entrepreneurial Web, The Ultimate Game of Strategy and Web Presence. For this class of problem solving, the approach cannot rely on designing clever computer programs to categorise and sort information; it has to be based upon devising efficient strategies for people who have problems to be able to find people who can point them to solutions. This is where the use of personal agents can play a vital role because individual human knowledge can be used to overcome the deficiencies of conventional database techniques.

Organic information spaces

The approach taken involved creating an information space, which can contain a variety of virtual meeting places where the people who have knowledge can send an agent on their behalf to give pointers to people who are looking for solutions.

It is not quite as straightforward as it sounds. Both the questions of what meeting places to create and what information might be needed are totally unknown and open-ended - conditions impossible to predict or plan for. The only practical solution was to use a system that didn't rely on planning or predesign, an organic structure that can evolve and design itself according to user needs and activity.

This led to the development of an organic, ordering and categorising framework, based upon the concept of Hilbert Space. This is a space that can "grow" its own parameters. It works much like a virtual database, where the categorisation fields are created dynamically in response to changing user needs.

The need for self-organization

Such a system needed to be self-organising and self-maintaining, to be able to run without incurring prohibitive overheads. The concept of stigmergy came to mind because this is the self-organising process that causes ants to collaborate, to tell each other where they have found sources of food. By using agent technology, this same technique can be applied to people, to help them help each other to find sources of information.

Stigmergy also manifests in a different way. It acts as a self-organising phenomenon that enables ants to build highly sophisticated nest structures. It was by learning how this happens that it became apparent that this same process could be employed to create self-organising websites .

By combining these two manifestations of stigmergy, it was possible to create a low-cost, self-organising website where people can interact with each other through agents.

Combining stigmergy with evolution

Here is where it became complicated, though. Organic systems grow and become more ordered and complex through an evolutionary process. Stigmergy, on the other hand, brings about order and organisation by means of individuals interacting with a system. These are two completely different processes, each bringing about order and organisation, but in different ways. They needed to be combined into a single system.

The solution came by looking beyond the activity of individual ants, in the ant colonies where self-organisation was taking place, to the wider system. This highlighted three separate processes:

1) Stigmergic self-organisation arises spontaneously from the activity of the ants.

2) The activity of the ants is brought about through genes given to the ants by the queen.

3) The queen has acquired the ability to give the ants the right kind of genes as a result of an evolutionary process.

This gave us a set of dependencies and an order of events crucial to the design of stigmergic systems. It told us that the evolutionary part comes first and the design of the ants emerges as a consequence of this process. The stigmergic self-organization is an end result.

It was a revelation. It suddenly became clear that we shouldn't be thinking about designing agents, but thinking about designing an evolutionary system that designed the agents for us.

Rethinking the role of a Web site

This paradigm shift led to a rethink about the role of a website. Instead of regarding a website as a stable interface between the system and the visitors, it was seen as a component of an evolving system that would be in a state of continuous change. These changes would be initiated by three separate influences: an evolutionary process and two types of stigmergy process - all three ultimately driven by the way users responded to the Web site.

The Web site creator program

A prototype for the first practical system was sponsored by the French pharmaceutical company, Sanofi-synthelabo, in 2002. This was for a website that would enable people to find available treatment trials for various kinds of cancer.

This system included all the features that had been developed previously: agents with the software equivalent of genes and an organic virtual database.

To arrange for the system to evolve and self-organise, a Web creator program was developed. This program can autonomously create a Web site from scratch and then keep changing and modifying it according to user activity. It is this ability of a Web site to change in response to user activity that brings about the phenomenon of stigmergic self-organization.

The evolution of the agents and their software genes is also arranged through this website creator program.

Critical mass

The most difficult problem to solve, for websites relying on visitor participation for their success, is getting sufficient people to participate. This is commonly referred to as "achieving critical mass" and has been the "gotcha" factor that killed off so many Internet start-ups that failed to reach this stage. This is similar to what in business terms is called the "start-up phase", the no-man's land, with the catch-22 situation where you first need to be successful to be successful.

It is self-evident that the evolutionary strategy used by biological systems is able to overcome this problem. It does so by a controlled trial and error process that inevitably leads to success.

As stigmergic system websites use an evolutionary strategy, it was suspected that they must also possess an inherent ability to overcome the problem of achieving critical mass. This prompted us to look more closely at how biological systems were able to do this. This led into the esoteric world of chaos and complexity theory, where progress is achieved by jumping between what is known as "attractor basins". This did not give us an algorithmic solution, but it provided a rational strategy that takes all the guesswork out of achieving critical mass.

It is these abstract models from mathematics that provide a mental framework to be able to see how it is possible to create a system which relies upon participation and brings about success without having to start with success. This is the basis of the strategy being used in the stigmergy research project.

A non-centralised, distributed, collaborative system

It soon became apparent, when looking at these biological type systems from an abstract viewpoint, that they could become more effective if there were no physical restraints placed upon their growth. They could evolve their own organisation, so why house them in a database? Why base them on a single website? This led to the development of Kempelen boxes - structures to hold independent software agents - that could exist anywhere on the Web, and be accessible from any number of websites.

By using Kempelen boxes, virtual systems could evolve that weren't reliant upon a single organisation, but could embrace the combined collaborative efforts of many.

The conceptual hurdle

The most difficult part of working with systems based upon biological principles is in shrugging off the concepts normally associated with computer systems. Computer systems are based upon algorithms and formatted information spaces, where order and organisation are predetermined. Biological systems, on the other hand, have no preconceived ideas as to form or organisation. Structure and detail emerge as a result of an evolutionary process.

Conventional business philosophy is based upon planning and monitored control. The idea that you can let a system self-evolve is anathema (although, ironically, businesses are perfect examples of organic, self-organising systems, especially in the early stages of their history).

Even trying to explain biological systems is fraught with difficulties. The moment you start talking about evolution, emergence, genes, positive feedback, stigmergy, pheromone trails, complexity theory and chaos, eyes glaze over. The concepts are totally alien to both computer experts and business executives alike.

To overcome the problem of explanation, this stigmergicsystems.com website was developed to bring together all the ideas behind biological system thinking.

In itself, this is unsatisfactory because what people really need is to see these ideas working out in practice. They need to see a 'proof of concept' working example.

Proof of concept example

The first step was to choose a problem area that would be suitable for a biological strategy approach. These areas are usually associated with change, uncertainty and competition. The subject area that immediately came to mind was 'search engine optimisation of a website'.

Anyone who has had anything to do with a website is aware of the intense competition to become noticed and attract visitors. With millions of websites and billions of web pages to compete against, this has attracted tens of thousands of professional advisors and consultants. How could you even begin to think about competing in such a volatile environment?

Here seemed an ideal situation to put a biological system to the test.

This 'proof of concept' example (The Example project) commenced in October 2003. Still, it immediately ran into the problem of trying to get the idea across to people who were unfamiliar with any of the concepts involved. The world of "search engine optimisation specialists" is populated by pragmatists who are trying to beat each other in a zero-sum game, and theoretical approaches seem totally out of place. Explanations of agent technology, evolutionary strategies and stigmergy left them cold.

Several attempts were made to dumb the application down by using metaphors to mask the underlying theory, but this dumbing down also masked the advantages of using a biological approach to information processing. The dumbed-down versions resembled database applications that had been given a superficially important appearance.

These failures changed the nature of the Example project, from one of trying to provide a working example to one of creating an example that would enable people to grasp the essential essence of biological system thinking.

August 2004 - New book and the Alter Ego Agent project

Developing these various projects necessitated keeping track of what was happening in many technical areas associated with this work. Of particular interest was the explosion of new research activity taking place in psychology and neuroscience due to the development of sophisticated brain imaging techniques. This is transforming the understanding of how the brain functions and the way in which the human mind works.

By the end of 2003, it was becoming apparent that these new understandings would have a major impact, not only on our projects, but also on human communication strategies in general. There was no alternative but to hold up all projects until we had a full understanding of the implications of this new thinking.

From December 2003 through to the end of June 2004, hundreds of research papers were studied to try to see where the new thinking was heading. This was not easy as the research was splintering up into hundreds of different directions. Different groups were putting their own interpretations on the results they were getting, biasing their findings towards their own particular specialist niche interests.

Robert Karl Stonjek, a moderator of the Evol Psych, Internet discussion forum, wrote, in June 2004:

"It occurs to me that Evolutionary Psychology is at the crossroads of numerous specialist disciplines. There is not just evolutionary biology and psychology, but numerous branches of biology and psychology, from memetics to genetics to psychopathy to social psychology to ethnology and so on.

I hesitate even to attempt to list all the underlying disciplines as I am bound to exclude more than I include. As consciousness is also an important issue, and consciousness is made up of at least 100 disciplines, possibly as many as 200, then the list for Evol Psych must be long indeed"

Herbert Gintis, Emeritus Professor of Economics, University of Massachusetts, was taking the view that all psychology should be founded upon the principles of game theory as only this discipline could adequately explain how the brain and human culture have co-evolved. His Santa Fe Institute Emotions Workshop, February 13-15, 2004, presented 33 papers that gave credence to this proposition.

Of most significance were the many papers describing how so much of biological evolution and human behaviour could be explained in terms of complexity and chaos theories. The influential work of Walter J. Freeman at the Neurophysiology Lab at the University of California at Berkeley was leading the way in showing how much of the workings of the brain are based upon the characteristics of chaotic networks and attractor basins.

The conclusion reached in July 2004 was that further progress could only be made if all this knowledge was incorporated into our development work. To do this, the vast assortment of information had to be ordered and focused on a particular goal.

To achieve this end, it was decided to spend six months assembling a reference resource to cover recent thinking and research relating to the new understandings about the brain and writing a book about it. The focus of the book is on applying this knowledge to information technology.

Second Life

Alongside writing the book, a new project was envisaged - to utilise this knowledge, aiming at creating "Alter Ego Agents" that would be able to represent people on the Internet, with a view to finding contacts and exchanging information.

The opportunity to take this development to a conclusion came when it was realised that the unique environment of the artificial world known as Second Life would make it possible without incurring prohibitive development costs. Work started on a project to provide avatars in Second Life with an emotional "brain", to enable the owners of these avatars to find compatible associates for a variety of personal reasons.