Chapter 13
Stigmergy
Two ways of looking at a database
In the previous chapter, I thought I'd done enough to explain the mechanics behind the business situation I was developing. It was with some surprise when one of the readers expressed disappointment with the explanation I'd given. He wrote:
"I still think that what I brought up back in chapter 10 is missing. Basically, my point is that a non-negligible challenge in Peter's proposal is to establish and maintain (or rather, let evolve) an appropriate framework. Even if smart clients are installed on users' computers, to let them process their own set of attributes, the main data needs to be properly qualified first.
In Peter's matchmaking algorithm, 96 different Boolean attributes (they could have also been dimensions with fractional values) are used to model each individual. Adding, modifying or deleting individuals from the entire distributed network does not represent a big challenge. But how can the system be made to accept new attributes and to reject obsolete ones?
Worse than obsolescence, redundancy has to be dealt with. Not because it wastes data space, but because it leads to inconsistencies. Now, this has more to do with politics than technology.
In my previous posting, I mentioned two existing approaches: - The Yahoo approach: regardless of who is feeding the data, there is a small committee of designated people who constantly monitor the system and modify the framework accordingly - The Newsgroup approach: self-designated users creating enough momentum to let the framework evolve one way or another. Regardless of how dictatorial or anarchic Peter's recommended system is, it has to tackle this political issue.
Vahé Kassardjian
This chapter, I hope, will answer most of Vahe´ Kassardjian's reservations. Firstly, it must be pointed out that chapter 12 dealt with a very general overview of the idea. If it is to be applied to a specific situation for an e-business purpose, there would have to be an element of human control over the main categories of the people space. This will be explained later. However, the other difficulties envisaged would apply only if a server side database were used in a conventional way.
At the same time as I received Vahé Kassardjian's post, I received a post from another reader, Kate Cooper, who has a senior fellowship at Warwick University – running their Innovation progammes for various MSc degrees in Engineering Management. Kate is not a database technologist but has been a consultant to UK blue chip companies since 1983, mostly advising on human communication in large organisations. She saw the beyond the nuts and bolts of the technicalities to visualise databases as part of an interactive system of communication. She wrote:
Šthink Experian/Acorn and how they're used by businesses - turn that on its head and think how individuals can use the same information . . . now *that* is maybe a lucrative idea!!!
<snip>
Š enable a user to latch into a known psychological construct - there are some human universals. (e.g. an understanding of hierarchy is universal, probably because of family relationships, but knowing about Cinderella is a parochial). A hugely pervasive and successfully exploited construct is that of a 'journey' - already used of course on the Net with terms such as 'path' and 'navigation tools'. There are many others that could be used - see stigmergy later. Also metaphors to assure people - signposts, junctions, fastroutes, superhighway, meandering path, etc. etc. PLUS you are, I presume, familiar with the social network stuff, 'small worlds' -
<snip>
ŠI have a theory that stigmergy trails (as I think of it) within an Internet system might identify the equivalent of the mavens, connectors and salespeople that Gladwell talks of - now imagine that the *geometry* of such trails were evident to the observer of this hive mind *and* that an individual could be both participant in the mind and observer of it. Some kind of stigmergy-driven search engine???
<snip>
An aside: humans find it very difficult, if not impossible to 'see' the dynamics of a system. So we create geometries to help us - Poincare's 'phase space' - and, as highly visual primates, that suits us. The only 'geometry' of the Internet that most of us have is 'bricks and mortar' - we need, obviously, a new model that reflects the dynamics - hence my 'stigmergy' idea in which the rate of change of the trail matters more than the trail itself; fashion fashions itself. That we can handle these kinds of dynamic geometries as representations is evident in our nightly view of the TV weather forecast . . . Whatever, the Internet could do with a geometric model . . .
Kate Cooper
Here was a view from somebody who claims to be non technical but is an expert in business systems. Kate's mental model wasn't influenced by the technicalities of database design, she was seeing a more abstract representation that eliminated all the problems anticipated by the database expert.
The two key models that came into Kate's mind when she read the previous chapters were Experian/Acorn and stigmergy. Let's take a look at these.
The Experian/Acorn model
These are models well known to the marketing experts of large companies. They represent the techniques used by two of the largest players in the so called "geodemographic and lifestyles industries". In effect, these models are people spaces that seek to map the relevant details of large populations of people in an n-dimensional space for marketing purposes.
Experian (now a subsidiary of Great Universal Stores) with annual sales around £1 billion / $1.5 billion and employing more than 12,000 people, maintain records on millions of consumers. Through a variety of sources, they create socio-economic profiles of consumers, categorising them according to age, income, family status, credit rating and all manner of other classifications that turn up on the billions of records that are sourced for information.
This huge volume of data can then be subjected to exploratory analysis, or 'data-mining', using sophisticated programming tools - such as, automated cluster analysis, probability modelling, neural networks, genetic algorithms, etceteras.
Founded on simulation technology in 1962, CACI International Inc (www.caci.com) have developed a system called ACORN™ (A Classification of Residential Neighborhoods) to underpin many of their various services. It is a demographic information system, based upon a people space that categorises people according to where they live. The general idea is that residential localities can give indications of life style and spending patterns. This allows businesses to more efficiently target particular sections of the community. From this people space, CACI has evolved a diverse solutions portfolio for the Net economy. With approximately 5,000 employees and more than 90 offices in the US and Europe, CACI integrates the networks, systems, and software for telecommunications, e-Commerce and marketing information.
Clearly, such customer profiling and lifestyle databases are immensely costly to build and maintain. They require large revenue streams, so, the benefits can be made available only to very large organisations who can afford to pay the necessary fees. Customers benefit from this system only indirectly, through the questionable advantage of having their needs targeted more specifically. The disadvantage is that these needs are approximated to cater for the statistically significant averages and are biased heavily towards the products of the suppliers who can afford to use these customer targeting services.
Turning this system on its head would involve handing the construction, control and running of such a system over to the customers: letting them enter their own data and choosing for themselves the suppliers who would be most likely to satisfy their needs. This is what we are proposing to do.
Stigmergy
The idea that clients, acting independently, could work together to create anything so complex and highly organised as the giant databases created by the geodemographic and lifestyles industries might seem highly fanciful. Yet, similarly complex structures are a feature of many insect colonies. Ants, termites, bees are examples of insects that create elaborate nests with intricate architecture, create bridges, use complex food gathering strategies, coordinate their activities to find new nest sites and defend them against intruders or invaders.
There are no central databases in these systems of complex insect organisation; there is no central control, no leaders, no algorithms. Quite independently of each other, every insect plays some small part in a system of organisation without any training, learning or instructions.
It is only in recent times that scientists have understood how this seemingly mysterious process of unorganised organisation comes about. The trick, it seems, is to see the environment as part of the system. Individuals make changes to the environment and these changes affect the behaviour of the individuals. It is a feedback loop between the insects and their environment, not as previously thought, a process of communication between the insects themselves.
The name "stigmergy" was given to this process by Pierre-Paul Grasse in the 1950's, who used it to describe the indirect communication taking place among termites as they built their nests. He explained how the regulation and coordination of the building activities do not depend on the termites themselves, but, are the result of responses being triggered in the termite workers by the structures of the nest: as the nest evolves. Changes in the structure induce different worker activity, sometimes in the same termite.
Appreciating how an insect can alter the environment to affect the behaviour of other insects, a new light was thrown on the pheromone (chemical complex) trails laid by ants. Instead of seeing pheromones as messages that are passed from one ant to another, they are seen as contributing to a large chemical infrastructure, overlaying the landscape, that ants are genetically programmed to respond to.
When this paradigm shift was first pointed out by Grasse, it seemed of little consequence. Now however, in the age of information technology, it has been rediscovered and is being employed in a wide variety of applications. (Note: If a Web search for "stigmergy" is made with the GOOGLE.COM search engine, it will reveal 282 references, many of these illustrate novel Internet applications for this concept).
Decentralised control
It seems that stigmergy is a powerful way to coordinate activity over both time and space in a variety of different systems. It requires no planning, no leadership, no direct communication between participants. It is self organising. This sets it apart from the conventional organisational methods usually associated with human cooperative activity. Let's see how this maps across to a system of organisation for an e-business environment.
The main features of a stigmergic system are:
Individuals do not rely upon instructions.
Each individual gathers information itself and decides for itself what it should do.
Information is gathered from a shared environment.
There is no centralised decision making.
Individuals need only a few elementary rules
In an ant colony, the ants will lay down a trail of pheromones. They will have receptors to recognise the pheromones of other ants. There will be a decision function that activates appropriate ant behaviour when it encounters pheromone trails. The components used by the ants are genetic; the genes being supplied to each worker by the queen ant.
Translating this to the system of clone agents and people space as described in the previous chapter:
The queen ant would be represented by a business that would create a questionnaire based upon the needs of people who would benefit by being able to be put in contact with each other. The questions would equate with the genes that a queen ant passes on to the worker ants, which enable them to create pheromones. The pattern of pheromones laid down by the ants would equate with the digitised answers that a client would produce when completing the questionnaire. These records would be deposited in a people space in a way analogous to an ant laying a pheromone trail.
Ants can sense the pheromones laid down by other ants and this would equate with individual bots detecting the records deposited in the people space by other individuals. The ants will be equipped – through genes provided by the queen ant – a computing mechanism to interpret the pheromones of other ants. This would equate with the cafe software – provided by the business – that would allow an individual to identify suitably interesting contacts from the many records that would be deposited in the people space.
When translating this into human activity, the most difficult concept to grasp is that such a system is not organised or controlled from a central source. Every individual acts independently. The information provided does not come directly from the business but from the environment of records created by the individuals making use of the service. This is far outside of the concepts covered by conventional business strategies. It can only happen in the unique environment of the Internet.
You do not have to look hard to see this process of stigmergy already taking place on the Internet. News groups and discussion forums manifest in this same way, where people find places of interest where they discover messages that are useful to them. The messages prompt other people to respond, by adding their own messages. People will move from one group to another, choosing to participate only in those where the messages are of specific interest them.
There has been no centralised, organising body to create the tens of thousands of news groups and discussion forums that have spontaneously birthed on the Internet. Organisations do not provide content. People do not collude with each other as to what messages will be sent. The whole system has evolved organically and is driven by people acting independently.
It is this strange phenomenon of stigmergy that is now becoming of increasing interest to e-businesses. It can be employed to create self organising databases that cost little to build and maintain, yet, provide a service and efficiency that far outstrips anything that can be created through rational design and control.
Two prime examples of stigmergic systems, growing and evolving without central control or organisation, are The Open Source Movement and Napster. Both of these phenomena appeared almost spontaneously; growing to such power and influence, they out compete the industry giants.
Even with its vast organisation and huge financial resources, Microsoft is no match against the cumulative efforts of tens of thousands of independent programmers in the battle for supremacy in the server market. Building upon each other's efforts, the unorganised, independent programmers have created a free and highly flexible system known as "Linux" which has consistently held Microsoft at bay in providing the server of choice for ISPs and maintaining UNIX as the preferred operating system.
Similarly with Napster, a simple system devised by students, evolved into a people space that created a distributed database spread over hundreds of thousands of private computers. The combined efforts of the whole of the music industry had to resort to legal action to stop the free distribution of its copyright protected, musical assets. Even so, the system adapted and evolved into new forms that had no central control for the legal injunctions to aim at.
This demonstrates the enormous potential of stigmergic systems. However, these systems have one vital omission that puts them a step behind the stigmergic systems of the insects: the ability to automatically update, to rid themselves of duplication, redundancy and out of date information. This essential feature is provided in the insect world by evaporation.
Evaporation
There is one further feature of a stigmergic environment that needs to be emphasised. This is its impermanence. Stigmergic environments are designed to be short-lived, continuously self updating through a process of evaporation. For instance, when an ant lays a trail of pheromones across a landscape, the pheromones last for only a short time before they evaporate. This ensures that recent pheromone trails are not confused with older pheromone trails – which might relate to circumstances that have changed.
This is a critical element in the design of a stigmergic system. Any records deposited in a people space would have to be given an appropriate "life span" such that they "die" after the elapse of a certain period of time. This safe guards the environment against getting cluttered up with out of date information. This solves for the problems of obsolescence and redundancy, envisaged by the database expert, Vahé Kassardjian, at the beginning of this chapter.
Using a stigmergic environment for an e-business solution
Although this concept of stigmergy may appear to be wildly esoteric, it is an extremely powerful construct to use in many e-business situations. It can bring buyers in contact with sellers, it can bring information to people who want to know, it can bring help to the needy, it can bring specialist assistance to developers, it can put job seekers in touch with employers.
The first step would be to create a boundary for a people space and give it a formatting appropriate to the requirements of the business. The boundary would be the overall subject area. For example, if the business were concerned with the matching of people to jobs, the boundary condition would be to limit the space only to job seekers and employers looking for suitable job applicants.
The first level of categorisation might specify broad categories of industries. A second level in each of these primary categories might categorise according to specific job descriptions. A third level in each of these categories might divide into different areas of specialisation within those categories. Other levels may divide each of these sub divisions even further. This process of hierarchical categorisation, would format the environment sufficient to provide job seekers and employees with appropriate meeting points in the people space – in which to deposit their records.
To create a record, a job seeker or employer would simply click on a button to have a questionnaire automatically downloaded from the Web directly into their clone creation software where the questions will be presented on the computer screen. From the answers to the questions, a record would be created to represented them and their needs. This record would then be deposited in one or more of the categorised meeting points in the people space as chosen by a job seeker (or an employer).
Note:
The actual mechanics of this process would be completely invisible to the users. They would simply click on a button to get a list of questions. When they answer the questions, they click on another button to see a the route map to guide them to an appropriate areas of interest. When they reach a point of interest, they simply click another button. This click automatically inserts their record at this point and at the same time invisibly download all the other records that are there.
The records automatically downloaded from a selected point of interest will go directly into a client side application – something like the virtual cafe software described in the previous chapter. With this client side application, both employers and employees could use the search functions to further categorise their selections. In effect, this is extending the categorisation process of the people space out to everyone's own computer – in a way that allows everyone to make the final selections according to their own specific needs.
Note:
This is where clients are able to make their own categorisations: customising the people space to suit their own particular needs. As this is done on the client side, within the client's personal application, this has no effect of the ordered space on the server side. Clients simply choose or add their own criteria to be able to filter out a suitably short list of people to make actual contact with. With each record having a limited life span – being removed after a certain number of days – all records would be fresh.
These records can be designed to carry the URLs of a personal, or company, Web site, to provide more extensive details than can be covered by the records. Employee Web sites can hold CVs, examples of work, photographs, references, etceteras. Employer Web sites can show diagrams of the organisation, photographs of working conditions, terms of service, etceteras. The overall effect would be that employees would be able to see what kind of employers would have need of their services and have sufficient information to be able to draw up a short list for person to person contact. Similarly, employees would be able to draw up a short list of candidates for direct communication or interviews.
The great advantages a stigmergic system would have over any conventional employment agency is speed, cost and its ability to self organise. There are advantages that an employment agency would have, but, it is highly likely that an employment agency would use such a system themselves to source clients and to satisfy their needs.
A self regulating system
An interesting aspect of such a system is that it would automatically evolve to become progressively more efficient. In this example, job seekers would get the records of employers who were looking for someone like them and employers would get the records of the type of employees they were looking for. But, besides this, they would also see the records of their competitors in this categorised space.
This would create a competitive market situation in every category – where the superior will drive out the inferior or cause them to improve their offer or presentation. For example, a job seeker might see other job seekers using better Web sites to present themselves, offering better examples of work, etceteras. Similarly with employers, their presentation must be at least as good as the other employers to be able to attract the best available employees. By allowing each record a short life span, it provides the opportunity for job seekers and employers to improve their offers or presentations at subsequent visits to a selected meeting place - or, change to a different meeting place where there might be less severe competition.
This is the way market places become progressively more efficient and organised – a self regulating process that works far better than any human rules, controls, regulations or selection procedures.
The commercial advantage
Although there are a number of additional extras that could be added (such as making provisions for privacy and keeping out spammers), the great advantage of this stigmergic system is that it costs relatively little to set up and maintain. Not only this, the development costs would be minimal because exactly the same system of basic components could be used for a large variety of quite different situations where it would be useful for people to make contact with others. It is like a spread sheet, the starting position is a grid of empty spaces. Any system can be created simply by labelling some of the spaces and devising an appropriate questionnaire.
For instance, the job seeking example could easily be used by large companies to fill vacancies within their organisation. It could be used to make optimum use of sub contractors and consultants. It could be used to leverage the use of specialist employees, allowing them to be work in various different parts of the business as and when they are needed.
The system could also be used in specialty areas of interest. For example, gardening: where gardeners could be put in touch with others who share a particular interest in a specific type of horticulture. Seed merchants and specialist gardening services could set up such systems to gain good will to help them sell their products.
It could be used in all kinds of ways to bring people together for the purpose of collaboration or information sharing. It could be used in education, to put students in touch with suitable courses of study.
Particularly, it would be a useful organisational structure to create areas for micro money trading. So far, very little use has been made of the ability of the Internet to reduce transaction cost to practically zero. Given a low cost infra structure, such as can be provided by a stigmergic structure, "penny bazaars" can be opened – where a variety of software components, MP3 files, tutorials, specialist information, etceteras can be traded for pennies.
It wouldn't be viable for a single small trader to set up a Web site to trade in items selling for pennies , but, collectively, many coming together within a stigmergic organisational structure could attract millions of visitors - enough to provide many small businesses with substantial remunerative cash flows.
It may seem to go against the grain, charging money for small software applications, music clips, tutorials and information. But, the originators of these kind of products are the people we need to support. A few pennies appreciation for their efforts, given by many thousands of people may enable them to work on their creations full time. Let's support the small artisan. It is the only way we can prevent the Web being dominated by large companies.
### End of chapter 13 ###
Back to index
Note: This book lead to the creation of the stigmergicsystems.com website
Copyright 2001 - Peter Small
E-mail: peter@petersmall.net
All rights reserved by Pearson Education (Longman, Addison-Wesley,Prentice Hall, Financial Times for FT.COM imprint