Search engine strategy
A definitive source of information on methods currently being used for information indexing and retrieval is The Spire Project (set up by an Internet research resource and think tank).
It is worth spending some time looking around this site because there are some excellent descriptions of most of the major search engines in use today - together with explanations of their strategies to index and retrieve useful information.
Their limitations are discussed in two articles by the manager of this project, David Novak:
Keeping up to date with Internet Research Technique and Theory
The Growth of the Internet Should Still Scare You
In these articles, David Novak writes:
"I am concerned about our attitude to the growing quantity of information on the Internet.
"...organizing the internet is not a problem that will be solved. It only looks that way to people unaware that babies put on a lot of weight fast. The truth is that despite much effort, much of the internet will forever be un-catalogued and not indexed. The volume, trends and logic has made this a predestined impossibility. We will only ever arrange a fractional (and diminishing) percentage of the internet".
It is in these areas of limitations, where stigmergic systems can be applied: to get results which would not otherwise be achievable.
Note: This is part of a more extensive article on Organic portals in the 'System explanations' section of these references.