From the CD-ROM "Concepts and strategies"
by Peter Small
There are several important areas of information where search engines and basic stigmergic systems cannot cope very well. This is in areas where there is: ambiguity; high volatility; subjective content; high rates of technological change; classification of people; classification of services; etc. These areas are a search engine's blind spots.
In these areas, there is a need for multiple levels of categorization and a means of being able to compare conflicting or alternative information.
This calls for a completely different mind set. It requires a system outside of the confining metaphors of database technology and an enhanced form of stigmergy.
Search engine and stigmergic system blind spots
RingWeb and The Open Source Directory. are examples of basic stigmergic systems used to create open source directories. They are complementary to algorithmic searches and highly effective is many subject areas. However, there are certain areas of information where they are no more efficient than conventional algorithmic search engines. It is in these problem subject areas, where we shall need a more sophisticated form of stigmergic system than those used by RingWeb and The Open Source Directory.
Let's look at a few searches, to see what kind of deficiencies are exposed
This will present us with 16 rings covering 175 sites. This is far more efficient than Google, giving us only 175 sites to consider. The rings can be searched independently for key words to cut this list down even further.
Now let's try searches in a more complex area of information. Supposing you had a friend who'd been diagnosed with rectal cancer with metastasis (spreading to other parts of the body). Suppose you'd been asked to see if you can find a treatment trial she could take part in to increase her chances of recovery.
Using Google to search for 'Cancer' produces 12,700,000 references.
You can cut this down through the sensible use of key words. For example:
rectal cancer produces 127,000 references
rectal cancer treatment produces 87,100 references
rectal cancer treatment trials produces 21,000 references
rectal cancer with metastasis treatment trials produces 3,120 references
rectal cancer with metastasis treatment trials UK produces 625 references
Even here though, the number of references are still huge, there is no way to cross reference them or make comparisons. You cannot be sure you've allocated the correct key words and you cannot know what information is missing. There is also the problem of what other factors have to be taken into consideration and whether or not you will be able to understand some of the highly technical papers you will come across.
You'd expect the search results would be better by using The Open Directory and WebRings, because these results are selected and edited by humans. However, what you find is The Open Directory does not have a hierarchical route to a "Cancer" category.
The situation isn't much better in the 'Web Rings' Directory either. There you'll find only a three level hierarchy path (Health: Conditions and Diseases: Cancer), which leads you to rings that list 1,581 references, but cover only 18 of the 420 different types of cancer - none of them dealing specifically with rectal cancer.
However, when it comes to something like cancer where there are 420 different types of cancer, each of which has many variations. There are simply too many variables to be taken into consideration to be able to conduct a reasonably efficient search. This causes the system to break down completely.
In summary then, there are certain areas of information that cannot be handled efficiently by either a conventional search engine or a simple stigmergic system. This would be where;
1) there are too many variables to be classified in a hierarchical system.
2) The content is highly subjective technically complex
Figure 1 - There are some areas of information where search engines cannot produce efficient search results
Clearly, there is no evidence that there is a satisfactory way of searching though subject matter like cancer - certainly not by using conventional methods and techniques. Even stigmergic systems seem to be hopeless for dealing with the problem.
Directories and indexed lists would be helpful, but, when you start adding levels in a hierarchy (equivalent to adding a new key word) the size of the directory starts to increase exponentially. This imposes a practical limit to the extent of the categorization.
The answer lies in a completely different approach: using the techniques that have evolved in nature to solve similar problems of selection within a vast range of possibilities.
The solution is to turn the idea of a hierarchy on its head. Start with the classifications at the bottom of a hierarchy and then 'grow' the hierarchy upwards. This may sound crazy, but it is the ingenious and elegantly simple solution that nature has come up with.