The basis of a solution

Paraphrasing Sherlock Holmes: if the solution doesn't appear in the probable, then we must start to look for the solution in the improbable. In this case, if databases aren't able to deal with this kind of problem then we should be looking for a solution that doesn't involve databases. The principal tenet being that where uncertainty, change and volatility render logical and algorithmic search techniques useless, people are needed. Only the human brain can provide the kind of intelligence required to solve this kind of information supply problem.

Let's start by examining a conventional database – an outline view of which is shown diagrammatically in figure 3.

Information in a conventional database can be cleverly formatted and indexed to facilitate retrieval. Data can be sorted, mixed, matched, compared and selected using all manner of criteria and conditionals, but, the algorithms used have to be predetermined and require direct human reprogramming to change them.

Such databases are adequate as long as the data is reasonably long lasting and can be maintained within a constant format. But, if there are any chaotic or unpredictable elements of change, there will be problems because it will be extremely difficult to arrange for the database to question the validity of data; anticipate the possibility of error; check that it has become redundant or is incomplete.

Many important categories of information and knowledge – such as the expanding and evolving areas of technology – have such a high level of unpredictable volatility that reliable output is prohibitively expensive to maintain. Databases quickly become choked with redundant and conflicting information, the efficiency soon declines to a point where the information retrieved has very little practical value. This is the problem confronting anyone who would want to create the kind of database that Tillman Pearce is looking for.

In these situations, when accurate information and knowledge is not readily available, people resort to the age old custom of asking around: finding somebody who may have the information they need. This involves a bottom up strategy – following a trail, routed through person to person contact, with one piece of information leading to the discovery of another until the information or the solution to a problem is found.

In pre Internet days, the telephone was perfectly adequate for this kind of strategy. Databases could take the form of directories, yellow pages, etceteras. These provide contact names – where bottom up, seek to find strategies can be used to begin a search for people who might point to any kind of ill defined, fuzzy or elusive types of information that might be needed.

Such a strategy is illustrated in generic form in figure 4.

Figure 4 shows how it is necessary to have some method of formatting the space where people can be found; together with a procedure for finding the most suitable people to provide any required information. Viewed in this diagrammatic form, it becomes obvious that in the massive communication environment of the Internet and with the rapidly changing and evolving nature of knowledge and information, the same sort of problems will occur as with any conventional database.

The number of people needing to be included in the people space would be impossibly large. There would be too many categories and the categories chosen would have to be constantly changed and added to. There would be too many ways in which the people space can be formatted. Even for the people that could be listed and categorised, there would be no certainty that they could provide currently accurate or up to date information. There would be no precise way of determining who knew what and the extent and exact nature of their knowledge.

With these thoughts in mind, let's turn to the opportunity offered by the problem of creating a treatment database for cancer patients as outlined by Tillman Pearce.