In recent years, in a well-argued book (Hjørland, 1997) and subsequent papers (e.g. Hjørland, 2004), Birger Hjørland has criticised various aspects of information science methodology, and linked their defects to faulty philosophical beliefs. I often find myself agreeing with his criticisms of the methodology, and with his opinion of the philosophies, but am never convinced of the linkage between the two. I do not think he would claim that someone employing a particular methodology is consciously working within a particular philosophical tradition, of which indeed he/she may be ignorant, but I am often doubtful whether even the link made by analysis is justified.
My doubts arise for the following reasons. First, if one looks at the methodologies of the sciences in general, I do not think that each can be related closely to a particular philosophical approach. I believe that scientific method is far more wide-ranging and flexible. A particular philosophical approach - say, empiricism or rationalism - arises from abstracting certain elements of scientific and other forms of practice, refining them and exploring their consequences. Each such approach is therefore a limited, one-sided account of one aspect of the much richer reality of practical method. Second, if one looks at methodologies in information science, I believe that their defects arise, not from a faulty philosophical basis, but from a too limited application of scientific methodology. In what follows I will discuss the first issue only briefly (since this does not aspire to be a philosophical paper), and then consider examples illustrating the second.
Some philosophical theories
Everyone except a solipsist (if such there be) will agree that there is a world external to himself; that activities and events in that world result in physical impacts on our sense organs; which give rise to sensations, images, percepts; that in the mind there are named concepts (e.g. table, collision) that we attach to such images; and also more abstract concepts (e.g. support, thing, interaction, event) by which we group immediate concepts; and various kinds of linkages between concepts (propositions, relations, laws, deductions, theories). Different philosophical approaches offer different views as to the status, origin and relationships of these various elements.
According to Hjørland, in empiricism “the scientific process is viewed as the collecting of verified observations and as generalising from such a collection by induction; […] sensations and experiences are regarded as ‘given’, what we see is independent of our theories, conceptualisations, culture and political interests” (Hjørland, 2004, p.134). In the diagram above, empiricism thus sees step B as arising simply as a result of step A, independent of whatever may already exist in our mind, and further sees step C as induction, an activity of the mind that forms an immediate concept from assembling a set of similar percepts.
Again according to Hjørland, rationalism “does acknowledge the role of observations […] as chemical-physical stimulations of sense-organs”, but “looks at our concepts as inborn structures, which match and classify our perceptions […] Rationalism tends to use a ‘top-down’ analysis in the processing of information, i.e. to approach a given set of data from some pre-established categories” (ibid., p.135). In the diagram, therefore, rationalism sees step C as one with the arrow reversed: concepts are “inborn” in the mind, which uses them to match and classify percepts. Instead of induction from data, “pre-established categories” are used to group perceptual data, the categories deriving from some preconception of what classes might or could or should exist in the external world. (I am not going to explore here just what “inborn” might mean, e.g. created from a source of knowledge other than the external world?)
Yet again according to Hjørland, for logical positivism, “knowledge can be reduced to private immediate experiences. Or, rather, it may be reduced to [or constructed from] verbal reports about immediate experience” (ibid., p.137). Theories must be translatable into such verbal reports. Now it is evident that a verbal report about percepts must be expressed in the form of concepts. In the diagram, therefore, the derivation of concepts in step C is presumably the activity of induction. Steps D and E are allowed to be meaningful only if such abstract concepts and the theories into which they are linked can be reduced to inductively established immediate concepts.
Discussing these three viewpoints as a whole, Hjørland states that they share the assumption “that concepts are formed in the individual mind and that perceptual processes are mechanical processes that process chemical and physical stimuli. Critics have pointed out that our perceptual processes are influenced by our language and culture, […] that our most primitive sensing is already ‘theory-laden’” (ibid., p.144). In terms of the diagram, this criticism implies that the forward arrows in steps B and C should be replaced by arrows pointing both ways - that the formation of both percepts and concepts is an interactive process between the external and internal worlds, both contributing to what we perceive and what we conceive.
Relation to scientific methodology
The viewpoints discussed above might, between them, be said to include a number of assertions, for example:
Though I do not propose to demonstrate it by example and quotation, I believe that all these assertions are compatible with scientific methodology. This methodology includes observation of the external world, experimental control of the external situation in order to focus observation, collection of selected data about the external situation, analysis and manipulation of that data, interpretation of the results of data analysis in the light of already existing concepts, laws and theories, creation of new concepts, generalisations and theories that may be used to explain the observed data (pulling hypotheses out of any area of our total experience), application of these new conceptual structures to existing and newly collected data, and so on. At some stages in research, an investigator is behaving as an empiricist, at other stages as a rationalist, sometimes concerned solely with perceptual data, sometimes solely with abstract concepts, at times he is concentrating on an individual experience, at other times he is exploring the widest of generalities. When unexplained data drive the development of a new theory, the investigator is using a bottom-up approach. When he employs the theory to organise a new structure for a field of knowledge, he is using a top-down approach. The essence of scientific method is that all these aspects are interrelated and integrated. It is philosophers who, by isolating particular aspects and giving them pre-eminence, disrupt the process of research into incompatible elements.
The status of information science
Information science is basically concerned with facilitating the transfer of knowledge from one human mind to another through the medium of documents recording knowledge and associated retrieval mechanisms.
The knowledge transferred is conceptual, recordable in some “language” (natural language, mathematics, chemical and other symbolism, diagrams, etc). Information science therefore has nothing to do with steps A, B and C of our earlier diagram. Its raw material is conceptual knowledge - most often, indeed, verbalised knowledge. How concepts are derived from percepts is not relevant to its activities. Strictly speaking, therefore, what philosophical views about perception and immediate concept formation an information scientist might hold should not affect his professional activities. On the other hand, he may well be interested in steps D and E of the earlier diagram, i.e. in the relationships in the mind between concepts, since part of his work is to make use of conceptual relations in retrieval systems.
So let us now look at some examples of information science methodology that have been criticised by Hjørland.
Knowledge structures
My first example of a methodology in information science arises from Hjørland’s comment: “The facet-analytical school of classification founded by Ranganathan […] is seen as a rather strong example of a rationalist philosophy. It is a position that does not consider the empirical basis (or testing) of systems very much. It is strong, however, in providing clear definitions and rules. Systems such as thesauri or classifications based on this approach often display a high degree of structure and clarity, which is lacking in systems developed by other traditions” (ibid., p.144). I will not be concerned with faceted classification as such, but with the wider issue of structured retrieval languages.
All subject retrieval is based on matching a tag associated with an information request with tags associated with documents, the tags purporting to represent the subject of the request or document. A “subject” is a concept, usually expressed as a linguistic word or phrase, which itself may be used as the “tag”. A tag is either extracted from the request or document, or assigned by an information professional (IP) such as an indexer.
A retrieval system such as an Internet search engine simply extracts every word in a document or request and uses each of them as a tag. But many retrieval systems are based on the assumption that something more is needed than bare extraction or assignation of a tag. It is claimed that a searcher often finds that the tag in his initial information request does not lead to the delivery of potentially relevant documents, and that he or she would welcome system guidance from his initial tag to others that are held to be usefully related.
That such guidance is indeed useful has been abundantly demonstrated by the results of the TREC retrieval experiments. Sparck Jones reported that “query expansion […] whether done in a conventional way before searching, via a thesaurus, or after searching, using [relevance feedback] from retrieved documents” did improve performance, in fact it was “the most critical factor in retrieval” (Sparck Jones, 2000, p.33). But, as to what strategy of query expansion worked best, or best in particular circumstances, she reported that “there are no solid conclusions to be drawn“ (p.34).
So, given this situation, the issue facing the IP creating a structured retrieval aid is: what conceptual relations between tags should be selected and presented to searchers for use in query expansion? In principle, there is perhaps an indefinitely large number of other concepts that can be regarded as related to any individual concept. Should the IP attempt to identify all of them (or as many as can be discovered) and present them all (say, in browsing mode) to every searcher? This is, in effect, what a universal ontology such as CYC is attempting (Vickery, 2004, p.388). Or should the IP assign a particular conceptual tag to one or more subject domains, and present to the searcher only those relations that are found within a particular domain? This policy is followed by CYC in its use of the domains it calls “microtheories”; it is also used in retrieval systems that limit their scope to a particular domain, or use “scope notes” to limit the application of individual tags.
Another way of limiting the number of relations presented to the searcher is to present only relations of a certain type. Traditional classification first allocates concepts to domains (“main classes”), and then presents only hierarchical class relations between concepts in those domains, plus an assortment of ill-defined “cross references“ between classes. Faceted classification narrows the domains to facet categories within main classes. Most thesauri are limited to a domain, and present hierarchical, whole/part and ill-defined “related terms”. Some experimental thesauri are seeking to expand the last group into a set of more definite relations (Soergel, 2004).
Any partition of the universe of concepts into domains, or use of a restricted set of conceptual relations, necessarily involves a “top-down” approach: the domains and relations used are decided in advance of actual tagging, even if preceded by extensive study of existing documents (“literary warrant”). In the documentary record, from which concepts are ultimately derived, new juxtapositions and interrelations among the things conceptualised are continually occurring, so that any preconceived conceptual structure becomes increasingly ill-fitted to cope with the new relations being engendered. The problem of keeping retrieval languages up-to-date is all too well-known. For this reason, they are inherently likely to become defective.
I agree with Hjørland that the systems now used, such as classifications and thesauri, that severely restrict the types of conceptual relation displayed, may well be particularly defective. Are the relations they use in fact the most useful to the searcher? The evidence from the TREC retrieval tests was inconclusive. As far as I know, no analysis has been undertaken within TREC of the causes of individual “retrieval successes”, comparable to the analysis of “search failures” in earlier tests such as MEDLARS (Vickery, 2004, p.278), so TREC provides no clues as to what particular types of conceptual relation (if any) are more helpful in search.
Even if we can, by research, identify the more useful conceptual relations to present to the searcher, and offer a wider range of them, can we avoid a “top-down” approach? Indeed, should we try to? Earlier I have noted that scientific method does not. Is a “bottom-up” approach (presenting every possible relation to be found) realistic and useful in practice? These are not rhetorical questions. They could, hopefully, be answered by further research. But in any event, I do not see them as philosophical questions. They pose exactly the same problems as indexing a book: should the index pick out the most “important” words and topics (according to some preconceived idea of what is important), or should it be a concordance to every word in the book? This is a practical issue.
Relevance research
Hjørland states that “thousands of studies on relevance in information science have failed to advance our knowledge of the underlying mechanisms in the production of non-relevant items in information systems”. In the preceding section I have also written of the production of relevant items, equally in need of further explanation. Hjørland sees this failure as an example of positivism in action - “positivism neglects the stratified system of causes” (Hjørland, 2004, p.147). It may be true that some people who have positivist views neglect causes, though their basic philosophical standpoint, as described by Hjørland, does not seem to necessitate this, and other accounts of positivism do not assert this. In any event, this does not imply that all who neglect causes are influenced by positivist philosophy.
It is a commonplace of research methodology that there are three types of question that investigations can address: descriptive (this is what happened), relational (when the relations - usually quantitative - between two or more aspects of the situation described are studied), and causal (when attempts are made to determine whether and how some aspects of the situation determined other aspects). It is also a commonplace that causal studies are the most demanding of the three.
Let us look at the procedure used in the TREC retrieval tests (Vickery, 2004, p.310). Sets of test documents are assembled; for each set, a list of appropriate information requests is assembled; associated with each request are judgments as to the relevance of each document in the set to the request; each participating investigator in the test creates his own retrieval system, with its particular characteristics A, B, C, etc; he uses the system to retrieve documents from a document set for each of its associated requests; the set of retrieved documents is sent for independent evaluation; performance (based on precision and recall) is measured. The performance of a number of different retrieval systems used with a particular document set can then be related to their characteristics A, B, C.
As can be seen, the TREC tests are relational investigations. To go beyond them to causal analysis would be a very strenuous undertaking, though I agree that this is what is now needed. I would like to see some investigations along the following lines. Taking the results of particular test runs, to examine each document retrieved, and the search steps by which it was retrieved, and to ask: if relevant, why and how was this document retrieved (success analysis)? if it was judged not relevant, why and how was it retrieved (failure analysis)? For relevant documents not retrieved, to ask why not (more failure analysis)? Tests such as Medlars already gave some clues concerning failures (ibid., p.278), so the important new aspect here would be the success analysis. Yes, a heavy task. But it would be the beginning of causal analysis, that could give us clues to the design of retrieval systems. Once again, I do not see this as a philosophical issue. It is a methodological issue - to go beyond relational analysis to the more demanding but ultimately more rewarding causal analysis.
Individual and social
Hjørland criticises the standpoint of “methodological individualism”. In information science, he states, “the point of departure for this view was the idea that psychological studies of human beings might provide the basis for the design of information systems”. He advocates “more sociologically-oriented approaches” (Hjørland, 2004, p.148). Later he states that “relevance research is typically positivist in its tendency to psychologise criteria for what is relevant, and thus seeking the secrets of relevance in the individual rather than in scientific norms” (ibid., p.149). (I regret that I do not understand what the reference to “scientific norms” here means.)
Hjørland continues by listing his own set of seven possible causes for the retrieval of non-relevant documents. One of these relates to indexing procedure (false drop), one to semantics (multi-meaning), the other five to individual judgments made by the searcher. Judgments based on the searcher’s state of knowledge might or might not be called “psychological”, but they are indubitably individual. Conceivably, the study of individuals, and the “reasons why” they made certain judgments, could throw light on “the secrets of relevance”. This is a policy that Hjørland elsewhere recommends for the study of indexing (ibid., p.146).
What would be a more sociological approach to relevance research? Certainly, there must be awareness that a searcher’s reaction to, interpretation and relevance judgment of a document might be due to his general cultural and social experience rather than to the document’s specific knowledge content. Similarly, in the study of indexing, Hjørland points to a need to take into account an indexer’s “interpretations, subject knowledge or world-views” (ibid., p.146).
But beyond that, surely sociological research seeks to establish generalisations that apply to groups of individuals in types of situation, and to identify social causes (i.e. causes that are common to all members of a group). Hjørland seems to criticise the tendency “to provide abstract or generalised models of users”, urging the need to take “domain-specific” views in information science, so he appears to be arguing for a sociological or group approach at an appropriate level of group, a level at which we might hope that user behaviour is sufficiently uniform to be generalisable. Whether being concerned with the same “subject domain” is the appropriate level is of course a matter for empirical investigation. Studies of information users and uses have in fact suggested many other variables that might affect user behaviour and so potentially define an appropriate level for generalisation (Vickery,1973).
My view is that both individual and group studies of information behaviour may throw light on the design of information systems for particular groups of user, and that to denigrate one or the other approach on philosophic grounds is not helpful.
To conclude. I believe that Hjørland’s insights into and criticisms of information science practice are often well founded. But I do not consider that seeking to associate defective practice with particular limited philosophical views adds to either the clearness or the cogency of his criticism.
References