Link resolvers

Link resolvers are here to stay. Resolvers are invaluable in many institutions which otherwise bathe their library visitors in a sea of divergent, inconsistent, and occasionally incomprehensible interfaces to a lot of information that once was browsed, simply, by walking along rows of library stacks. Link resolvers are critical to delivering what libraries are supposed to deliver: access to the wealth of knowledge relevant to their user communities, through manageable and dependable means, across otherwise disparate systems.

If you've never used one, a link resolver enables users of online library resources to navigate from, say, a reference citation at the end of one journal article to a full-text copy of the cited item, or maybe from an abstract in a database to a catalog search for materials about the same topic, or from a cited reference search report to interlibrary loan requests for several of the citing articles. What does this mean, in practice? If you use a resolver-enabled library from the web, then when you search databases for articles, and arrive at lengthy lists of article 'hits' that relate to your topic, you will also see a little web link button to the resolver from each of the items on that results list. The button takes you to a resolver screen which offers choices of what you can do relative to the reference you selected. Often, this will include another link to take you straight to the text of that referenced article (even if it's to be found in a product and interface yet again very different from the one you're already in). If the library can't get you to the full article, it can probably at least get you closer, by telling you where the paper copy is on the shelf (if it's not online), or by asking another library to loan a copy (if it's not at all in your library), or by connecting to ancillary services which can, for example, look up how many other articles also cite the one you clicked on (to see if you can cross it off your list!). All these functions are listed as links on the resolver screen that first button led you to.

A better explanation, perhaps, is in the details of what happens when there's no resolver. When there's no resolver, you print out that list of search results, then go to the library web site and poke around an A-Z list of online journals, finding perhaps 20% of what you want from that list online. Each of those individual articles appears, however, on a different publisher or product website; it takes eleven clicks and two searches to find one, then the next one you go through a similar pattern of browsing lists of issues and authors only to find the one article you want actually is too old -- or, often enough, too new -- to be in the actually online product you're looking at, rather it's in a different one. After a few times through this maze, "well," some folks think, "I guess that's enough, I can stop there." Or, if they're old-school, maybe they'll click over to the library catalog and type in the "Abbv J Name" for the articles they want, but the catalog doesn't know what to do with the abbreviation, and returns zero hits. The ambitious souls persist, and actually walk over to the library and along its shelves, because, "it'll be on paper, right?" But, a-hem, the sign says, we cancelled our print subscription, sorry.

Better to have a resolver, right?

Link resolvers are all about library collection management. To make them work, librarians manage lists of the online journal and database collections to which they have secured access, and set certain bulk preferences for, say, whether one scientific abstract database from one vendor should lead to a fulltext journal collection from a separate publisher. The librarians establish these connections, and update them whenever products, licenses, or preferences change, on behalf of all library users. In this way, link resolvers are "collection-centric," in that they can dynamically connect library visitors from any one library resource to, potentially, any other.

Another benefit of link resolvers is the common service thread they tie through otherwise very disparate resources. The many databases, journal packages, and reference sources they move us through come from different vendors and publishers, each with its own website look and feel, few of which mirror a library's own site style. In The Before Time, library goers thought nothing of moving from paper journals to card catalogs, and from big reference books to book stacks. Although moving through sources through resolvers can still be a jarring web user experience, the common denominator of the link resolver screen is a big advance over leaving users to fend for themselves in a sea of inconsistent interfaces.

The best thing about link resolvers in 2004 is that a formal standards committee has produced a reasonable specification, called "The OpenURL Framework for Context-Sensitive Services," for the information that passes through resolvers. Which is to say that there's some hope that more vendors and publishers and libraries might be able to plug a higher percentage of our diverse online resources into link resolver servers. If you know anything about standards, you'll know that saying "there's some hope" in that last sentence is about the best we can wish for. But the community of OpenURL users has proven over its short lifetime that they will often play well together, at least eventually.

Bibliographic reference managers

A few weeks ago I started out to do some new research in preparation for a paper I'd like to publish. Nothing fancy, just the normal cycle: search in a few databases, look up whatever results I can online (using our handy local link resolver, which uses SFX), divert off from a few references there, and export the choice references into EndNote along the way. When my to-read list of online papers shortens, I print a sorted-by-journal-title list of paper-only articles to go hunt down in our library. I find roughly half of these, take some notes on paper, photocopy a few others, put them in their own folder, and so on.

I've found early papers about the area of the work I'm doing I hadn't seen before, and additional authors to look for in future searches, and have a clearer understanding of what ground I need to cover to get editors and readers to recognize my work as a worthwhile contribution. Tidy up the work, address the right issues in the paper, and there you go, right? Well, actually, no... I'm left with a mess of printouts, scribbled notes, pdf files with random names, and an EndNote library that doesn't quite tie it all together. While there might be many traditional approaches to imposing order over this process, it seems it would all be easier if there were just an extra link on the link resolver screen that let me "log this reference". After years of automating isolated steps in the doing-your-homework part of the scholarly communications cycle, I want my library to help automate my entire record-keeping processes.

That's not to say that progress hasn't been made. EndNote and its peers (including the stalwart (La)TeX+BibTeX duo, and clients like Pybliographer that connect the two with higher-level tools like LyX and OpenOffice) are critical in simplifying manuscript preparation. That they search databases directly has long been a great help, too. New products like RefWorks, WriteNote, Quosa , and MetaLib appear to integrate two or more steps in this process, shortening the workflow in one way or another, and appear to offer real benefits.

Even so, I heard Lee Iverson of UBC say something really interesting at his talk at Access 2003 on "Digital Library Research Agendas." He spoke of how libraries need to be "publishers of groupware tools" that manage the workflow of this entire process. And that we would do well to stop thinking solely in terms of shortening one step or combining two steps, and to reconsider the entire lifecycle of the bibliographic research process. And that library-goers should become as much users of our groupware services as they already are of the resources we procure. If I haven't mangled his intent entirely, the Scenarios section below will illustrate one interpretation of Iverson's vision, wherein tools like reference management software move from being ancillary training opportunities to becoming core library services.

Weblogs

If you're not already familiar with weblogs, browse through the Libdex list of library weblogs to get a flavor for what they're about. Scrolling short entries describing events from a blogger's day, musings on any topic, real journalism, opinions all over the map, and plenty of links to stuff bloggers find interesting elsewhere on the net are the meat of what goes into weblogs. Interesting things have been happening in the weblog world, though, beyond the ordinary tedium (and, often enough, extraordinary salience) of a million monkeys typing away.

The first trend to note is the growth of "trackback". As described in "A Beginner's Guide to TrackBack", authored by the progenitors of trackback, Mena and Ben Trott (authors, also, of the popular Moveable Type weblog manager):

'In a nutshell, TrackBack was designed to provide a method of notification between websites: it is a method of person A saying to person B, "This is something you may be interested in." To do that, person A sends a TrackBack ping to person B.'

Sounds mundane enough. But there's more going on than meets the eye:

'...the TrackBack ping has created an explicit reference between my site and yours. These references can be utilized to build a diagram of the distributed conversation. Say, for example, that another weblogger posted her thoughts on what I wrote, and sent me a TrackBack ping. The conversation could then be traced from your original post, to my post, then to her post. This threaded conversation can be automatically mapped out using the TrackBack metadata.'

That probably sounds familiar to anyone who has ever followed a cited reference at the end of a scholarly article: TrackBack creates references. Innovation on this front is happening within scholarly journals also, such as BMJ online, and its peer journals published by Highwire Press, which provide visual maps of citation patterns as in this BMJ citation map. At an even simpler level, TrackBack looks much like the same kinds of citation practices appearing in scholarly and other publishing contexts for generations.

What can we glean from this parallel? Like scholars, webloggers feel a natural urge to connect what they have to say directly to the words of others. This is not surprising, and not only because some webloggers are likely also scholars. Despite broad awareness of the old "open your mouth and remove all doubt" adage, folks -- fools, included -- will find ways to thread their conversations together. That the blogosphere has defined techniques for accomplishing this further reminds us that people want to have their say, and that they are willing to do it publicly. And that they will create ways to bolster connections forward (by leaving TrackBack ping URLs) for others as readily as backward (by citing preceeding sources).

The corollary to wanting to have your say is the need to know what others are saying. And again we have ample evidence that the non-scholarly world is finding new ways to track fashions in current awareness. Many in the library world have caught on to the usefulness of RSS, and several have written thoughtfully about it, notably Steven Cohen, among others. This means of syndicating headline content from one website into another is even the central focus of a site devoted to library-related site content syndication, LISFeeds. In 2003 RSS newsfeeds landed on more and more library public websites, too, and in ILS software like Koha, which allows RSS-based syndication of new book lists and the like.

Users can also harvest RSS feeds to create their own "daily newspaper" of the weblog world subset they care about with new applications called "aggregators". Aggregators (examples of which include Bloglines, Feedreader, and Rawdog) let anyone multiplex several weblogs into their own clipping service-like weblog, and even mark which items they've read, and add links into their own weblogs.

RSS feeds get even more interesting when post-processing indexing and analysis is performed on aggregated streams. The best example is probably Blogdex, which indexes outward link patterns from thousands of weblogs to determine which web pages -- anywhere on the web -- make up the daily zeitgeist. If you want to know what people the world over are reading These Days, Blogdex is the fastest way.

One step beyond Blogdex is del.icio.us. Delicious lets anyone sign up and quickly add links they find interesting, and arbitrary tags describing those links, onto a rolling communal linklog (a weblog of just links to other places). On one hand, this is faster than Blogdex, which by definition requires a delay of pattern watching before it declares a site or page blogdexworthy. Delicious lets you watch over people's shoulders Right Now: users have bookmarklets which automatically add a link at del.icio.us to any page they are currently reading, even without any extra work like filling out a web form. So the links you see at the top of del.icio.us right now were probably added to it at about the time you were reading, here, about aggregators. On the other hand, Blogdex is all about post-processing, though it starts at a disadvantage to Delicious in that it scrapes urls from other sites. Delicious perhaps allows easier up-to-the-minute dynamic analysis, as evidenced by the recently-added visualization bit of shading popular urls, along with running totals counting how many times each particular link has been added by different users.

And there are variations on the linklogging theme. Witness Furl, a shared linklog with a slightly more user-friendly front-end promising also to help archive links. And then visit biologging, which directly connects weblogging to the Pubmed database by allowing users of HubMed, a custom Pubmed interface, to blog any reference that interests them to a shared space.

What do these developments add up to? An opportunity, for one. This world outside of libraries has created new protocols and a public service model that might be useful inside of libraries, so it is to our advantage to consider ways to reuse their work directly. That many smart hackers are approaching similar problems with similar solutions is often a good sign that they've hit upon a good match.

We might also read into what we see happening outside of our libraries that increasingly people don't necessarily mind sharing information about what they are reading, and what they have to say about it, even in the least formal context. To the contrary, they will go out of their way to define and use methods for sharing this information seamlessly, and instantly. And, in so doing, they will ensure that the practice of citing references is not merely a scholary exercise. As of 2004, it's just another information commodity market.

In the near future, we'll likely see smaller domain-specific blogdexes and deliciouses. And, with models and software developing for how to do these things well, there's an opportunity to build layers on top of these. Add to the mix that a few major institutions are bringing up public weblog services for their community members (major as in Harvard Law School and MIT). We might do well do consider a world in which these largely informal, non-scholarly communications blur seamlessly with scholarly research activities.

The undergraduate

Our example undergraduate student is a busy, busy soul. Aside from the classload of a full-time student, he organizes for a student political group, and reviews film for the campus paper. Here's a whirlwind tour of his information needs: For his classes, he writes papers for which he makes heavy use of online aggregate fulltext sources like Academic Universe, and he has to prepare an honors thesis of at least fifty pages which will be somewhat heavy with references. For another class in media studies, he has a group project where he and his project mates survey economic news reporting in the European Union by tracking websites of major news agencies in several European nations for two months. His campus political group has a website with a weblog for both local information and a tie-in to the national campaign website of their favorite candidate. Like I said, a busy soul.

Let's consider some things he might want to do with our not-yet-built infrastructure:

• Add references and fulltext from online sources like Academic Universe to his web-based reference library.
• Add EU news links, from one of several sites he's assigned to track, to a weblog shared by members of his group project, commenting on some of them, but not all, and grouping most entries by a simple categorization system they've devised (e.g. "finance," "constitution," "law," etc.). This group log is also available to other students in their course.
• Add news stories to his student political group's weblog, and comment on links his group peers have sent.
• Add links to some of the same references and links he adds to these other sites on his own weblog.

The first task isn't too innovative as of 2004; we already have both free/open source (Citation Manager) and proprietary (several) software products to accomplish that. The second and third sound more like what many weblog backends and Delicious already do, but with an extra measure of integration with his courses, and some group-defined access controls, say, for the political group and course project weblog. If you blur the lines between his reference library and the weblogging activities, and imagine he is using library groupware to do all these things, it gets more interesting.

The assistant professor

Our assistant professor is perhaps more focused than our undergraduate friend, but is nonetheless very busy also. Her obligations include teaching four courses a year, working as an editor for a new journal and a reviewer for another, and moving forward on her own research, in polymer science. She also manages a handful of research and staff assistants in a lab she shares with other faculty members.

She spends a lot of her online time doing the following:

• Managing the online syllabus for her classes, adding very current articles from her specialization, a fairly new branch of science. Sometimes she adds references to both her reference library and her course reading list at the same time.
• Checking references on papers she's reviewing, sending these through a page on the library website which turns out links to the referenced papers themselves, which she can then scan for relevance and potential value to her own work.
• Tracking links to popular news stories, grant information, and patents related to her field, all posted to a group weblog shared among members of her lab and peer labs.

It isn't hard to imagine that activities like the last, which used to happen on mailing lists, will move more to the web as email becomes generally less efficient for various reasons -- notably spam, among others. The weblog might be eagerly maintained by ambitious graduate students at the various labs. The first action happens often already, albeit only when professors and their assistants log in to dedicated courseware services, and use those tools' interfaces to augment a reading list. It would save our professor much time to just be able to use a bookmarklet in her web browser to automatically add such links, and for the course server to read in links she sends and allow her to quickly place the linked article beside a certain week's other readings. As for the second example, current link resolvers can already support some of that functionality, but how many libraries have extended resolvers to allow this kind of batch action?

The librarian

Our librarian friend is at no loss for work, either. He does reference consulting in the graduate library of his state-university employer, he manages several web-based subject guides, and he is on a campus-wide e-resource collection development committee. A few tools we could assemble would help several areas of his work:

• When working with a particular researcher to refine her complex searches, the researcher gives the librarian web access to her otherwise private reference libraries. This way, as the consultation develops over email, and the librarian tries various changes to the search strategy, he can compare potential search results to the articles the researcher has already collected. The researcher doesn't have to share what she's working on with anyone else.
• One-click bookmarklets for adding new sites to the various subject guides he maintains, and which also tell the library proxy server to collect hit statistics for those new sites.
• A shared weblog for queueing and discussing new resources his committee is considering.
• A secure way to troll through through the aggregated additions to user reference libraries that lets him see what resources and articles are popular during a given time period, but without sacrificing the privacy of users choosing not to share their information.

These activities, all typical of librarians' online work, occur in a range of public-to-private and individual-to-aggregate contexts. The first, helping a single user, needs a private access control toggle. The second is a private hook for adding to a public page. The third is a group space, and the last implies a private, scrubbed aggregation context, which would also have familiar useful public functions, such as a "people who are reading this are also reading this," with a terrific amount of granularity of algorithmic refinement (imagine "people reading this subject guide have also read these articles," or, even better, a librarian's holy grail twist on these algorithms, as perfected by Amazon.com: "people reading this article also read this subject guide").

From general link resolvers to personalized link routers

One view of how to build these services requires a simple reconception of our general-purpose link resolution systems. In most deployments, OpenURL linking takes a single anonymous user at a single library from one information source to one of many services of use for that source: from a reference to a fulltext article, or from an article to a cited reference list, or from a metadata record to a loan form, and so on. Hence the term "resolver": the system reports which services (as pre-defined by librarians) are available relative to a given information artifact, and resolves the user's service choice by forwarding the user to the right site. The entire transaction is stateless, in that which service one user chooses for any arbitrary source has no bearing on his next choice for a different source, or the next user's available choices for the same source, and so on. There's one source, one set of suggested services for that source, and (usually) one service request resolved per each source, and that's it.

That's not to say that there aren't interesting artifacts from these transactions, such as usage logs, and analysis thereof, or anecdotal user feedback. But there is no typically no remembered state, in that there is no attempt within this transaction for the system to recognize the user involved (aside from simple authentication/authorization in, say, an off-campus proxy context), to determine any preferences that user might have for potential service targets. A user cannot specify his own source and target categories; he is left to enter into the equation only from, and exit only to, sources and targets determined by the library staff, based on broad knowledge of their collection. There is no opportunity for insertion of per-user rules that will trigger secondary services for a given source link, such as link logging to a private or group weblog, or an automated subject-specific indexing based on the referring source, as we considered in the scenarios above. Users cannot configure resolvers to automate services so they happen in the background at the same time as they select a resolution service ("log all my links automatically but this time I want the full text"). And users cannot stipulate that, perhaps, they want resolution to happen within a different library's resolver ("I'm just visiting one university library for the week, could you please forward these requests to my home institution").

Phew, that's a mouthful. But those possibilities can be boiled down into two terms: "link routing," and "personalization." "Link routing" is about arbitrary rewiring of the current single transaction (source -> service list -> target) paradigm. The term "routing" implies that new hooks at and between each phase of the current resolver pipeline allow multiple, arbitrary, parallel or sequential actions to be added in. "Personalization" adds functions that will vary depending on the user, either as predetermined by the librarians or as specified by the user.

To explore these further, here are examples of how some of the actions listed in the scenarios above might get wired in to the various steps:

• adding a bibliographic reference to a weblog

  This is simply a new service to be added to the list of services. The router would need the user to log in and specify which of several weblogs (if necessary) to send the bibliographic metadata to. The weblog target would need to be able to parse, store, and later render, route again, and resolve the bibliographic reference (the context object, in the new OpenURL parlance). If the weblog backend is also a public service of the library, the link to the weblog backend would be added at the service list phase, and librarians configuring the router could set it to offer the link only for users registered as having weblogs. If the weblog engine is private, or otherwise off-site, the user would need to add it to their preferred target list.

• verifying reference lists

  For the professor to be able to bulk-verify a reference list, the system would need to account for multiple concurrent "sources"; perhaps in a queued-batch model, where the professor uploads the list, and the queue tracks the professor's progress as she steps through each article. This also might imply setting a "verify-reference" bit in the service list generation phase, tweaking the list of suggested services as appropriate for the task.

• checking search results against a user's reference library

  Private access to a user's private reference library implies an access-control-driven authorization step for our librarian (or anyone else), who might first log in to the router and specify that he is now in "search-compare mode for user janeqpublic". Then whenever the router returns a service list from the librarian's searches, an indicator appears showing whether "this article does/does not already appear in janeqpublic's library."

• visiting researcher router bounce

  For the visiting researcher without a local account, the router could offer temporary guest access, within which the researcher could specify his home institution. From then on, router lookups could be automatically bounced to his home library's router, where that router's knowledge of his own preferences would manage what happens next. This model of "bouncing" or "chaining" routers makes sense in today's world, where remote resources guess a user's affiliation based on IP addresses (and, thus, might continue to first send the visiting researcher to the local link router, rather than his home library's router, even if he enters databases from his home library's home page). Request "chaining" might come into play if the researcher wants both to store the reference in his reference library and to publish it to a weblog.

This "personalized routing" metaphor seems very conducive to imagining additional scenarios and information paths. These paths cross one continuum, from private to public, and cross another continuum from individual and group to aggregate. Adding in the wide range of potential uses for RSS flows of collected information from any arbitrary point in these systems to any other makes the mind spin just a little.

Implementation choices

How would we go about doing all of this? There seem to be two clear implementation paths, both of which involve enhancing existing systems. The first, and most obvious, is to bolt these services onto existing resolvers. Their backend databases are already well-tuned. It seems likely that expanding the rule engines they have for link resolution to add new hooks in the different request phases and more flexible routing/bouncing/chaining shouldn't be too complicated. Layering in functions into a user/group management toolkit shouldn't be an obstacle either, especially if we leverage recent work such as the Open Knowledge Initiative specifications for such systems.

A second implementation path involves integration with MyLibrary and UPortal-type systems. Interesting questions would include how close a binding might be necessary between personalized link routers and portals. Should all the personalization happen in a portal, and the routers be just arbitrary rule engines for service resolution? Should the portals just be tuned to be well-behaved sources and targets, but the personal routing functions live in the routers? It's easy to imagine that different institutions, with their widely varied I.T. administration models, would want different pieces to live under different management branches.

This need for design flexibility dovetails nicely with another reason to value solutions which enforce a clean separation of services. Ideally, it should be easy to integrate library groupware with external non-library toolkits (by "non-library" I mean "blogosphere and otherwise general internet community"). After all, as highlighted earlier in this article, many of these technical innovations are occuring nowhere near libraries. Honoring those distinctions by placing the "integrates well with others" design principle high on the agenda, perhaps we can find new ways to feed some of our own system innovations back to the general internet community, from which we have gained so much over the years.

Metadata requirements

The best news about all of this is that it seems the many smart folks behind the new OpenURL specifications thought through many potential service models, surely to include the kinds discussed here. In that context, it is again helpful to remember the for-libraries, not-for-libraries systems dichotomy, and how the services discussed here might span those boundaries. It probably wouldn't be difficult to find useful points of integration for nascent from-libraries specifications like OpenURL and nascent not-from-libraries specifications like the Atom API. If there isn't a clear way to do that, then it shouldn't be too hard to demonstrate something at a lower code-level, by offering up plugins for one or two major weblog toolkits to allow bibliographic entries and OpenURL linking.

The question of which bibliographic datatypes to support is, as ever, a thorny one. Ideally many types, from binary MARC to MODS XML, from arbitrary OWL instances in RDF triples to simple Dublin Core, would be supportable. To do that, we'd need to have a system implementing something like OCLC's metadata switch plugged in near the router, available for schema transforms, or ad hoc record enrichment, at any phase in the routing process. Layer on top of the metadata management layer an export/import/update system which can speak WebDAV, and the possibilities for integrating any user's data with other systems open up more widely.

A brief note on security

If you spend any time reading personal weblogs, you can see the shifting balance between information individuals choose to make public and what they keep private. Perhaps the trend to expose one's life more vividly and in excruciating, machine-processable detail is simply a corollary reaction to our growing awareness that private corporate and government structures are already aggregating and exchanging more data about us as individuals and groups than ever before. This backdrop, however, is no excuse to take privacy and system security lightly.

Without diving into the mess of shifting U.S. laws surrounding a library's responsibilities to its users and to governmental authorities, suffice it to say that great care would be necessary when wiring up systems such as one we've discussed. Any library system that allowed users to make choices about which data points of their own can be shared -- anonymously or not -- would expose libraries to managing more data about their users than we currently handle. And by definition, retaining more user information means we might be asked to divulge more, if so tasked by relevant authorities.

So we cannot merely hand-wave this issue away, but at least we can take solace in knowing that the smart folks working on efforts such as Shibboleth have made great headway in enabling environments like this. And that the testing already being done on those kinds of tools in the context of user authentication for library services will be very beneficial. Keeping in mind also that the general software community is already focused on delivering tools for academia, such as with the recent Mellon Foundation funding of the Chandler project, and the path to building solutions seems even shorter.