Copac Developments

I thought I’d write a note about why we lost the Copac service for a couple of hours yesterday.

The short of it is, that our database software hung when it tried to read a corrupted file in which it keeps track of sessions. The result was that everyone’s search process hung and so frustrated users kept re-trying their searches, which created more hung sessions until the system was full of hung processes and with no CPU or memory left. Once we had deleted the corrupted file, everything was okay.

The long version goes something like this… From what I remember, things started going pear-shaped a little before noon when the machine running the service started becoming unresponsive. A quick look at the output of top showed we had far more search sessions running than normal and that the system was almost out of swap space.

It wasn’t clear why this was happening and because the system was running out of swap it was very difficult to diagnose the problem. It was difficult to run programs from the command line as, more often than not, they immediately died with the message “out of memory.” I did manage to shutdown the web server in an effort to lighten the load and stop more search sessions being created. It was proving almost impossible to kill off the existing search sessions. In Unix a “kill -9″ on a process should immediately stop the process and release its memory back to the system. But yesterday a “kill -9″ was having no effect on some processes and those that we did manage to kill were being listed as “defunct” and still seemed to be holding onto memory. In the end we just thought it would be best to re-boot the system and hope that it would solve whatever the problem was.

It took ages for the system to shut itself down — presumably because the shutdown procedures weren’t working with no memory to work in. Anyway, it did finally reboot and within minutes of the system coming up it became overloaded with search sessions and ran out of memory again.

We immediately shut down the web server again. However, search sessions were still being created by people using Z39.50 and so we had to edit the system configuration files to stop inetd spawning more Z39.50 search sessions. Editing inetd.conf didn’t prove to be the trivial task it should have been, but we did get it done eventually. We then tried killing off the 500 or so search sessions that were hogging the system — and that proved difficult too. Many of the processes refused to die. So, after sitting staring at the screen for about 15 minutes, unable to run programs because there was no memory and wondering what on earth do we do now, the system recovered itself. The killed off processes did finally die, memory was released and we could do stuff again!

A bit of investigation showed that the search processes weren’t getting very far into their initialisation procedure before hanging or going into an infinite loop. I used the Solaris truss program to see what files the search process was reading and what system calls it was making. Truss showed that the process was going off into cloud cuckoo land just after reading a file the database software uses to track sessions. So I deleted that file and everything started working again! The file got re-created next time a search process ran — presumably the file had become corrupted.

Some of you may have noticed that there is now a facility on the Copac search forms to search your local library catalogue as well as Copac. You’ll only see this option if you have logged into Copac and are from a supported library.

The searching of the local library catalogues and Copac is performed using the Z39.50 search protocol. Due to differences in local configurations the query we send to Copac and the various library catalogues have to be configured very differently.

When we built the Copac Z39.50 server, we tried to make it flexible in the type of query it would accept within the limitations imposed upon us by the database software we use. Our database software was made for keyword searching of full text resources. As such it is good at adjacency searches, but you can’t tell it you want to search for a word at the start of a field.

Databases built around relational databases tend to be the complete opposite in functionality. They often aren’t good at keyword searching, but find it very easy to find words at the start of a field.

The result of which is that we make our default search a keyword search, while some other systems default to searching for query terms at the start of a field. Hence if we send the exact same search to Copac and a library catalogue we can get a very different result from the two systems. To try and get a consistent result we have to tweak the query sent to the library so that it performs a search as near as possible to that performed by Copac. Working out how to tweak (or transform or mangle) the queries is a black art and we are still experimenting.

Stop word lists are also an issue. Some library systems like to fail your search if you search for a stop word. Better systems just ignore stop words in queries and perform the search using the remaining terms. The effect is that searching for “Pride and prejudice” fails on some systems because “and” is stop worded. To get around this we have to remove stop words from queries. But we first need to know what the stop words are.

The result is that the search of other library systems is not yet as good as it could be, though it will get better over time as we discover what works best with the various library systems that are out there.

Now that you have the option to log-in to Copac to use the personalisation features, here are some tips to make logging in as easy as possible.

Typekey/Typepad:  if you have a Typekey or Typepad account, and were wondering where your login option was, worry no longer!  From the drop-down list of organisations on the login page, you need to choose ‘JISC project: SDSS (TypeKey Bridge)’.  It’s not immediately obvious, but it is the correct login option for any TypeKey users.

Navigating the list:  the list of organisations is very long, and weighted heavily towards ‘U’.  To navigate it more easily, you can jump straight to any letter by typing it on your keyboard.  You may find it even easier to enter a keyword search in the search box.  This will work for partial words as well – entering ‘bris’ will give you the options of the City of Bristol College and the University of Bristol.

Remembering your selection:  once you have found your organisation, there are options to have your selection remembered, either for that session (the default) or for a week.  You can also choose ‘do not remember’, which is especially useful if you are on a public computer.

Please contact us if you experience any problems with logging in to Copac.

It’s finally here!  After months of very hard work from the Copac team, and lots of really useful input from users on the Beta trials, the new Copac interface is now live.

We have streamlined the Copac interface, and you can still search and export records without logging in to Copac.  This is ideal if you want to do a quick search, and don’t need any of the additional functionality.  Users who choose not to login will still be able to use the new functionality of exporting records directly to EndNote and Zotero, and will see book and journal tables-of-contents, where available.

You now also have the option to login to Copac.  This is not compulsory, and you only need to login if you want to take advantage of the full range of new personalisation features.   These have been developed to help you to get the most out of Copac, and to assist in your workflows.

‘Search History’ records all of your searches, and includes a date/time stamp.  This allows you to keep track of your searches, and to easily re-run any search with a single click.

‘My References’ allows you to manage your marked records, and create an annotated online bibliography.

You can annotate and tag all of your searches and references.  There is no limit to how you can use this functionality:  see my post from March for some suggestions about how you might use tags and annotations.  We would love to hear how you are using them – please get in touch if you would like to share your experiences and ideas.

Users from some institutions will now have the option to see their local catalogue results appearing alongside the Copac results.  We are harvesting information from the institutions’ Z39.50 servers, and using this to create a merged results set.  If you are interested in your institution being a part of this, please get in touch.

Some people have expressed concern that the need to login means that Copac is going to be restricted to members of  UK academic institutions only.  This is not the case.  We are committed to keeping Copac freely-accessible to all.  Login is required for the new features to function:  we need to be able to uniquely identify you in order to record your search history and references; and we need to know which (if any) institution you are from to show you local results.  We have tried to make logging in as easy as possible.  For members of UK academic institutions, this means that you can use your institution’s central username/password, or your ATHENS details.  For our users who aren’t members of a UK academic institution, you can create a login from an identity provider:  ProtectNetwork and TypePad.  These providers enable you to create a secure identity, which you can use to manage access to many internet sites.

We are very grateful to everyone who has taken the time to give us feedback on the recent Beta trials.  But we can never get enough feedback!  We’d love to hear what you think about the new Copac interface:  you can email us; speak to us on twitter; or leave comments here.

One of the new features we are trailing in the new Copac Beta is the searching of your local institutions library catalogue alongside Copac. To do this we need to know which Institution you are from and whether or not your Institutional library catalogue can be searched with the Z39.50 protocol.

To identify where you are from, we are using information given to us during the login process. When you login, your Institution gives us various pieces of information about you, including something called a scoped affiliation. For someone logging in from, say, the University of Manchester, the scoped affiliation might be something like “student@manchester.ac.uk”

Once we know where you are from, we search a database of Institutional Z39.50 servers  to see if your Institution’s library is searchable. If it is we can present the extra options on the search forms, and indeed, fire off any queries to your library catalogue.

Our database of Z39.50 servers is created from records harvested from the IESR. So, if you’d like your Institution’s catalogue available through Copac, make sure it is included in the IESR by talking to the nice people there.

Many thanks to everyone who tried the Beta interface early on and discovered that this feature mostly wasn’t working. You enabled us to identify some bugs and get the service working.

Users from some Institutions had been unable to login in Copac Beta. Thanks to help from colleagues we think we have now resolved the issue which was related to an exchange of security certificates between servers. The result was that a handful of Institutions were not trusting us and so were not releasing the anonymised username that we require. This seems to be fixed now and we’ve noticed that users from those Institutions can now login.

So, if you tried to login to Copac Beta and received a “Login failed” message, please try again. And please let us know if you still can’t get access.

[Jargon alert] I just noticed that the Shibboleth TargetedIds (read anonymised usernames) we are seeing when people log into Copac Beta are much longer than I expected. Some of them may have become truncated when saved in our user database. So, I’ve just increased the field size in the database. This may mean that some people will have lost their search history and references. Sorry about that. But a Beta test version is all about finding out about such niggles.

Thanks for persevering everyone.

[Added 30/3/2009] Tagging in ‘My references’ currently broken. We’re looking into it and hope to have it fixed soon.

[31/3/2009] Tagging issues now look to be fixed. However, when I was looking through the logs I spotted another problem which may reveal the reason some people are having difficulties searching Beta. Investigations are in progress.

[1/4/2009] It looks like some people are unable to gain access to Copac Beta because their Identity Provider isn’t providing us with an anonymised userid, or to use the jargon, a TargetedID. We do need this so we know which is your Search History and References.

There’s not a lot we can do about this. It is up to your Institution to release the TargetedID to us. However, if you are getting a “Login Failed” message please contact us, telling us which Institution you are from and we’ll try hassling your system admins.

With the new Copac interface, we wanted to make the Search History and Marked List (now re-named My References) more useful. Previously, these features were session based — that is, if you re-started your web browser, your search history and saved records were lost. For us to be able to retain that data over multiple sessions, we need to know who our users are. Hence, for Copac Beta we are forcing you to login.

The advantage of logging in is that you can use Copac Beta from multiple machines at different times and still have access to the searches and references you saved yesterday or last week – or even last year.  Unfortunately, log-in is currently restricted to members of UK Access Federation institutions (most UK HE and FE institutions, and some companies), but don’t worry – there will always be a free version of Copac open to everyone, and we will be widening the log-in scope in the future.

You can tag your searches and references and use a tag cloud to see those items tagged with a particular tag. We are automatically tagging your saved searches and references with your search terms, and you can remove these automatic tags, and add your own.  These tags are then added to your tag cloud, so that you can easily navigate your saved records through tags which are meaningful to you.  Why would you want to delete the automatically generated tags?  Well, records are tagged with all of your search terms so, if you limit your search to ‘journals and other periodicals’, the tags for records from that set will include ‘journals’ ‘other’ and ‘periodicals’.  If you find these confusing, you can just delete them, and have only tags that have meaning for you.

You can also add notes to any of your references – perhaps to remind yourself that you have ordered the item through inter-library loan, and when you should go and pick it up, or perhaps to make comments about how useful you found the item.  This ‘My References’ section was developed as part of the JISC-funded project Discovery to Delivery at Edina and Mimas (D2D@E&M) as a Reusable Marked List workpackage.

You can also edit the bibliographic details of the item.  These edited details are only visible to you, so you don’t have to worry about making any changes.  You could use this to correct a typo or misspelling in the record, or add details that are not visible in the short record display, such as information about illustrations or pagination.

The search history feature allows you to re-run any previous search with a single click, from any screen.  This could be especially useful for anyone who is doing demos, as not only do you know that the search will return results, but it saves you from the jelly fingers that haunt the even the most proficient of typists when in front of an audience.  The date and time of previous searches are recorded, so that you can see what you have searched for and when.  This could be useful for tracking the progress of a project over time, or showing at a glance what effect refining a search has on the number of results.

Many journal records now contain the latest table-of-contents.  Clicking on an article title will take you through to the Zetoc record for that article, and from there you can use the Open-URL resolver to link directly to full-text (if your institution has access), or order the article through your institution or directly from the British Library.  The table-of-contents allows you to get an idea of the scope of the journal, and whether it will be of interest to you, without going to another website. This makes it easier to avoid wasted travel or unnecessary inter-library loan requests.

We’d love to know what you think of these new features – and any suggestions you might have for new ones!  Once you’ve used the new features, please fill-in our questionnaire, to help us learn what we’re doing right, and what you’d like to see changed.  As thanks for your feedback, there’s a £35 Amazon voucher up for grabs for one lucky respondent.  The survey has 9 questions, and shouldn’t take more than 10 minutes of your time.  Of course, you can always give us additional feedback through the comments on this blog, by emailing copac@mimas.ac.uk, by phone or post, or Twitter.  But we’d really like you to do the survey as well

The Search History and My References feeatures of the Copac Beta Test Interface are stored in a database with an Atom Publishing Protocol (APP) Interface. The idea is to make the database open to use by other people and services and so enable re-purposing of the data.

Authentication poses a problem. We need to authenticate so that we can identify the user and show them their records and not someone elses. We didn’t want people to have to register to use Copac and neither did we want to get into developing a mechanism to handle user registration, etc. So, we have used the JISC supported UK Federation (aka Shibboleth) Access Management system. This allows users to login to Copac using their own instiutional username. Registering separately with Copac is not needed to gain access.

The downside is that Shibboleth is designed to work with web browsers. I don’t know the technacalities of it all, but a login with Shibboleth seems to involve multiple browser redirects, possibly a WAYF asking “Where are you From?” and a web page with a bunch of Javascript that the browser has to interpret that redirects the browser yet again. I’ve tried accessing the Shibboleth protected version of our APP Interface with some APP client software and none of it could get past the authentication — however, it is very hard to diagnose where the problems are.

I also tried the command line program “curl” to access the APP Interface and while it can handle the redirects and the username and password I think it fails when it gets to the page with the Javascript. Which is fair enough, “curl” isn’t a web browser, it is just a program that retrieves urls.

So, can we make do without Shibboleth? Well we can, but the options are either not terribly insecure or not practical. The options I can think of are:

1. We put a token (eg a unique id) in the url. This effectively makes the users collection of records and search history public if the url is published.
2. We put the token in a cookie. This is still insecure and subject to cookie highjacking, but is more private as the token isn’t in the url. Many high profile web sites seem to use such an cookie for authentication, and if they do, then I don’t see why we shouldn’t? However, I’m not sure how practical it is to get third party APP clinet software to send the cookie — unless the APP client was written as part of a web browser that already has the cookie.

You can try accessing the Shbboleth protected APP server for yourself at the following url:

• https://copac.ac.uk/atom/

If you’ve already used the Copac Beta then your Search History and My References collections can be found at the following urls in the form of Atom feeds:

• https://copac.ac.uk/atom/saved-searches/
• https://copac.ac.uk/atom/my-references/

Please let us know how you get on! I’ve tried the above urls with Firefox and Safari. Firefox gets through the authentication and displays the Atom feeds and Service Documents. Safari seems to put itself into an infinite loop whilst trying to display the feed (maybe this is something to do with the XML in our Atom feed?)

We’d be very interested to hear your thoughts on the above.

As the new Copac beta test interface is now storing users’ search history in a database we needed Copac search urls to be stateless (or RESTful.) If you look at the current Copac urls, you will notice as you navigate through a result set, just how much saved state is encoded in the url. There are references to the session ID and the number of your query within your session.

In the new scheme of things, that is all gone and I believe our search urls are now stateless — that is, all the information needed to display a search result is now encoded in the url. The CGI script serving the url does not have to go delving into a database to work out what to do.

I’ll attempt here to explain the new url scheme and hopefully you will see how it can be used as a machine to machine interface to Copac. I should point out though, that this is describing the beta version and things may change in the future.

So, to perform an author query against the Copac database, all you need is a url like this:

http://beta.copac.ac.uk/search?au=sutter

The above url will perform an author search for “sutter” and will display an HTML rendered page showing the first page of brief records. If you would like the results sorted, then you can add a “sort-order” element to the url as follows:

http://beta.copac.ac.uk/search?au=sutter&sort-order=ti

The above url will sort the query by the record title field. If the result set is too large to sort, then you will be redirected back to the same query without the sort-order.

If you want to view the first full record in a result set, then add an “rn” element to the url:

http://beta.copac.ac.uk/search?au=sutter&rn=1

Similarly, to view the second page of brief records:

http://beta.copac.ac.uk/search?au=sutter&page=2

All the above urls return an HTML display — not what you want for machine to machine communication. So, to get some programmer friendly XML you can add the “format” element to the url:

http://beta.copac.ac.uk/search?au=sutter&page=1&format=XML+-+MODS

The above url returns a page of MODS XML records. A page, by default, is 25 records. If you’d prefer more or less records in a page, then you can set the page size by sending a “Page-size” header with the HTTP request. And, so that you know how large the result is, a “Result-set-size” header is returned with the HTTP response when a “format” is specified in the url.

You can, of course, specify a “sort-order” along with a “format”. You’ll be able to discover the various query fields, sort and format options by delving around the user interface and performing a few queries. I’m not going to document them here and now as it is all still beta and they may change before we go live.