User Feedback Results – Super 8

In an effort to find the magic number the SALT team opened its testing labs again this week.  Another 6 University of Manchester post graduate students spent the afternoon interrogating the Copac and John Rylands library catalogues to evaluate the recommendations thrown back by the SALT API.

With searches ranging from ’The Archaeology of Islam in Sub Saharan Africa’ to ‘Volunteering and Society: Principles and Practice’ no aspect of the Arts and Humanities was left unturned, or at least it felt that way.  We tried to find students with diverse interests within Arts and Humanities to test the recommendations from as many angles as possible.  Using the same format as the previous groups (documented in our earlier blog post ‘What do users think of the SALT recommender?), the library users were asked to complete an evaluation of the recommendations they were given.  Previously the users tested SALT when the threshold was set at 3(that is 3 people borrowed the book which therefore made it eligible to be thrown back as a recommendation), however we felt that the results could be improved.  Previously, although 77.5% found at least one recommendation useful, too many recommendations were rated as ’not that useful’,(see the charts in ‘What do users think of the SALT recommender?’).

This time, we set the threshold at 15 in the John Rylands library catalogue and 8 in Copac.  Like the LIDP team at Huddersfield, (http://library.hud.ac.uk/blogs/projects/lidp/2011/08/30/focus-group-analysis/), we have a lot of data to work with now, and we’d like to spend some more time interrogating the results to find out whether clear patterns emerge.  Although, our initial analysis has also raised some further questions, it’s also revealed some interesting and encouraging results.  Here are the highlights of what we found out.

The Results

On initial inspection the JRUL with its threshold of 15 improved on previous results;

Do any of the recommendations look useful:

92.3 % of the searches returned at least one item the user thought was useful, however when the user was asked if they would borrow at least one item only 56.2% answered that they would.

When asked, a lot of the users stated that they knew the book and so wouldn’t need to borrow it again, or that although the book was useful, their area of research was so niche that it wasn’t specifically useful to them but they would deem it as ‘useful’ to others in their field.

One of the key factors which came up in the discussions with users was the year that the book had been published. The majority of researchers are in need of up to date material, many preferring the use of journals rather than monographs, and this was taken into account when deciding whether a book is worth borrowing. Many users wouldn’t borrow anything more than 10 years old;

‘Three of the recommendations are ‘out of date’ 1957, 1961, 1964 as such I would immediately discount them from my search’ 30/08/11 University of Manchester, Postgraduate, Arts and Humanities, SALT testing group.

So the book could be a key text, and ‘useful’ but it wouldn’t necessarily be borrowed.  Quite often, one user explained, rather than reading a key text she would search for journal articles about the key text, to get up to date discussion and analysis about it. This has an impact on our hypothesis which is to discover the long tail. Quite often the long tail that is discovered includes older texts, which some users discount.

Copac, with a threshold of 8 was also tested. Results here were encouraging;

Do any of the recommendations look useful;

Admittedly further tests would need to be done on both thresholds as the number of searches conducted (25) do not give enough results to draw concrete conclusions from but it does seem as if the results are vastly improved on increase of the threshold.

No concerns about privacy

The issue of privacy was raised again. Many of the postgraduate students are studying niche areas and seemed to understand how this could affect them should the recommendations be attributed back to them. However, as much as they were concerned about their research being followed, they were also keen to use the tool themselves and so their concerns were outweighed by the perceived benefits. As a group they agreed that a borrowing rate of 5 would offer them enough protection whilst still returning interesting results. The group had no concerns about the way in which the data was being used and indeed trusted the libraries to collect this data and use it in such a productive way.

‘It’s not as if it is being used for commercial gain, then what is the issue?’ 30/08/11 University of Manchester, Postgraduate, Arts and Humanities, SALT testing group.

Unanimous support for the recommender

The most encouraging outcome from the group was the uniform support for the book recommender. Every person in the group agreed that the principle of the book recommender was a good one, and they gave their resolute approval that their data was collected and used in a positive way.

All of them would use the book recommender if it was available. Indeed one researcher asked, ‘can we have it now?’

Janine Rigby and Lisa Charnock 31/08/11

What do the users think of the SALT recommender?

Following internal in house testing the recommender was open to the users. In the last week of July 18 humanities postgraduates passed through the SALT testing labs, (11 PhD students, 3 taught Masters students and 4 research students). Lisa and I held three focus groups and grilled our potential users about the SALT recommender. The research methods used were designed to answer our objectives, with an informal discussion to begin with to find out how postgraduate students approach library research and to gauge the potential support for the book recommender. Following the discussion we began testing the actual recommender to answer our other research objectives which were:

  • Does SALT give you recommendations which are logical and useful?
  • Does it make you borrow more library books?
  • Does it suggest to you books and materials you may not have known about but are useful and interesting?

As a team we agreed to set the threshold of the SALT recommender deliberately low, with a view to increasing this and testing again if results were not good. As our hypothesis is based on discovering the hidden long tail of library research we wanted the recommender to return results that were unexpected – research gems that were treasured and worthy items but had somehow been lost and only borrowed a few times.

42 searches in total were done on the SALT recommender and of those 42, 77.5% returned at least one recommendation, (usually many more) that participants said would be useful. (As an aside, one of the focus groups participants found something so relevant she immediately went to borrow it after the group has finished!)

However the deliberately low threshold may have caused some illogical returns.  The groups were asked to comment on the relevance of the first 5 recommendations, but quite often it was the books further down the list that were of more relevance and interest.  One respondent referred to it as a ‘Curate’s egg’ however, assured me this was in reference to some good and some bad. His first five were of little relevance, ‘only tangentially linked’, his 6th, 7th, 8th, 9th, 11th and even 17th recommendations were all ‘very relevant’. Unfortunately this gave disappointing results when the first 5 suggested texts were rated for relevance, as demonstrated in the pie chart below.

However the likelihood of borrowing these items gave slightly more encouraging results;

Clearly we’ve been keen on the threshold count.  Lessons need to be learnt about the threshold number and this perhaps is a reflection of our initial hypothesis. We think that there would be much merit in increasing the threshold number and retesting.

On a positive note, initial discussions with the researchers (and just a reminder these are seasoned researchers, experts in their chosen fields familiar and long term users of the John Ryland’s University Research Library) told us that the recommender would be a welcome addition to Copac and the library catalogue. 99% of the researchers in the groups had used and were familiar with Amazons recommender function and 100% would welcome a similar function on the catalogues based on circulation records.

Another very pertinent point, and I cannot stress this strongly enough, was the reactions expressed in regards to privacy and collection and subsequent use of this data. The groups were slightly bemused by questions regarding privacy. No one expressed any concern about the collection of activity data and its use in the recommender. In fact most assumed this data was collected anyway and encouraged us to use it in this way, as ultimately it is being used to develop a tool which helps them to research more effectively and efficiently.

Overwhelmingly, the groups found the recommender useful. They were keen that their comments be fed back to developers and that work should continue on the recommender to get the results right as they were keen to use it and hoped it would be available soon.

What do the library users think?

As the SALT project and the Activity Data programme progresses, I’m finding the results of the various user engagement exercises really interesting.  As Janine’s already mentioned, we’re planning a structured user evaluation of our recommender tool with subject librarians and researchers, but before that we wanted to talk to some students to test some of our assumptions and understand library users experiences a little better.

So, last week I took myself off to the JRUL and interviewed four students (three postgraduates and one undergraduate).  In the main, I was (of course) interested in their opinions about recommenders, and whether they would find such a tool useful in the JRUL library catalogue and in Copac.  There is a lot of evidence to suggest that researchers would find the introduction of a recommender beneficial (not least from the other blogs on this programme), but what would the Manchester students and researchers think?  I was also interested in their research behaviour – did they always know exactly what they were looking for, or did they do subject and keyword searches?  And finally, I wanted to sound them out about privacy.

So what did they tell me?

On recommendations

There was varied use of recommenders through services like Amazon, but all of the students could see the potential of getting recommendations through the library catalogue, Copac, and eventually through the Library Search (powered by Primo).  There were some concerns about distractions, with one student worried that she would spend her limited research time following a never ending cycle of recommendations that took her further and further away from her original purpose.  However, the biggest concerns from all four was the possibility of irrelevant material being pushed to them – something that they would all find frustrating.  A recommender could certainly help to widen reading choices, but all of them wanted to know how we were going to make sure that the suggestions were relevant.  I noticed that the postgraduate focus group participants in the Rise focus groups needed to trust the information, and were interested to know where the recommendation has come from.  It’s clear that trust is a big issue, and this is something we’ll definitely be re-visiting when we run the user evaluation workshops.

On research behaviour

On the whole, the participants knew what they were looking for when they opened the catalogue, and suggestions of material came from the usual suspects – supervisors, tutors, citations, or specific authors they needed to read.  All of them felt that recommendations would be interesting and especially useful during extended research projects such as dissertations.  However, what was most interesting to me was that, although they all said they would be interested to look at the suggestions, they all seemed unconvinced they would actually borrow the recommended books because they, on the whole, visited the catalogue in order to find specific items.  So what does this mean for our hypothesis – that using circulation data can support research by surfacing underused library materials?  These students didn’t have the opportunity to try the recommender, so you could argue that some scepticism is inevitable, and Hudderfield’s experience suggests that underused books will resurface.  However, again we need to explore this further once we can show some students a working prototype.

On privacy

I wasn’t sure whether privacy would be an issue, but none of the students I spoke to had any concerns about the library collecting circulation data and using it to power recommendations.  They considered this to be a good use of the data, as long as anonymity was taken into consideration.  On the whole, the students’ responses backed up the findings of the 2008 Developing Personalisation for the Information Environment Final Report, which found that “students place their trust in their college or university. They either think that they have no alternative to disclosing information to their institution, or believe that the institution will not misuse the information.”  They felt that, by introducing a recommender, the library was doing “a good thing” by trying to improve their search experience.  No concerns here.

Next Steps

Obviously, this was only the views of four students, and we need to do more work to test the usefulness of the tool.  We’re now planning the user testing and evaluation of the recommender prototype, and recruiting postgraduate humanities researchers to take part.  As Janine outlined, we’ll be introducing the tool to subject librarians at JRUL and humanities researchers to see if the recommendations are meaningful and useful.

I’m looking forward to finding out what they think, and we’ll let you know the results in a later blog post.