Search This Blog

Wednesday, 12 October 2016

The number behind the number: suggesting a truer measure of academic impact

A close colleague and friend of Information Resources, Dr Chris Carroll has just published a paper on the limitations of citations, in particular the single counting of them within a paper's reference list. Chris wrote about his paper in the excellent LSE Impact Blog and we thought it would be of interest to visitors of the Information Resources Blog. We have republished it under their Creative Commons licence. Dr Carroll is a member of HEDS and worked previously as an information specialist as part of our team.
The limitations of simple ‘citation count’ figures are well-known. Chris Carrollargues that the impact of an academic research paper might be better measured by counting the number of times it is cited within citing publications rather than by simply measuring if it has been cited or not. Three or more citations of the key paper arguably represent a rather different level of impact than a single citation. By looking for this easily generated number, every researcher can quickly gain a greater insight into the impact (or not) of their published work.
The academic research and policy agenda increasingly seeks to measure and use ‘impact’ as a means of determining the value of different items of published research. However, there is much debate about how best to define and quantify impact, and any assessment must take into account influence beyond the limited bounds of academia, in areas such as public policy. However, within academia, it is generally accepted that the number of times a paper is cited, the so-called ‘citation count’ or ‘citation score’, offers the most easily measured guide to its impact. The underlying assumption is that the cited work has influenced the citing work in some way, but this metric is also viewed as practically and conceptually limited. The criticism is loud and frequent: such citation scores do not tell us how a piece of research has actually been used in practice, only that it is known and cited.
Image credit: beyond measure by frankieleon. This work is licensed under a CC BY 2.0 license.
Is there a better solution?
So, the obvious answer is to see if there is an easy way to measure how a paper has been used by its citing publications, rather than simply recording whether it is cited (the standard ‘citation score’). I did this by using one of my own frequently-cited papers as a case study and applying a basic metric: impact of the case study paper was considered to be ‘high’ if it was cited three or more times within the citing publication; ‘moderate’ if cited twice; and ‘low’ if cited only once.
The case study paper had 393 unique citations by November 2015: 59% (230/393) of publications cited the paper only once, suggesting its impact on these publications was low; 17% (65/393) cited it twice, suggesting moderate impact; but 25% (98/393) cited it three or more times, suggesting that this paper was having a genuine influence on those studies. The citation frequency within this ‘high impact’ group of publications ranged from three to as many as 14 in peer-reviewed studies and 17 in academic dissertations (with their longer word count). Primary research studies published in peer-reviewed journals were the principal publication type across all levels of impact.
I also noted where these single or multiple citations appeared within these publications. Single citations tended to appear only in the introduction or background sections of papers. These instances appeared to be simple citations ‘in passing’, the necessary ‘nod’ to existing literature at the start of a publication. However, when there were three or more citations, they tended to appear across two or more sections of a paper, especially in the methods and discussions, which suggests some real influence on the justification or design of a study, or the interpretation of its results. These findings on the location of the citations confirmed that the number of times a paper was cited within a publication really was a good indicator of that paper’s impact.
Pros and cons
A metric based on within-publication citation frequency is unambiguous and thus capable of accuracy. The assessment of where the citations appear in the citing publications also provided contextual information on the nature of the citation. In this case, it confirmed the viability of within-publication citation frequency as an impact metric. The approach is transparent and data can be verified and updated. The proposed metric, the ‘citation profile’, is not a measure of research quality, nor does it seek to measure other forms of impact or to address issues such as negative findings producing relatively lower citation scores. It merely seeks to contribute to the debate about how citation metrics might be used to give a more contextually robust picture of a paper’s academic impact.
Of course, even this ‘citation profile’ is not without its problems, but it is arguably less of a“damned lie” or “statistic” than some other metrics: only a more in-depth analysis of each citing publication can more accurately gauge a paper’s influence on another piece of research or a policy document. Yes, there are also questions concerning this metric’s generalisability: as with other bibliometrics, publications in different disciplines are also likely to have different citation profiles. However, these issues can be easily addressed by future research, given the simplicity of the metric.
If academics or funders want to improve their understanding of how research is used, then this easy-to-generate metric can add depth and value to the basic, much-maligned, and ‘damned’ ‘citation score’.
This blog post is based on the author’s article, ‘Measuring academic research impact: creating a citation profile using the conceptual framework for implementation fidelity as a case study’, published in Scientometrics (DOI: 10.1007/s11192-016-2085-0).
Note: This article gives the views of the author(s), and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the author
Chris Carroll is a Reader in Systematic Review and Evidence Synthesis at the University of Sheffield. His role principally involves synthesising published data to help inform policymaking, particularly in the fields of medicine and health, as well as the development of methods to conduct such work.
Print Friendly

Tuesday, 4 October 2016

Librarian on the run!

Claire Beecroft
Claire Beecroft
Continuing ScHARR Library's support for the Sheffield Charity Inspiration for Life, Claire Beecroft (that's me) and Angie Rees from the ScHARR Library team will be running the 'challenging' (for me at least) TenTenTen 10k in Endcliffe Park and Whiteley Woods on Sunday 9th October. This is a trail race with four nasty hills and lots of opportunities to fall over in the mud. Yay!

Inspiration for Life is the charity set up to honour and celebrate the memory of Tim Richardson, a University of Sheffield colleague, who passed away on 5 February 2013 from cancer. More recently, following the passing of another colleague, Victoria Henshaw, also from cancer,  the charity has also dedicated itself to supporting the memory and work of both these special people. They aim to promote lifelong learning, and encourage the public understanding of science through publications, lectures and other events. They also directly support local cancer charities.

I have done a fair few races over the years, though nothing beyond 5k for a while, and you can hunt me down at Sheffield Hallam Parkrun most Saturday mornings- here's me at last week's (just to prove I CAN run):

                                                                               Image by Dougalpics

You can sponsor the team I'm running for at:

Posted by Claire.

Wednesday, 21 September 2016

MmIT 2016 Conference Review - Digital Citizenship : What is the library's role?

Photo of Sheldon Korpet
Sheldon Korpet
Our latest recruit to ScHARR's Information Resources Sheldon Korpet attended the MmIT Conference last week and writes about her experiences over the two days. Sheldon is an Information Officer at ScHARR and a current MSc Digital Library Management student at the University of Sheffield.

This year's Multimedia and Information Technology Conference (MmIT) focused on “Digital Citizenship: What is the library's role?” and included a fabulous range of talks from librarians, head of services, computer specialists and suppliers which really reflects the scope and depth of the topic. As a relatively new professional, one who had never attended a conference before, I decided to join in mainly because I’d heard good things about the food. However I am always keen to broaden my horizons and I’m pleased to say not only did the food exceed my expectations, so did the conference itself. MmIT are a special interest group within Cilip.
MmIT Journal
MmIT Journal

Dr Chris Stokes (Joint Director of Digital Learning, University of Sheffield) spoke about his team’s process to make the University of Sheffield’s first Massive Open Online Course (MOOC). This was inspired by an outreach scheme designed to give 16 to 18 year olds the knowledge, information and guidance to make a competitive application to the University of Sheffield’s Dental School. This digital course aimed to use technology to increase access to a course that promoted inclusion within higher education and unexpectedly inspired an online community that facilitated communication between a range of user groups. This group ranged from aspirational A Level students to individuals with fear of dentists and even dental nurses refreshing their knowledge before returning to work after maternity leave.

What I learnt from Chris: MOOCs aren’t just for graduates, or a specific age range - they can be all inclusive and empowering if you design interesting, interactive content which simplifies the subject.

This issue of safe and knowledgeable digital access for the masses was something which inspired Lee Fallin and Mike Ewan to create a website called “The Digital Student”. This project aimed to educate students at the University of Hull by creating a mobile optimised website applicable to all students which could also be embedded within the VLE (Virtual Learning Environment). This idea not only gives students convenient access to information from a point on the university system which they already utilise but also standardised, quality information which can be updated centrally and easily replicated over a large number of users quickly. Further to this, the site is also mobile enabled and also designed to appeal to a range of different abilities using a content layout inspired by the popular website, Buzzfeed. After creating this content to improve the digital literacy of their students, the project will next release guides which aim to help students sell the digital skills when applying for jobs.

What I learnt from Lee and Mike: Take content to channels students are already using and display it in a way they are familiar with, in a format they like.

The socio-economic benefits of digital literacy was the focus of Helen Milner CEO of the Tinder Foundation. Her keynote speech, based on digital social inclusion, explained work of the Tinder Foundation which aims to connect and empower individuals. This is as a result of an aim to raise not only digital literacy locally, but also for the awareness of the need and benefits of digital literacy and digital access on a national scale.

What I learn from Helen: In 2015, a shockingly large amount (12.6 million) of the population were still offline. What was even more shocking was finding out was that if you were uneducated, retirement age, disabled or had a low income you were less likely to have access to the internet despite the potential benefits including being able to apply for jobs (25% of which are online-only applications), making online savings as well as maintaining social connections.

Check out this brilliant infographic that captures the data around digital literacy and accessibility

Ian Clark
Ian Clark
Communication and digital access was a key part of Ian Clark's talk focusing on the digital privacy divide. He highlighted the clash between the government’s desire to protect society by observing their browsing and borrowing history versus the negative effects of surveillance on information exchange. Although there are benefits of surveying information exchange between individuals and groups undertaking illegal activities however GCHQ have undertaken projects (such as “Karma Police”) which aim to create unique profiles for each individual of the population, regardless of whether they are law-abiding citizens or not.

Although there are methods to avoiding online observation, such as encryption technologies and browsers such as Tor, these have received a negative spin as a result of illegal activities linked to the Dark Web and the popular attitude, “you have nothing to worry about, if you have nothing to hide”. These tools, while increasing security for individuals who want to protect their communications, are also complex to execute.

What I learnt from Ian: Individuals who thought they were under surveillance changed their information searching behaviours and automatically censored themselves. This arguably reduces freedom of thought which is indusive to critical thinking, idea generation and the democratic process.

Check out Infoism for more information

Dr Kevin Curran
Dr Kevin Curran
Another highly informative talk was hosted by Kevin Curran, who is a Reader in Computer Science at Ulster University. His overview, “Hacking: Child’s Play” highlighted just how hackers can locate unsecured databases and webcams by using Google dorks, hold a company to ransom with a simple denial of service attack program and receive money from illegal activity anonymously using a cryptocurrency such as Bitcoin.

What I learnt from Kevin: Great tips to improve your security online.

  • Check if your details have been breached. You can also sign up to the “Notify me” service which will tell you when you need to change your password so no unauthorised snoopers can access your accounts
  • Don’t invest in expensive anti-virus. Windows Defence is freely available available from Microsoft and will do just as good a job as Norton and other popular paid-for antivirus softwares.
  • Don’t use the same password for every website. You can use a password manager like LastPass (link to simply this for you

All of the presentations I attended were thoughtful and particularly made me reflect on the idea of using the benefits of technology to empower - Chris and his team's outreach MOOC gave disadvantaged students the chance to participate in a highly competitive career path. Additionally, Helen and the Tinder Foundation, as well as Lee and Mike from University of Hull, had recognised the need for further digital literacy training and the benefits of enabling individuals to use technology, in terms of both increasing employability and confidence.

Digital Privacy & Digital Citizenship
Digital Privacy & Digital Citizenship
The knowledge brought to the conference by Ian of Infoism and Kevin of Ulster University did however highlight some of the areas in which there is much at stake if individuals and organisations use the technology in an irresponsible way. The main thing I’ve taken away from the conference is the idea that using digital technology is like driving a car - it doesn’t matter how bright, young or reactive you are - if you aren’t taught how how to operate the machine and navigate the system there’s the potential for a crash.

Tuesday, 20 September 2016

Transforming your service: the right evidence at the right time and place: Virtual Issue of HILJ edited by Anthea Sutton

To mark the 2016 Health Libraries Group (HLG) Conference, Health Information and Libraries Journal (HILJ) has published a virtual issue on the topic of “Transforming your service: the right evidence at the right time and place”. The virtual issue, edited by Anthea Sutton, contains articles and features published in HILJ in the past two years (2014-2016), on the theme of service transformation, and the related key strategic themes outlined in Knowledge for Healthcare (KFH).
The selected articles and features all demonstrate initiatives in health care library and knowledge services in line with these strategic themes, including optimising investment,quick and easy access to relevant evidence, and planning and development of the knowledge service workforce. The virtual issue aims to give an overview of KFH, highlighting examples of good practice and complementing the presentations and discussions at HLG 2016.
The 2016 HILJ Virtual issue is freely available online, and includes a video abstract.
This post originally appeared on the Knowledge for Healthcare blog and has been reproduced with permission.

Public Health Informatics

As this is the beginning of the academic year, I get to meet our new Master’s students to chat to them about their module choices this year. This is part of my role as co-ordinator of a public health informatics module. Students can study the module as a CPD opportunity or as part of one of our Master’s programmes.
Informatics brings together two underlying disciplines: computer science and information science. Public health informatics is about applying an informatics solution to public health problems. It offers additional approaches to disease surveillance and facilitates service planning and public health research.  In the module we take an overview approach and look at various case studies on how informatics can be applied, such as in health promotion activities or in a disaster response situation. We also focus on how to evaluate an information system by identifying and applying an evaluation framework.  
Last year we created a new session on ethics in public health informatics looking at ethical theories and how they are enacted in practice. Ethics should be a live issue, and we considered the limitations and usefulness of ethical codes by discussing key papers in a journal club format.

So I’m looking forward to working with our new students and the fabulous public health informatics team for the new academic year. For more information on the module which we provide in an online and on-campus format please see the relevant module pages or contact me at 

Picture credit: activity tracker by Zed Huang via Flickr (CC-BY-NC-SA 2.0)

Wednesday, 7 September 2016

Altmetric Ambassador of the Month

On the back of his edited book - Altmetrics, A Practical Guide for Librarians, Researchers and Academics - Andy Tattersall has been picked for's Ambassador of the Month. He was interviewed by about his role and work with altmetrics.

The interview is below

Tell me about your current work at The University of Sheffield. What does a typical day involve for you?

I am an Information Specialist at The School of Health and Related Research (ScHARR) at The University of Sheffield. My job has increasingly changed over the last few years, if anything I have become a digital research specialist/technologist with a strong focus on scholarly communications. I don’t have a typical working day and probably have one of the most diverse jobs within academia which suits me fine. I spend a lot of time helping colleagues and students make better use of technology to be more productive, communicate their research and work smarter on a modern digital campus. This can be everything from helping them understand altmetrics, to social media, to mobile apps to using video. I very much practice what I preach and use all of the technologies myself as part of my teaching and research. I try to find out the pros and cons to help others further down the way – I’m not afraid to fail so others don’t.

Where did you first learn of altmetrics?

It was possibly late 2011, but certainly early 2012 when I emailed this link to myself: – thank goodness for a well-ordered email file structure. I had been a big user of Mendeley since 2009 and by following them I was becoming aware of various things that were happening in the altmetrics space. I also spoke at the same parallel session as Mark Hahnel from Figshare at the defunct Online Information conference in London. At the time, I was not fully aware of what he and others were doing but made it my mission to find out more.

You’ve recently published a book, ‘Altmetrics-A practical guide for librarians, researchers and academics’. Can you tell us about the book and what inspired you to write on this subject?

I was approached by my publisher, Facet Books, in mid-2013 to write a book of my choosing and it was a simple case of deciding between two topics, those I was and am still most passionate about: open research and information overload. Both can feel like different sides of the same coin at times, but in the end I knew that information overload had been covered before very well, altmetrics in my mind needed that extra special attention. I realised that there were two issues that needed covering, the theory behind why altmetrics had come about and the practical ways academics and librarians could get behind it. With that in mind I approached some of the best experts I knew, William Gunn from Mendeley and Euan Adie from Altmetric to explain about altmetrics in the purest sense. Ben Showers, who at the time had was working for Jisc and had published on bibliometrics, and Andrew Booth and Claire Beecroft, who between them had years of experience in the library and information world, completed the title. I wrote just over half of the book and tried to capture why we had got to where we are and how best to navigate this brave new world. Hopefully readers have found it a useful addition to the growing literature.

How do you think Altmetric data can help researchers?

Quite simply, it offers them a feedback loop and window onto the world of how their research is being received globally. I think there is this misconception that most academics are not that interested in how their work is perceived. I think most are, they are just worried about negative feedback, criticism and miscommunication – but the world is changing, especially where impact is now so important. The idea that you can publish a piece of research and it just sits in a journal on a shelf has long gone, we have the potential to discover so much more with alternative metrics. Of course the balance is maintaining that high quality output and not being drawn into a world of just gawping at numbers – but that is the misconception for me. Altmetrics is about finding out much more than numbers; it’s also about more than just your papers, but also your data, books, posters among other outputs. It feels like we should have done this all along when the Web began.

How do you think altmetrics can help lecturers/professors?

I think altmetrics offers another dimension to their hard work, one that with very little effort can be explored. As I’ve written before, the Altmetric score is at its most interesting when it is zero. At that point, we have some idea that a piece of research is not being communicated or discussed–it may not be the absolute case, but it is a good indicator. For any scholars wanting to open up that dialogue, seeing a zero score can get them thinking about the scholarly communication process and what they can do, or who can help them facilitate that. I think altmetrics are part of a bigger picture that includes traditional metrics, impact, digital technologies and social media. They are part of an emerging ecosystem, we just need to take care that those who want to engage with it are given the right amount of support – I’m doing my best at my end. 

Wednesday, 10 August 2016

Mendeley Masterclasses - Videos to help you master your references

Andy Tattersall
Andy Tattersall has created a new series of short instructional videos to help you make the most out of using Mendeley. The series of 15 videos are called 'Mendeley Masterclass' and help users find their way through the various web, desktop and mobile versions. More videos will be added a later date but you can see for yourself a couple below. Mendeley is a very worthwhile and simple tool to use and really helps academics and students save time and be organised whilst studying and conducting research. The videos are only a few minutes long, so with a cup of tea and half an hour you could master the software easily. The videos are on YoyTube and will be added to the University of Sheffield iTunes U collection later this year.

The Mendeley Masterclass playlist