Wednesday, 12 October 2016
A close colleague and friend of Information Resources, Dr Chris Carroll has just published a paper on the limitations of citations, in particular the single counting of them within a paper's reference list. Chris wrote about his paper in the excellent LSE Impact Blog and we thought it would be of interest to visitors of the Information Resources Blog. We have republished it under their Creative Commons licence. Dr Carroll is a member of HEDS and worked previously as an information specialist as part of our team.
The limitations of simple ‘citation count’ figures are well-known. Chris Carrollargues that the impact of an academic research paper might be better measured by counting the number of times it is cited within citing publications rather than by simply measuring if it has been cited or not. Three or more citations of the key paper arguably represent a rather different level of impact than a single citation. By looking for this easily generated number, every researcher can quickly gain a greater insight into the impact (or not) of their published work.
The academic research and policy agenda increasingly seeks to measure and use ‘impact’ as a means of determining the value of different items of published research. However, there is much debate about how best to define and quantify impact, and any assessment must take into account influence beyond the limited bounds of academia, in areas such as public policy. However, within academia, it is generally accepted that the number of times a paper is cited, the so-called ‘citation count’ or ‘citation score’, offers the most easily measured guide to its impact. The underlying assumption is that the cited work has influenced the citing work in some way, but this metric is also viewed as practically and conceptually limited. The criticism is loud and frequent: such citation scores do not tell us how a piece of research has actually been used in practice, only that it is known and cited.
Is there a better solution?
So, the obvious answer is to see if there is an easy way to measure how a paper has been used by its citing publications, rather than simply recording whether it is cited (the standard ‘citation score’). I did this by using one of my own frequently-cited papers as a case study and applying a basic metric: impact of the case study paper was considered to be ‘high’ if it was cited three or more times within the citing publication; ‘moderate’ if cited twice; and ‘low’ if cited only once.
The case study paper had 393 unique citations by November 2015: 59% (230/393) of publications cited the paper only once, suggesting its impact on these publications was low; 17% (65/393) cited it twice, suggesting moderate impact; but 25% (98/393) cited it three or more times, suggesting that this paper was having a genuine influence on those studies. The citation frequency within this ‘high impact’ group of publications ranged from three to as many as 14 in peer-reviewed studies and 17 in academic dissertations (with their longer word count). Primary research studies published in peer-reviewed journals were the principal publication type across all levels of impact.
I also noted where these single or multiple citations appeared within these publications. Single citations tended to appear only in the introduction or background sections of papers. These instances appeared to be simple citations ‘in passing’, the necessary ‘nod’ to existing literature at the start of a publication. However, when there were three or more citations, they tended to appear across two or more sections of a paper, especially in the methods and discussions, which suggests some real influence on the justification or design of a study, or the interpretation of its results. These findings on the location of the citations confirmed that the number of times a paper was cited within a publication really was a good indicator of that paper’s impact.
Pros and cons
A metric based on within-publication citation frequency is unambiguous and thus capable of accuracy. The assessment of where the citations appear in the citing publications also provided contextual information on the nature of the citation. In this case, it confirmed the viability of within-publication citation frequency as an impact metric. The approach is transparent and data can be verified and updated. The proposed metric, the ‘citation profile’, is not a measure of research quality, nor does it seek to measure other forms of impact or to address issues such as negative findings producing relatively lower citation scores. It merely seeks to contribute to the debate about how citation metrics might be used to give a more contextually robust picture of a paper’s academic impact.
Of course, even this ‘citation profile’ is not without its problems, but it is arguably less of a“damned lie” or “statistic” than some other metrics: only a more in-depth analysis of each citing publication can more accurately gauge a paper’s influence on another piece of research or a policy document. Yes, there are also questions concerning this metric’s generalisability: as with other bibliometrics, publications in different disciplines are also likely to have different citation profiles. However, these issues can be easily addressed by future research, given the simplicity of the metric.
If academics or funders want to improve their understanding of how research is used, then this easy-to-generate metric can add depth and value to the basic, much-maligned, and ‘damned’ ‘citation score’.
This blog post is based on the author’s article, ‘Measuring academic research impact: creating a citation profile using the conceptual framework for implementation fidelity as a case study’, published in Scientometrics (DOI: 10.1007/s11192-016-2085-0).
Note: This article gives the views of the author(s), and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the author
Chris Carroll is a Reader in Systematic Review and Evidence Synthesis at the University of Sheffield. His role principally involves synthesising published data to help inform policymaking, particularly in the fields of medicine and health, as well as the development of methods to conduct such work.
Tuesday, 4 October 2016
Inspiration for Life is the charity set up to honour and celebrate the memory of Tim Richardson, a University of Sheffield colleague, who passed away on 5 February 2013 from cancer. More recently, following the passing of another colleague, Victoria Henshaw, also from cancer, the charity has also dedicated itself to supporting the memory and work of both these special people. They aim to promote lifelong learning, and encourage the public understanding of science through publications, lectures and other events. They also directly support local cancer charities.
I have done a fair few races over the years, though nothing beyond 5k for a while, and you can hunt me down at Sheffield Hallam Parkrun most Saturday mornings- here's me at last week's (just to prove I CAN run):
You can sponsor the team I'm running for at: https://mydonate.bt.com/events/timsteam2016
Posted by Claire.
Wednesday, 21 September 2016
This year's Multimedia and Information Technology Conference (MmIT) focused on “Digital Citizenship: What is the library's role?” and included a fabulous range of talks from librarians, head of services, computer specialists and suppliers which really reflects the scope and depth of the topic. As a relatively new professional, one who had never attended a conference before, I decided to join in mainly because I’d heard good things about the food. However I am always keen to broaden my horizons and I’m pleased to say not only did the food exceed my expectations, so did the conference itself. MmIT are a special interest group within Cilip.
Dr Chris Stokes (Joint Director of Digital Learning, University of Sheffield) spoke about his team’s process to make the University of Sheffield’s first Massive Open Online Course (MOOC). This was inspired by an outreach scheme designed to give 16 to 18 year olds the knowledge, information and guidance to make a competitive application to the University of Sheffield’s Dental School. This digital course aimed to use technology to increase access to a course that promoted inclusion within higher education and unexpectedly inspired an online community that facilitated communication between a range of user groups. This group ranged from aspirational A Level students to individuals with fear of dentists and even dental nurses refreshing their knowledge before returning to work after maternity leave.
What I learnt from Chris: MOOCs aren’t just for graduates, or a specific age range - they can be all inclusive and empowering if you design interesting, interactive content which simplifies the subject.
This issue of safe and knowledgeable digital access for the masses was something which inspired Lee Fallin and Mike Ewan to create a website called “The Digital Student”. This project aimed to educate students at the University of Hull by creating a mobile optimised website applicable to all students which could also be embedded within the VLE (Virtual Learning Environment). This idea not only gives students convenient access to information from a point on the university system which they already utilise but also standardised, quality information which can be updated centrally and easily replicated over a large number of users quickly. Further to this, the site is also mobile enabled and also designed to appeal to a range of different abilities using a content layout inspired by the popular website, Buzzfeed. After creating this content to improve the digital literacy of their students, the project will next release guides which aim to help students sell the digital skills when applying for jobs.
What I learnt from Lee and Mike: Take content to channels students are already using and display it in a way they are familiar with, in a format they like.
The socio-economic benefits of digital literacy was the focus of Helen Milner CEO of the Tinder Foundation. Her keynote speech, based on digital social inclusion, explained work of the Tinder Foundation which aims to connect and empower individuals. This is as a result of an aim to raise not only digital literacy locally, but also for the awareness of the need and benefits of digital literacy and digital access on a national scale.
What I learn from Helen: In 2015, a shockingly large amount (12.6 million) of the population were still offline. What was even more shocking was finding out was that if you were uneducated, retirement age, disabled or had a low income you were less likely to have access to the internet despite the potential benefits including being able to apply for jobs (25% of which are online-only applications), making online savings as well as maintaining social connections.
Check out this brilliant infographic that captures the data around digital literacy and accessibility
Although there are methods to avoiding online observation, such as encryption technologies and browsers such as Tor, these have received a negative spin as a result of illegal activities linked to the Dark Web and the popular attitude, “you have nothing to worry about, if you have nothing to hide”. These tools, while increasing security for individuals who want to protect their communications, are also complex to execute.
What I learnt from Ian: Individuals who thought they were under surveillance changed their information searching behaviours and automatically censored themselves. This arguably reduces freedom of thought which is indusive to critical thinking, idea generation and the democratic process.
Check out Infoism for more information
|Dr Kevin Curran|
Another highly informative talk was hosted by Kevin Curran, who is a Reader in Computer Science at Ulster University. His overview, “Hacking: Child’s Play” highlighted just how hackers can locate unsecured databases and webcams by using Google dorks, hold a company to ransom with a simple denial of service attack program and receive money from illegal activity anonymously using a cryptocurrency such as Bitcoin.
What I learnt from Kevin: Great tips to improve your security online.
- Check if your details have been breached. You can also sign up to the “Notify me” service which will tell you when you need to change your password so no unauthorised snoopers can access your accounts https://haveibeenpwned.com/
- Don’t invest in expensive anti-virus. Windows Defence is freely available available from Microsoft and will do just as good a job as Norton and other popular paid-for antivirus softwares.
- Don’t use the same password for every website. You can use a password manager like LastPass (link https://lastpass.com/) to simply this for you
All of the presentations I attended were thoughtful and particularly made me reflect on the idea of using the benefits of technology to empower - Chris and his team's outreach MOOC gave disadvantaged students the chance to participate in a highly competitive career path. Additionally, Helen and the Tinder Foundation, as well as Lee and Mike from University of Hull, had recognised the need for further digital literacy training and the benefits of enabling individuals to use technology, in terms of both increasing employability and confidence.
|Digital Privacy & Digital Citizenship|
The knowledge brought to the conference by Ian of Infoism and Kevin of Ulster University did however highlight some of the areas in which there is much at stake if individuals and organisations use the technology in an irresponsible way. The main thing I’ve taken away from the conference is the idea that using digital technology is like driving a car - it doesn’t matter how bright, young or reactive you are - if you aren’t taught how how to operate the machine and navigate the system there’s the potential for a crash.
“The Digital Student” website
Tuesday, 20 September 2016
Transforming your service: the right evidence at the right time and place: Virtual Issue of HILJ edited by Anthea Sutton
This post originally appeared on the Knowledge for Healthcare blog and has been reproduced with permission.
As this is the beginning of the academic year, I get to meet our new Master’s students to chat to them about their module choices this year. This is part of my role as co-ordinator of a public health informatics module. Students can study the module as a CPD opportunity or as part of one of our Master’s programmes.
Informatics brings together two underlying disciplines: computer science and information science. Public health informatics is about applying an informatics solution to public health problems. It offers additional approaches to disease surveillance and facilitates service planning and public health research. In the module we take an overview approach and look at various case studies on how informatics can be applied, such as in health promotion activities or in a disaster response situation. We also focus on how to evaluate an information system by identifying and applying an evaluation framework.
Last year we created a new session on ethics in public health informatics looking at ethical theories and how they are enacted in practice. Ethics should be a live issue, and we considered the limitations and usefulness of ethical codes by discussing key papers in a journal club format.
So I’m looking forward to working with our new students and the fabulous public health informatics team for the new academic year. For more information on the module which we provide in an online and on-campus format please see the relevant module pages or contact me at firstname.lastname@example.org
Picture credit: activity tracker by Zed Huang via Flickr (CC-BY-NC-SA 2.0)
Wednesday, 7 September 2016
On the back of his edited book - Altmetrics, A Practical Guide for Librarians, Researchers and Academics - Andy Tattersall has been picked for Altmetric.com's Ambassador of the Month. He was interviewed by Almetric.com about his role and work with altmetrics.
The interview is below
The interview is below
Tell me about your current work at The University of Sheffield. What does a typical day involve for you?
Where did you first learn of altmetrics?
You’ve recently published a book, ‘Altmetrics-A practical guide for librarians, researchers and academics’. Can you tell us about the book and what inspired you to write on this subject?
How do you think Altmetric data can help researchers?
How do you think altmetrics can help lecturers/professors?
Wednesday, 10 August 2016
The Mendeley Masterclass playlist