Science metrics, LitRoost, and the networked era

Engraving after 'Men of Science Living in 1807-8' by George Zobel, and  William Walker

Engraving after ‘Men of Science Living in 1807-8’ by George Zobel and William Walker (1862)

Too many PhDs? Or not enough?

In the United States, it is estimated that there are approximately 21 million individuals with a Master’s degree or higher educational attainment. In many English-speaking countries, debate continues as to whether universities are producing “too many PhDs”. This debate is often framed around the question of whether academia has enough space or resources to accommodate all those doctoral graduates. Framed this way, the answer is obvious to anyone who has spent time pursuing an academic position. There are reportedly hundreds of highly-qualified applicants for every tenure-track position and the expectations of candidates keeps climbing.

But what if we framed the question differently?

What if we asked, instead:

  • Are there enough highly-qualified individuals to tackle the world’s greatest challenges, such as poverty, disease, energy, and climate change?
  • Can the pace of science be accelerated so that we can avert catastrophes such as peak energy, runaway climate disruption, or economic collapse?
  • Can we connect scientists, experts, policy-makers, and industry leaders with each other, and incentivize collaboration towards solving global problems?

The trick to solving a hard problem is often a matter of asking the right questions. How science is done is changing. How scientists connect and interact is changing. And the change will be good for science, for the academy, and for scientists as well.

How the internet is changing science

The new economy is a knowledge economy and networked economy. So where do science and scientists fit in? And how is the internet changing the way science is done?

In a Wired article titled “The Internet is Changing the Scientific Method”, Alexis Madrigal comments on a recent editorial in the journal Science by Ben Schneiderman. Madrigal summarizes Schneiderman’s argument, saying that

[…] studying the interactions between people will be more important than studying the interactions between particles in bringing scientific solutions to big problems like disaster response, health care and energy sustainability.

I believe that science in the 21st century will be characterized by how scientists connect with each other through the internet to tackle important problems. So how do you connect with the right people and make sure that your work has impact?

Alchemist Sendivogius, by Jan Matejko.

Alchemist Sendivogius, by Jan Matejko.

I want to write about an Open Science startup called LitRoost that’s trying to make a difference, but first…

A little history

Scientific journals were born in the 17th century. It was also in this century that several European countries established their first scientific academies. Before this time, there were few career scientists. If you were a scientist, you were more likely to be an inventor, hoping to profit from your invention, perhaps by discovering the alchemical recipe for transmuting lead into gold.

The invention of scientific journals was a revolution in open science. No longer did people have to use cryptographic cyphers to encode their knowledge (as the likes of Newton and Galileo did) so as to establish primacy if a rival claimed the same discovery. Newton and Leibniz famously feuded over who had first discovered the calculus.

Eventually, governments realized that science could benefit society and began subsidizing scientific research. In an age of nationalism, this was also a way for one government to technologically overtake another.

Fast forward to the modern day and national borders no longer are barriers to international collaboration, and research teams often span dozens of scientists from multiple continents.

How do you measure the output of a single scientist?

Several systems have been devised to try to quantify the output of a single scientist. Unfortunately, it is still true that the only real, portable wealth that scientists have are their published papers. Your contributions to outreach, teaching, side-projects, websites, new tools—none are as important as getting papers published. I hope that this will change during my lifetime, but scientists can be quite conservative.

In March 2011, I attended a TEDx conference in Waterloo, Ontario, where Michael Nielsen was giving a talk. Nielsen is a big proponent of Open Science and his essay, The Future of Science, is one of the clearest explanations of the problem. He greatly expanded on that essay in a book titled “Reinventing Discovery: The New Era of Networked Science“.

If you are a young, ambitious scientist, you are rewarded for writing papers, and little else. You are tracked by your citations. Some product of the number of papers and the number of citations determines your overall impact. Google is trying to track the impact of individual scholars (as an example, here’s my humble profile). Metrics such as citation count or the h-index can take a long time to accumulate. The only way to recognize what brand-new papers have high impact is to hear what other scientists are talking about.

Can we invent better metrics?

In a 2010 paper published in the journal Scientometrics (yes, that’s a thing), the authors noted that the Science Citation Index (SCI), which tracks citations across a swath of journals, was continuing to grow at a steady rate, but new publication channels including open access archives, conference proceedings, and web publications were growing very rapidly and were not being indexed, challenging the traditional scientific productivity metrics.

Suppose I publish one academic paper per year. But besides this, I write about my science in a blog. I also try to get attention for my work using social media. I connect to other scientists through social networks such as Mendeley. Perhaps I collaborate with another scientist to create a web tool to make collaboration easier. Maybe a popular science blog decides to write about something I’ve done. How productive am I?

[column size=”1-2″] Science 1.0

You have published 1 new paper. Now wait a few years to see if anyone cites it. In 10 years you’ll know if it had high impact.

[column size=”1-2″ last=”1″] Science 2.0

You are a highly-engaged scientist with 1 new publication that has already garnered significant traction through various outlets. People are talking about your work, which suggests that it’s high-impact work.

 An emergent alternative to traditional citations as an impact measure is altmetrics. By combining information about how often an article is downloaded, shared, blogged, cited, bookmarked, linked to, or tweeted, altmetrics seek to get a larger sense of the article’s impact

Can we build better science platforms?

Nielsen argued that to create an open scientific culture, we needed to

(1) build superb online tools; and (2) cause the cultural changes necessary for those tools to be accepted.

Online tools are starting to come of age. It’s a difficult scene to work in because Open Science projects have repeatedly cropped up and faded away because it’s difficult to get a critical mass of people to engage. Dedicated wikis for special area knowledge rarely gain any traction because scientists would rather spend their time working on their next publication, even as they recognize that new online tools would accelerate progress for the entire field.


LitRoost is a new open science platform trying to carve out a space in this area. I spoke to the founder, Clayton Bingham, last week to understand how LitRoost works and how it tries to differentiate itself.

The guys at LitRoost understand that the future of science is Open Access. Funding agencies in the UK and USA are pushing to make all publicly-funded research available online for the public to read. Open Access makes possible the application of data mining tools to scientific research results.

The idea of LitRoost is that each user can create a “roost” of scientific literature in the area they work in. The web application helps with the curation process, and the combination of traditional metrics and altmetrics will help the user access high-impact science as it happens. The individual scientists do not need to invest in “building” a open science platform themselves (like a blog or a wiki), but  instead are connected to a web of emerging research results.

Every scientist knows that the conversations they have during the coffee breaks at a conference are at least as important as the other aspects of the conference. This goes back to Schneiderman’s point that studying the interactions between scientists will be as important as the science itself. Connecting the right people in the most efficient way will lead to more expedient solutions to big problems.

ARPANET Logical Map, 1977

ARPANET Logical Map, 1977. Ironically, scientists invented the internet. Now the internet is changing science.

Currently, success is in this arena is rare (the Polymath Project is a notable exception), but the right tools are emerging and the culture towards using them is shifting.

When Mendeley first emerged, many thought that it wouldn’t be successful and wouldn’t be profitable. Mendeley now indexes million of articles and was recently purchased by publishing giant Elsevier for as much as $100 million. What the acquisition will mean for the Open Science movement remains to be seen.

The young Open Science contender LitRoost is currently in open beta and looking for testers. They are currently focusing on life and health sciences, where the need for better Open Science tools is great. Physicists like myself have enjoyed using arXiv for a long time, but we still need better ways of wading through the vast literature.

Part of changing the culture of science will happen at the grass-roots level, with individual scientist-bloggers, programmers, enthusiastic tweeters, and early-adopters.

Ironically, scientists created the internet. Now the internet is changing science.

Now it’s your turn.

If you care about Open Science, give LitRoost a try, or share this article, or leave a comment below about how you’re participating in the Open Science movement.



  • Scott Chamberlain

    I think there are far too many PhDs. I got one myself, and am getting out of academia. In my field – ecology – the best options are academia and govt jobs, which are all losing money quickly. There are no industry jobs in ecology – but fields like engineering have that option. Isn’t the fact that grad students and postdocs are payed so little a sign that there are too many of them? Wouldn’t fewer grad students/postdocs = better pay?

  • Pingback: Science 2.0 – Whither our upgrade? | ISSST()

  • Pingback: Science metrics, LitRoost, and the networked era | syndax vuzz()