The Mystical Path of Scientific Understanding

Reading and understanding the latest papers is a crucial part of being a scientist, but it's not something that we're ever taught to do, explicitly, as part of a scientific education. You take classes on genetics, and then maybe you become a grad student and you start doing genetics research: but there are no classes on reading genetics papers. It's something you pick up as you go along, if you're lucky.

But reading a paper isn't one single skill: as you learn more about a particular field your understanding of the published literature tends to progress through certain stages. At least, this is my experience. Like all such "stage models" what follows is a simplification, but it's something I think I'd have found useful to have been told when I was starting out.

Stage 0 : Huh?

You don't even understand what the paper is about. If I were to somehow find myself reading a paper on quantum chromodynamics, I would have no idea what it was trying to say, let alone whether it was right.

How to tell if you're at this level: The title has you stumped.

Next comes the most dangerous stage:

Stage 1 : Oooh!

You understand the paper's conclusions, but that's it: you don't get how the authors arrived at them, or how they relate to anything else. My understanding of chemistry is at this level: if someone claims to have found a new way of synthesizing some molecule, I know what that means, but I have to take the result or leave it: I can't criticize it, and in order to know how important the result is and what the implications are, I only have the author's word.

How to tell if you're here: you struggle with the Methods and the Results; you rely on the author's summary of their findings in the Abstract or the Discussion. The Introduction is all new to you.
How to get here: read a textbook until you grasp the basics of the field.

This is dangerous, because a paper could be completely wrong, and you wouldn't know - yet you know enough to be mislead by it, and to think you understand it. Incidentally, this is the stage inhabited by most journalists and politicians

These next two stages don't really come in any particular order. 2a does not necessarily precede 2b (it's just the one I chose to write about first.)

Stage 2a : Hmmm.

You understand the paper's conclusions and its methods, so you're able to judge how strong the argument is. If I were reading about a new cancer drug, and learned that had passed a large randomized controlled clinical trial, I'd be fairly confident that it works. Whereas, if I read that it had been "tested" in one patient (a case study), I'd be skeptical. I don't know anything about cancer drugs but I do know about clinical trials.

How to tell if you're here: you're comfortable reading the Methods and the Results.
How to get here: Read the Methods sections of papers in the field. If you don't understand the terminology, find a textbook or a review paper dealing with methods.

Stage 2b : Oh, interesting...

You understand why the authors decided to research this stuff, because you understand the specialist background literature about this sub-topic. You can judge important the research is and what the implications of it are. Note that the Introduction and the Discussion are meant to serve to explain all this context for the benefit of people who don't have this level of understanding of the topic, but in fact they're often either poorly written or actively misleading, so you can't rely on them.

How to tell if you're here: you find yourself either agreeing with, or criticizing, the Introduction and the Discussion.
How to get here: read recent review papers about the field. Textbooks are unlikely to be up-to-date enough, or detailed enough, to be of much use. Just remember that every review paper offers a different slant so make sure you don't just read one and take it as gospel.

Finally, we come to the highest stage, the moment of Enlightenment, saroti, Nirvana...

Stage 3 : Aha!

This is what happens when you have both of the previous kinds of understanding - you see what the authors did and why they did it. This is more than the sum of its parts, because it allows you to evaluate whether they chose the most appropriate way of answering the questions they set out to investigate. You can think up a better way of doing it, or design interesting follow-up work.

This is the stage at which you stop seeing papers as communications from a mysterious other world, and see them as something written by people much like yourself - which, or course, is what they are.

For example, if I were to read a paper using fMRI to study whether a new antidepressant raises dopamine levels in the brain, I'd be able to say that while that's an excellent question, fMRI is unable to show this directly, whereas PET can, so it's probably a better option; but they probably chose to use fMRI because it's a lot cheaper than PET and much less hassle.

How to tell if you're here: you pretty much know what the full paper is going to be like from the Abstract alone; you probably don't bother to read the whole thing.
How to get here: Get to 2a and 2b.