Wednesday, 11 April 2012

Psychology vs Astrology

Are personality tests any more accurate than astrology?

A lovely study I just came across examined this question: Science Versus the Stars. The researchers took 52 college students and got them to complete a standard NEO personality questionnaire. They also had to state the date, time and place of their birth.

Three weeks later, the participants were then given two personality summaries - one based on the personality tests, and one on their astrological chart generated with a computer program.

The trick was that everyone also got a pair of bogus summaries, one of each kind. These were simply someone else's results, picked at random from the other 51 volunteers. They weren't told which were the fakes and which were real - they had to work it out, based on which one matched them best.

The results showed that the subjects were no better than guessing when trying to tell which of the two astrology charts was theirs. They were able to pick their own personality scores better than chance, although only 80% of them got it right, and guesswork gets you to 50% - so this is not all that impressive. Psychology beat astrology, but hardly by a landslide.

This study is a modern update of Shawn Carlson's classic 1985 Nature paper, A double-blind test of astrology. In Carlson's experiment, though, people weren't even able to accurately pick out their own personality scores.

When asked to say which of the four reports was the best match overall match to their personality, 55% of the participants picked their own real personality one - but no fewer than 35% preferred one of the astrology charts, and 10% went for someone else's personality scores. Hmm.

The authors say
the present results represent less of an endorsement of psychological measures than a further indictment of astrology.
but I think it's interesting that even under very favorable conditions (only one fake personality test), people were well short of perfect accuracy at spotting their own psychological scores - which they had themselves produced by filling out a questionnaire, just weeks before. Whether that tells us more about the NEO test, the participants' memory, or the fact that all the students at Conneticut College are pretty much the same, I'll leave it for you to judge...

ResearchBlogging.orgWyman, A., and Vyse, S. (2008). Science Versus the Stars: A Double-Blind Test of the Validity of the NEO Five-Factor Inventory and Computer-Generated Astrological Natal Charts The Journal of General Psychology, 135 (3), 287-300 DOI: 10.3200/GENP.135.3.287-300

20 comments:

Gustavo said...

I think conceptually there's a difference (and often a very real one) between what a personality tests says (based on certain, specific answers to certain very specific questions) and what people think about how they are. Even if you ask someone about how they think they are right after just a single personality test, you will find answers not matching. People have really tough difficulties sometimes at looking at themselves (subjectivity, defense mechanisms, lack of abstraction capabilities, confusion between reality ands desired reality)...

lastpook said...

I agree with Gustavo. Very often people are biased to assign themselves features that they'd like to possess rather than that they actually have.

Neuroskeptic said...

I agree, I just think this is a striking demonstration of that effect i.e. the fact that a good proportion of people didn't endorse the results of an entirely self-report questionnaire.

I guess it shows the limits of self-report, really. Because this is pretty much a case of people contradicting their own reports.

Neuroskeptic said...

Also, the authors did try to look to see if people were more likely to endorse the "good" astrology signs vs the "bad" ones (on the theory that the odd-numbered signs are considered better than the even ones).

They weren't.

But then all the signs are generally pretty good so I'm not sure that tells us much.

Anonymous said...

Multiple problems with that study, I think. About the computer program (http://alabe.com/) used for the study : there are much better ones out there, with the help of which, maybe, more students would have recognize themselves... Yet, with the personality tests as with astrological readings, it's also a question of the extend to which people actually know themselves. Even more, young people (college students) are still creating the basis of what will become their identity...
What if they did the study again, with a good astrological program, and older people?

Unknown said...

I'm confused by the reactions here. As I understand it, the subjects filled out a psychological questionairre which presumably had relatively concrete questions like "rate on a scale of 1-5: I am often anxious in social gatherings" (or whatever) and then later asked to identify a set of scalar scores for each of the Big 5 traits. A lay subject might not do much better than 80% at guessing which of the traits were being distinguished by each question, let alone which of two sets of scores -- resulting from a *calculation* based on their previous answers -- was their own. Also: given a random distribution of answers/scores, some subjects would be expected to be handed two pretty *similar* results which could each plausibly be their own, even if they understood the whole process as well as the researchers.

Peter Hildebrand said...

@Neuroskeptic; "I guess it shows the limits of self-report, really. Because this is pretty much a case of people contradicting their own reports."

I'd like to suggest that this study isn't an indictment of personality tests, per se. We use tests over less formal self reports to more rigorously study personality. It's totally possible for a test taker to answer the questions completely honestly, and still be shocked by the (correct) results.

Now, debating the accuracy of these tests is a whole other discussion. But if we accept that they're sufficiently accurate, I'd tend to believe the test results, even in light of protests from the test takers.

Sanjay Srivastava said...

This is a "test of validity" only if you have a theoretical prediction that the Big Five structure exists in people's heads. Which would be a poor hypothesis, because there is no evidence that the within-person organization of the self-concept mirrors the Big Five structure. Yes, single NEO items are self-reports; but the scoring of the NEO combines items and assigns meaning to them in ways that are not intuitive or obvious. The fact that subjects' guesses were greater than chance and less than perfect is entirely consistent and unsurprising.

Anonymous said...

Is the NEO questionnaire forced-choice? A person forced to choose a mostly inaccurate option over a completely inaccurate option would still find the results inaccurate.

Anonymous said...

Don't be simple-minded in you conclusion, folks. All this means is that a lot of people lack insight into their own personality traits. Surprised? There are numerous studies pointing to the common gap between one's identity and one's reputation. Think you're highly conscientious? Ask people who know you, they may beg to differ... The more interesting study would be to ask friends and family of the subjects to match the subjects to the respective NEO profile - I'm sure they would be far more accurate...

petrossa said...

For the objective observer, those who are not boxed in the psychology framework as taught in educational institutions, the difference between psychology and astrology is not that big.

Both depend on abstract theorems encompassing a subjective observational based underpinning.

Both assume observed patterns of behavior can be described by uniform units. Be it people or stars.

Both are self confirmational. Both depend on a structure of concepts tested against themselves.

Seek and ye shall find.

Neuroskeptic said...

Thanks for the comments.

One thing that hasn't been mentioned yet in the discussion is that people might have struggled with matching their own personality, even if it was 100% reliable and they had perfect insight, if the randomly picked personality results were very similar.

Maybe the 20% who couldn't distinguish their results from someone else's, were just unlucky to have got matched to someone a lot like them.

It's not hard to see how students at the same college might tend to be pretty similar... kind of what I was getting at in the last line of the post.

In which case you'd expect that if you did this in a more diverse sample, accuracy would increase.

J Tan said...

@Neuroskeptic

That is a very good point, I think small liberal arts colleges (from experience) especially have a self-selecting population, relative to the world at large. It would be interesting to look at this at a bigger university, or maybe a community sample rather than college students, which would also answer the questions about continuing identity development at that age.

Ivana Fulli MD said...

With French female college students and many a male I will suppose a very clear knowledge of one's "astrological personality" indeed since astrology is very popular and so many people read the astrologers working for many popular newspapers like the free Metro and female magazines.

In France educated people will tell you things like :"I am impulsive because I was born on the 19th of April " and they know by heart what are their personality traits courtesy of the stars.

Idem for some vietnameses persons of my knowledge with the oriental astrology.

omg said...

I don't get it. I read the horoscope on the train daily. Not so much for me but for everyone else in my life because it's written in the stars.

Ivana Fulli MD said...

omg,

The question- to my mind- is : Will you be able to choose your "right" astrological summary if you were given the choice between it and somebody's else astrological summary?

Agreeing with one's (or one's significant people) section of the "astrological news and advice "of the day is a trusting (and honorable by all means) action of the mind.

Logically it should make people able to pick up the right summary if their trust is legitimate-that is if we really get a personnality according to our date and hour of birth(and year for the chinese horoscopes.

I find that subject fascinating since I studied hard homeopathy in the 90s:

Seriously trained homeopaths of at least average intelligence are able to discuss for fun themselves and significant others and offspring and whatever like novel characters and famous political or historical people with their homeopathic personality categories.

Just like psychologists or psychiatrists will use their big 5 or whatever.

But this is not to make you loose your peaceful moment reading the horoscope every day omg: all strong women know better than to let the psychologists interfere with their little everyday pleasures.
bitterness of life

omg said...

Time came from the stars. Stellar configurations fascinate me. We're made of star stuff. Birth and death we go back up there. When Jesus was born there was a bright star to guide the wise men who brought gifts for his buriel. Fate was written in the stars. I don't believe in zodiacs but it's a daily reminder of destiny and prophecy, and as human beings we really don't know anything about ourselves.

Ivana Fulli MD said...

Thanks omg.

DavidB said...

Worth reading the report before posting, despite a number of methodological shortcomings the authors still post this at the end of the research.

"Furthermore, in the present study, the majority of participants were able to recognize their NEO-FFI profiles"

Neuroskeptic said...

DavidB: Right. I have to say actually that a number of people linking to this post are interpreting it as more anti-NEOFFI than it is.