The Trouble With Exams

So the five month old DU Economics post had this reasonably long comments thread which instead of discussing DU or economics itself ended up discussing the stud-fighter framework, the best sort of examination system, what it is that an examination system should actually be examining, and the difference between CBSE Boards and the JEE. I promised a followup post by Sankranti, and then promptly forgot all about it and started blogging about other things like superhero underwear. However, I do get down to things… eventually, so here’s the post. Though by delaying it as much as this, I can now justifiably claim that it’s become topical.

Right. So. Question: what purpose do exams actually serve? I can think of:

  1. Establishing a minimum level of competence or knowledge or providing a pass-mark. Examples of this are getting 33% in the CBSE Boards, or 80% in my employer’s HIV-AIDS e-learning module, or 100% in my employer’s money laundering e-learning module (we take money laundering very seriously). Also, CA certification.
  2. Creating a filter so that you can select the top – the JEE works this way
  3. As a stage-gate between two-levels of education or employment, or between education and employment – in combination with #1 or #2 – the SAT/ GRE work a bit like this – being at the top isn’t essential because there are enough available seats to ensure that you can get in a long way down.
  4. To determine how much a student has learnt independent of formally certifying competence or university admission or things like that. For example, the practice tests a tutor or a coaching centre gives so that students can know how far ahead or far behind they are. Or the sample papers which a student practices ahead of the boards.
  5. To determine how successful the teacher or the examination system has been in passing on concepts. For example, the Azim Premji Foundation conducts its own tests at Class 5 and Class 8 (I think) for students in Chambal valley and North Karnataka rural schools to measure how well they are performing and which ones deserve aid.

Which is a fabulous framework if you stick to it. In practice, though, here’s the problem in real life. Where the CBSE is concerned, the board exam is being used for purpose #1: certifying minimum competence. Delhi University, on the other hand, uses it for purpose #2: as a filter to select top-performers, which is not what the boards are designed for. The boards are supposed to be high-scoring, easily crackable if you study the entire year, and use questions with standard answers that can be checked against a template to simplify the lives of the unfortunate examiner who has to check hundreds of answer sheets. Is it any surprise that college cutoffs start hitting 97% levels?

(An associated hoopla about the boards is that everyone taking them (and their parents, and their teachers) go up in arms every time a paper is difficult. All things being equal, it doesn’t matter, because a bad paper is just going to push the cutoff down – it’s still going to be the top five hundred people who convert the top five hundred seats. To be fair, all things aren’t equal, because students can have different subject combinations. So if the mathematics exam is particularly tough, someone who’s taken psychology or physical education instead gets an advantage.)

There’s another major problem with using the boards as an entrance / selection exam. To be fair, this is a problem with all entrance exams in India. It’s a one-shot exam which happens only once a year. It doesn’t measure how you perform in the classroom, and your long-term ability to learn, which is something that the teacher on the ground is much better placed to judge. And if you mess up a one-shot examination once, you don’t get a chance to make up for it until the next year. And by mess up I don’t only mean fail, I also mean get any score low enough to prevent you from getting the course/ college you want.

So basically, college admissions boil down to this: you get one opportunity once a year to display your excellence through an exam which is designed not to test expertise but minimum competence. As processes go, this is so thoroughly broken that re-engineering consultants would throw up their hands in despair and suggest restarting from scratch.

Not only is the process broken, it’s broken in a way that disproportionately hurts the poor. Why? Well, because:

  1. In the situation described above – you mess up your exams and need to wait another year, the opportunity cost of sitting out another year is much lower. If you’re rich, or even middle class, your family has enough savings (or enough of a standard of living to cut back a little) to not worry about starting your career a year later.
  2. If you have a one-year run up to the exams, that gives rich kids a year in which to hire tutors to help them prepare for the exams. This becomes even worse when the CBSE makes papers ‘easy’, because when papers are well-designed and difficult, topping them is truly a function of how smart you are. When they’re easy, students who have time to spare and money for tutors crack the scene. When they’re difficult because they’re badly designed, the top ranks become a lottery.

Which is why the new CBSE exam design scheme, called HOTS (Higher Order Thinking Skills) may turn out to be one of the more sensible things the CBSE has ever done:

HOTS was the new basis of the question papers in the Class 10 exams this year. It is an analytical problem solving process, geared to assess the students’ absorption of knowledge and its application.

“The average performance and pass percentage has increased across the board but the number of perfect scorers has gone down. The new question pattern to judge students’ knowledge base could be the reason,” said Ganguly.

There is a substantial decline in the number of candidates who have bagged the ‘perfect score’ in Mathematics. “As against 5,251 students last year, this year just 2,647 students have scored 100 out of 100 in Mathematics.”

Similarly, in Social Science, only 598 students have scored a full 100 as against 1,233 last year.

(Delhi Live)

In fact, HOTS may have resulted in better government school performance too, which sort of supports my point about conceptual examinations helping the poor:

That the results in government schools have improved is evident: it shows the largest increase in pass percentage, climbing from 77.12 per cent in 2007 to 83.68 per cent. And 100 of the 900-odd schools saw a pass rate of 100 per cent, as compared with 41 schools in 2007.

But Education Minister Arvinder Singh put the improved results largely down to “better teacher training” rather than HOTS.

(Indian Express)

So is it better training or HOTS? Probably both, but I suspect more HOTS than training – the effect of training shows up slowly over time, while HOTS was a sudden change. And I haven’t gone through enough data on the new HOTS papers to judge how conceptual they actually are.

But HOTS still doesn’t fix the core issue which is that university-entrance tests and school-leaving tests should be wholly different beasts. It also doesn’t fix the one-shot problem. Wonderful though the results of putting HOTS into place appear to be, HOTS is an incremental improvement on an examination system that needs radical redesign. Just to reiterate, the pieces which are still missing are:

  1. Measuring classroom performance
  2. Separating school-leaving competence and university entrance competence for everything, not just professional courses
  3. Making examinations better at measuring conceptual skills rather than mugging skills- HOTS is supposed to do this and may actually have accomplished it, but I would like that to be subjected to rigorous testing
  4. Redesigning exams (school leaving and university entrance ones) so that taking them over and over, or the amount of time you spend on preparation has less impact on how well you do- I think HOTS is meant to accomplish this as well, as was the JEE redesign a year or so ago.
  5. Making university entrance more dependent on multiple factors like extra-curricular skills, conceptual skills, ability to be a stud and a fighter, instead of just a since exam score. But this is probably not going to happen without a massive supply-side expansion of good universities (please see Abi and Ravikiran on this).

Tragically, I have been writing this post over a month and a half, after intending to write it six months ago, and it is now much less coherent than I hoped it would be. My apologies, and I aim to clarify confusion in the comments. Have away!

7 Responses to The Trouble With Exams

  1. Aishwarya says:

    Not only is the process broken, it’s broken in a way that disproportionately hurts the poor. Why? Well, because:

    1.In the situation described above – you mess up your exams and need to wait another year, the opportunity cost of sitting out another year is much lower. If you’re rich, or even middle class, your family has enough savings (or enough of a standard of living to cut back a little) to not worry about starting your career a year later.
    2. If you have a one-year run up to the exams, that gives rich kids a year in which to hire tutors to help them prepare for the exams. This becomes even worse when the CBSE makes papers ‘easy’, because when papers are well-designed and difficult, topping them is truly a function of how smart you are. When they’re easy, students who have time to spare and money for tutors crack the scene. When they’re difficult because they’re badly designed, the top ranks become a lottery.

    Yep. Also keep in mind that if you don’t do well in the CBSE, if your family are reasonably well off (/upper-middle-class/ People Like Us) it’s still possible to play the system to get into one of the better colleges. Particularly in an institution like DU. Obviously that isn’t an option if you’re poor. I will probably be posting on this myself soon.

  2. Shirish says:

    Yes.. Perfectly written.
    Having exams just once a year puts restriction and specially in case you cannot afford to wait for one more year. Lots of students in my opinion face this problem. Such students either end up in taking admission in college not of their choice OR if its a PG entrance (like CAT) they end up dropping the idea of applying for the exam and start with profession career. Recently, there was a suggestion from education ministry to conduct CAT exam twice a year. Not sure what happened to that suggestion.

  3. […] Ravikiran reacting to a recent post of his on regulating private players in higher education. Also, Aadisht has a very interesting post on the examination system in our schools: There’s another major […]

  4. […] seeks to understand the purpose of examinations and reviews the new CBSE exam design scheme, HOTS (Higher Order Thinking […]

  5. skimpy says:

    and our deve gowda goes around saying that the CET (for entrance to engineering/medical in karnataka) is anti-poor and anti-rural and only 12th board marks should be used for admission purposes

  6. Rajaram S says:

    “Where the CBSE is concerned, the board exam is being used for purpose #1: certifying minimum competence. Delhi University, on the other hand, uses it for purpose #2: as a filter to select top-performers, which is not what the boards are designed for.”

    — Well said. This is unfortunately the case across the country. I am from CBSE and the year that i appeared for my 12th (94) was when there was a big furore abt the difficulty of the math paper with even cases lodged across the country. So, they ended up evaluating the paper leniently thus increasing the number of perfect scores much more than the usual. I felt pretty bad because i had done the paper extremely well and this normalisation nullified my advantage!
    I am from TN and the state board syllabus there is close to pathetic, requiring the students just to vomit exact phrases of text from their books thus giving them extremely high scores thus giving them “high” credibility where the student might actually not deserve so.
    A few weeks back, i was talking to my cousin who finished his 12th from the TN Stateboard (Computer Science batch) and his knowledge in CS was pathetic. In fact, it looked like i knew more when i was in 12th , more than 14 yrs ago!

    It’s a pity that this mugging abt stuff would determine your career!

  7. Sanchit Jain says:

    Actually most of the Indians fail to understand what boards are….If u have time 2 spare & a bit patience, I’ll tell u what they are……..they r actually mugging exams..Don’t move over & delete this, plzz read it once…If U’ve prepared for IIT-JEE, U’d know dat boards require just mugging ability & Maths in boards requires just speed. But IIT-JEE is so tough dat U need 2 be a genius 2 crack it….The questions r 2 check 1’s analytical ability & logical reasoning…..Just compare the papers of IIT-JEE & boards – U’ll know da difference.
    The average Indian fails 2 understand da difference between a fact & a concept.
    Consult any IITian (after 96 batch) & U’d know who’s ryt.

    Learning an answer doesn’t makes 1 a genius…itz da logic dat does it

    Plzzzz check out http://www.sanchitjain.co.cc

Leave a Reply