23 March 2005

Ladies and gentlemen

The publication Annual Review 2003/4 is the record of our work in that year.

We also publish today on our website further related reports:

• The Annual Curriculum Report provides an overview of the findings from our monitoring and evaluation programme, highlighting both the successes and challenges of curriculum, assessment and qualifications practice and innovation in 2003-4.
• And our annual report into the performance of AQA, Edexcel and OCR in delivering GCSEs and A Levels, which reports against a number of performance metrics.

I don't propose to go through the Annual Review or the other reports in any detail: together they constitute a record of very considerable achievement. They are there to be read.

What they do show is the breadth and scope of our work.

As the Chairman has said, we are on about lifelong learning: our activities affect people in this country from age 3 to age 93, but so often people know only about one part of what we do.

Our challenge is to make sure that all the parts interconnect seamlessly, to support the child, the young person, and the adult pursuing perhaps several careers throughout working life.

There were some highly significant new developments in 2003/04, which will alter the course of education and training nationally, such as:

• our work on vocational qualifications, on unitisation and credit, on articulation between 14-19 education and adult education within the context of a new, simplified and navigable national qualifications framework, known as the Framework for Achievement;
• engagement with each of the sector skills councils, to develop for each industry an industry-driven sector qualifications strategy, to meet national skills shortages and to anticipate and respond to emerging needs;
• the transformation of examination offices in schools and colleges, to the great benefit of young people;
• the intelligent use of teacher assessment at key stage 1, an important step on the road to demonstrating the effectiveness of teacher assessment;
• much greater curriculum flexibility at key stage 4, reducing the statutory requirements and enabling pupils to pursue courses appropriate to their needs and aspirations;
• the KS3 ICT test, which QCA is piloting in hundreds of schools this summer - the first full cohort test to be entirely delivered and assessed electronically when it goes live in 2008, and paving the way for a new generation of assessment on-line and on-demand;
• and for the first time in history a religious education framework agreed by all faith groups, the result of an extraordinarily successful consultation, and a powerful basis for citizenship and understanding in a multicultural, multiracial society.

In regard to public examinations, the overall performance of the awarding bodies improved from 2003 to 2004.

There has been significant progress on the modernisation agenda, which David Gee will report on later.

I can certainly report confidence in the examination system, and that the issues of 2002 are well behind us.

But there was one major problem with the key stage tests: the failure of ourselves and our contractors to deliver the KS3 English results on time and to the satisfaction of our clients - a failure which has brought about major changes to the administration of the tests for 2005.

That failure was all the more disturbing because I have come to appreciate, perhaps more than those with less experience of assessment systems outside the United Kingdom, how rigorous and painstaking the test development process is in this country.

The development of the tests and the mark schemes is contracted out to several national agencies of great expertise and with international reputations.

The development of each key stage test extends over two years.

The first step is the trialling of more than three times the number of questions and themes that will eventually be needed, in schools, together with the draft mark schemes.

The performance of these test items and mark schemes is then reviewed by the NAA, by subject experts in Curriculum Division, by panels of teachers, and by test review groups of subject specialists including representatives of local education authorities and higher education.

They are modified as a result.

The modified test items and mark schemes are then pre-tested, not once but twice.

Two different versions of the English, maths and science tests, at both key stage 2 and key stage 3, are pre-tested with 400 students per version per test.

They are marked by the most experienced markers, who also advise on further refinement of the mark scheme.

We then review these again, by the same intensive process.

On that basis, revised questions and mark schemes go through a second pre-test, again with sample sizes of 400.

Following a third review the tests and the mark schemes are finalised: the live test, and a back-up.

So in summary, over two years the test items are narrowed down to produce a final test, which we know to be valid and reliable; and the mark schemes are increasingly refined, expanded and illustrated with example responses, as a rubric which all markers can be trained to apply.

You will not find a more thorough process anywhere in the world.

The mark schemes are state of the art.

For example, the mark scheme for the key stage 3 Writing paper requires markers to mark

• sentence structure and punctuation,
• text structure and organisation, and
• composition and effect;
• each has five bands of discrimination;
• each band has defining descriptors of performance at that band;
• and there are worked examples on which markers are trained.

Now, this is without doubt international best practice. It is the Rolls Royce of the test development world. It is much more thorough than other systems I know intimately.

On the logistics of marking however, we are far from international best practice.

International best practice is to distribute scripts to markers randomly.

We send all the scripts for a particular school to the same marker.

International best practice is to scan scripts and send them electronically.

We move millions of scripts physically around the country.

On-screen marking of scanned scripts, with automatic totalling of marks, is now the international benchmark.

We still do it manually.

Intensive second-marking to detect aberrant marking and correct it early is the rule.

We have too few markers, in relation to the demands of the overall testing program, to deliver the degree of second marking which in other countries makes it unnecessary to have our extended period for review of scripts on appeal from schools.

So, while we are state of the art in our test development and support, we have more work to do on the logistics side.

I point out however the additional and as yet untapped potential which lies in our national investment of expertise and resources in test development.

We produce extraordinarily rich material for use by markers, which can also be used by teachers.

This material is potentially as much a teaching tool as an assessment tool.

For example, in key stage 3 English

• we have put a huge investment into a test which occupies three and a quarter hours for 625 000 14 year olds on 5/6 May;
• their papers will be marked by August 22, and reviewed if necessary by November;
• their performance is marked by markers who do not know the pupils, and have no knowledge of their capacities other than the answers they have given under pressure in those few hours;
• and this determines the outcome of their year's work, the place of their school in the performance tables, and the success or otherwise of their teachers.

Yet the products of our test development process are so rich that they have immensely greater potential to be far more beneficial than this.

I support full cohort testing and national reporting.

I believe that assessment can be both nationally summative, and locally formative, in that it can be used to shape learning programs for the individual pupil.

I believe also that we can achieve both the summative and the formative objectives of assessment better than we do at present, by applying the products of our test development process in a different way.

What we have developed are frameworks and materials:

• which could be used to support schools in administering common tests, or tests from a common test bank, on chosen and repeated occasions throughout the year;
• which could be marked by teachers using a common mark scheme; and
• which could be closely monitored by trained assessors or chartered examiners, auditing the application of the materials and authorising the result.

We are, in this country, so much closer than our competitor countries to having the best possible basis for rigorous teacher assessment which:

• meets the legitimate requirements of government, by providing reliable national performance data based on testing of the full student cohort;
• meets the requirement of being comparable year on year;
• is based on far better evidence than at present, being gathered routinely and systematically by the children's teachers; and
• is formative as well as summative, in that it is genuinely useful and immediately available for shaping further learning.

I turn now to the issue of standards, a term which is used loosely and ambiguously.

There are two sorts of standards: assessment standards and performance standards.

The assessment standard is the height of the hurdle to be jumped.

The performance standard is the number who succeed in clearing that hurdle.

It is the job of the QCA to hold the height of the hurdle constant; and to ensure that from year to year it is not lowered, nor raised, but maintained.

We do so successfully, and on the basis of independent opinion, no country does it better.

It is the job of schools, colleges, the Department for Education and Skills and other elements of the education community to raise the performance standards year on year, and they are succeeding.

The debate on standards is fuelled by deep ambivalence in the community about the purposes of education.

Some see education as a winnowing device to sort the wheat from the chaff. They deplore the continuing increase in performance standards and take it as evidence that assessment standards have been eroded.

Others see education as a process for realisation and liberation of individual potential. They believe that educational innovations (the best of which is good teaching of good curriculum) actually work, and they see improved performance standards as a cause for national celebration.

From time to time, it has been said or implied that QCA has lowered national curriculum test or examination standards to make performance standards and the government’s performance tables look better.

No Minister or Secretary of State in my experience has ever exerted any pressure in that direction on the QCA. Were it to happen, I would immediately declare so publicly.

There is also, from time to time, misunderstanding or misrepresentation of what marks actually mean.

It has been said that you can get high grades for low marks - the latest example being a grade B in maths by scoring only, say, 17 per cent.

The reality is that the range of ability and achievement in mathematics is so great that there are three papers: higher, intermediate and foundation.

Students take the intermediate paper and one other.

Two thirds of the questions on the higher tier paper are targeted at grades A* and A.

Certainly, a B may be 17 per cent, but it is on the higher tier paper.

It is a highly creditable performance. It is as if the actual score were 217/300, not 17/100.

The essence of a test or examination is that it is a valid measure of what is to be taught and learned, and that it discriminates reliably between degrees of accomplishment.

This is achieved now as well as it has ever been.

But curriculum changes.

How many youngsters today could manually extract a square root of a large number or use a slide rule, as in my generation. Very few - not because of a drop in standards, but because of curriculum keeping pace with technological change.

How many young people of my generation could have answered all the questions on last year's A level physics paper? None, because so much of contemporary physics is the product of the last thirty years.

It is important of course that we maintain a time series year on year for the purpose of reporting change in performance standards.

But I believe we should not be afraid of holding an informed discussion about standards in tests and examinations, and how they should be maintained against the constantly evolving curriculum. Incorporating the Advanced Extension Awards into A Levels will be an important step forward.

I conclude on the subject of the recent 14-19 White paper.

The advice of the QCA Board to the Secretary of State is on the public record. That advice was not taken in full, and to that extent there is disappointment.

We believe nevertheless that the White Paper is an agenda around which much-needed reforms can be built.

The continuation of GCSEs and GCEs as qualifications is consistent with QCA advice that these qualifications should remain until there is a future diploma structure which would preserve and enhance their strengths.

The critical point is that the Government has now given the green light to the development of a diploma structure.

That is good news.

The diplomas can embrace both general studies and vocational learning.

GCSE and GCE qualifications, and units from within them, can be taken as part of the diploma structure.

There is now a real prospect of driving up retention, participation and performance post-16.

Young people will now have a choice of four routes:

• The traditional route of GCSEs and A levels, which include both general and applied (or vocational) studies, and in which each subject is a qualification. This can lead to a degree.
• The National Diploma route
  
  (It must have such a title to distinguish it from the occupational diplomas, although it isn't given one in the White Paper. Further, the terms 'specialist diploma' and 'open diploma' have lost much of their relevance in the context of the White Paper.)
  
  The diploma route:
  • includes both general and vocational (or applied) studies at levels 1, 2 and 3;
  • can include some general qualifications or units from them;
  • provides a broader range of learning at age 18 than the traditional three A Level diet; and
  • can lead to a degree or foundation degree.
• The apprenticeship route, which is wholly occupational, and can lead to a foundation degree or level 4 occupational diploma.
• The occupational route of awards, certificates and diplomas in named occupations - envisaged in the Framework for Achievement as the key employer-endorsed occupational route - which also can lead to a foundation degree or level 4 diploma.

And the Government will move forward on some other desirable reforms, including elements with strong appeal to employers, teachers and parents.

We will now set about developing the diploma so that it emerges as a qualification of status and value.

The traditional route and the diploma route must be allowed to evolve side by side.

Our advice was that the general qualifications should remain until 'something even better' can be introduced.

We now have the opportunity and the breathing space to develop it.

Clearly, the value and status of the diploma must be earned.

We must build a qualification which is recognised for its high quality by the learner, the teaching profession, parents, industry and higher education.

Its general components must be as robust as A Levels, and verified by universities.

Its vocational components must be similarly as robust, and verified by industry.

The vocational curriculum will not be designed to make young people job-ready, or to place them on an inevitable career path, any more than A-level French or history tracks you on a course inevitably to become a linguist or professor of history.

Its purpose must be to exercise and develop the 'learning muscle' - which is the purpose of all education at upper secondary level - regardless of whether the apparatus being used to build that muscle is French or engineering, history or joinery, physics or photography, biology or hospitality and catering.

It must be a genuine and real experience of part of our mainstream contemporary life, economy or culture, but beyond the bounds of traditional academia.

It must be taught in industry-standard facilities by industry-qualified teachers, capable of inspiring young people.

It needs teachers who understand that the co-ordination of mind, eye and hand, or reasoning, judging, doing - the essence of vocational education - is the platform on which to build higher order thinking, problem solving skills and other skills within the cognitive domain, as well as personal and interpersonal skills within the affective domain.

It must attract the full ability range of students, including the most able.

It must not be allowed to become a qualification with one hand tied behind its back.

In matters of qualifications reform, governments tend to follow rather than lead.

It will be ultimately the demand-side coalition of learners, industry and universities - together with parents and teachers - which will determine whether A Levels remain and for how long, and whether the National Diploma succeeds or fails.

If all of us who believe in this reform are successful, the question in ten years time will not be 'Why would you bring the A levels and the Diploma together?'

The question will be 'Why would you not?'

Ken Boston