Edit Sinka – Dániel Horn

Chapter 8
The Quality and effectiveness of public education

There is no professional consensus as yet about the meaning of the terms quality and effectiveness in education. This chapter uses the terms quality and effectiveness as synonyms. This approach reflects the understanding that a school is more effective than its peer if it can attain higher educational outcome from an identical starting point, i.e. it is capable of high performance relative to its capabilities and resources.

8.1. Effectiveness measured by pupils' achievement

8.1.1. Nationwide measurements of pupils' achievement

Several national and international performance measurement studies were carried out in Hungary during the period under review (between 2003 and 2006).

Table 8.1. Nationwide Measurements of Pupils' Performance, 2003–2006

	International	Domestic
2003	TIMSS – Survey of mathematics and sciences study; pupils in grade 4 and 8, IEAPISA – Mathematics, reading-literacy, natural science, problem solving; 15 years of age, OECD	National Assessment of Basic Competencies– Mathematics, reading-literacy; 6th and 10th gradesMeasuring the level of foreign language skills– English, German; 6th and 10th grades
2004		National Assessment of Basic Competencies – Mathematics, reading-literacy; 6th, 8th and 10th grades
2006	PISA – Mathematics, reading-literacy, natural science, problem solving; 15 years of age, OECDPIRLS – reading-literacy, 4th grade, IEA	National Assessment of Basic Competencies – Mathematics, reading-literacy; 4th , 6th, 8th and 10th grades

Each of the international and domestic surveys identified the same major problems regardless of the area of cultural domain studied:

1. 20-40% of Hungarian pupils lack or barely have the fundamental skills necessary for orienting themselves in everyday matters, i.e. their achievements are weakest at level 1 or below in the various grades and cultural domains.
2. PISA test results indicated that the achievement of Hungarian pupils in mathematical literacy and in reading literacy was below the international average, whilst they scored at around the international average in natural sciences and problem solving.
3. Hungary ranked second after Turkey from among the countries participating in the PISA survey in terms of the size of the difference between the achievements of schools.
4. It is in Hungary that the qualification of parents and other features of family background have the largest impact on a pupil's educational achievement at school.

8.1.2. Mathematical literacy

This cultural domain was covered by both PISA and TIMSS as well as the Ministry of Education and Culture. The two international surveys offer Hungary shockingly different views of Hungary. Whilst Hungarian 15 year-old pupils scored slightly below the international average in the mathematical literacy test of PISA, the TIMSS mathematical scale ranked the achievement of both 8th grade (average age: 14.5 years) and 4th grade (average age: 10.5 years) pupils significantly above the average. Hungary has continued to excel at TIMSS tests: the mathematical classroom achievement of both 8th and 4th graders has remained practically unchanged since 1995, which is in contrast with our achievement on the PISA tests, which puts us in the poorest one third.

The National Assessment of Basic Competencies calculated both pupils' test scores and (5) proficiency levels. 43% of sixth graders, 39% of eighth graders, 35% of tenth graders achieved at or below skill level one, as opposed to 6% of sixth graders and 9% of both 8th and 10th graders whose achievement reached the highest proficiency level. The PISA survey differentiates seven proficiency levels of mathematical literacy. 23% of Hungarian pupils scored at or below level 1 of the consolidated mathematics scale and although that achievement approximates the OECD average (21.4%), it is substantially higher than the 10% benchmark set by the EU. 10.7% of Hungarian pupils performed at proficiency levels 6 and 5 as compared to 14.6% of the pupils in OECD countries.

8.1.3. Reading literacy, native language competences

Not unlike in mathematics, Hungarian students scored somewhat below the international average in the 2003 PISA survey of reading literacy. An analysis of the proficiency levels shows similar results as in mathematics. 19.1% of the students in OECD countries scored at or below level one of the scale of six proficiency levels in reading and literacy. The same ratio is 20.5% for Hungarian students. 4.9% of Hungarian pupils reached the highest level of proficiency as compared to the international average of 8.3%.

The results of the National Assessment of Basic Competencies (NABC) are not more encouraging either. The competence test administered in 2004 classified 21% and 32% of Hungarian 6th and 8th graders, respectively, in or below level one, and the performance of 30% of all 10th graders put them in the same category, i.e. one out of three students is potentially incapable of solving everyday reading and literacy tasks.

8.1.4. Scientific literacy (natural sciences)

In contrast with their results in reading/literacy and mathematical literacy, Hungarian pupils (average score: 502) did not perform below the OECD average of 500 points.

Moreover, Hungarian pupils fared substantially above the average on the TIMSS test: 8th graders scored 543 points, which put Hungary's performance at the top of the list of European countries, whilst Hungarian 4th graders ranked third among the countries of Europe following the UK and Latvia.

8.1.5. Interdisciplinary competences: complex problem solving

The PISA survey of 2003 measured interdisciplinary competences, particularly the ability to solve complex problems. Interestingly, whilst Hungarian pupils performed below the average in mathematical and reading literacy, their average scores on the complex problem solving scale was not at variance with the OECD average. Hungary is one of five the countries (along with Brazil, Japan, Germany and Russia) that outperformed their own mathematical literacy score in problem solving: pupils scored 10 points more on average on this test. The OECD report suggests that it might mean that the instruction in mathematics fails to mobilize the general competences of the pupils in full.

8.1.6. Foreign languages

During the 2002/03 academic year, foreign language skills were surveyed on a representative sample of 6th and 10th graders and showed that both 6th and 10th graders were more proficient in English than in German. In the category of primary and secondary schools, respectively, those in larger settlements and general secondary schools excelled. The foreign language skills of 10th graders varied widely across school types. Vocational school pupils scored close to zero on composition tests. Not unlike other types of educational performance, the proficiency of pupils in foreign languages correlates closely with the qualification of their parents, and there are also significant differences between the foreign language skills of pupils studying at schools in different types of settlements and regions. The variances by settlement type among 6th graders can be accounted for by the lower availability of qualified language teachers in villages, whilst the lack of schools other than vocational training schools and vocational secondary schools in smaller settlements explains the differences found at secondary level.

8.2. Factors determining the effectiveness of learning

The degree of difference between the performance of schools and school types is one of the largest in Hungary from among the countries participating in the PISA survey. At 66.0%, the gap is almost twice as large as the average calculated for PISA countries, moreover, it continued to widen slightly during the three years after the PISA 2000 survey, when it was found to be 60.3%. In Hungary, 53.2% of the variation is between schools (which is the largest gap internationally) that is due to the social differences associated with the composition of pupils and is indicative of an extremely high degree of inequality. The education and the profession of parents and pupils' access to cultural goods have a larger impact on educational achievement in Hungary than in other OECD countries on average.

The analysis of the composite effect of these variables is supported by what is known as the economic, social and cultural status index. The impact of this index on pupils' performance is one of the largest in Hungary among the participating countries as it accounts for 27% of the variation of educational achievements, or in other words 27% of the performance of a child depends on the social status of the parents.

Chart 8.1. Relationship between family background and the achievement of pupils on the basis of mathematical literacy scores of the PISA 2003 survey

Source: Balázsi–Szabó–Szalay, 2005
N.B. The difference in pupils' achievement in mathematics triggered by a unit of change in the value of family indices means for instance that in case the educational attainment of a pupil's parents is higher by an extra year, the pupil's score in mathematics will be on average 32.5 points higher in Hungary as compared to no more than 22.4 points in other OECD countries. The PISA survey uses indices with mean at 0 and deviation at 1 to describe parents' educational attainment, occupational status, cultural possessions of the family (such as literature and the arts) and family structure (e.g. parents sharing the same household). The interpretation of column three: If a pupil belongs to the bottom quartile on a particular family index (e.g. parents' occupational status), the likelihood that the pupil will achieve in the bottom quartile on mathematics tests will increase by the factor indicated here (i.e. 2.1 times). The fourth column (explained variance) shows the percentage ratio of deviation of the maths score of pupils that a particular variable accounts for. The higher the value, the more significant the variable is in terms of influencing pupils' performance.

Further analysis of the OECD PISA 2000 data revealed that educational achievement is determined simultaneously by several more or less independent factors. Pupils' performance may be affected by factors beyond school control just as well as by those schools can influence. In Hungary, factors beyond school control, including the average social background of pupils, tend to have a much greater impact on success at school than in OECD countries on average. That is mostly owing to the strong segregation present in the system, which is why school type and the related average social background determine the achievement of those attending.

Table 8.2. The effect of various factors accounting for the variance of reading performance across schools in some OECD countries, PISA 2000 (%)

	Pupil features (%)	Factors schools cannot influence (%)	Factors schools can influence (%)	Unexplained variance (%)
Austria	56	19	8	17
Belgium	64	19	10	7
Czech Republic	60	24	4	11
Denmark	57	7	8	28
United Kingdom	33	35	11	21
Finland	14	3	16	67
Netherlands	72	17	3	7
Poland	50	16	15	19
Hungary	39	42	5	14
Germany	36	44	7	12
Norway	36	7	12	45
Italy	29	29	14	27
Portugal	86	7	m	7
Spain	43	18	5	34
OECD average	50	24	8	19

Source: School Factors…, 2005

8.3. Secondary school final examination

Exams play an important role in assessing the effectiveness of public education. Although exams serve to evaluate and recognize the performance of pupils individually, they may also give important feedback about the performance of an institution or even the system of education as a whole. That is an especially valid aspect of standardized examinations that offer more objective and comparable results. The new two-tier system of secondary school final exams was introduced in 2005. The new exam is standardized, is based on standardized requirements and is administered in line with a standard description. However, it fails to satisfy another requirement of standardization as it is not based on a tested and validated set of tests. But even with that deficiency, it is a more suitable means of capturing the effectiveness of public education and certain schools. The new examination system aims at measuring complex skills. Despite expectations to the contrary by professionals, students achieved better average scores on the new school final exam than on the “old type” tests used earlier. The results suggest that the requirements set for both levels were attainable and that teachers/examiners and also students had been prepared properly. (See also Chapters 4 and 5.)

Examination results seem to show even geographic distribution. Nevertheless, there are major differences between institutions of identical profile. The relatively sizeable deviation may influence the options of continuing education heavily and hence raises questions of equal opportunity. The results of the standardized school final exam also lend themselves to evaluating school performance. Data show that the school results of the intermediate level school final exam correlate very closely with the results achieved by the same schools on competence tests.

Chart 8.2. The distribution of schools by the results of pupils taking secondary school final exams after conversion to entrance test scores (%)

Source: Calculations by Dániel Horn based on school final exam results in 2005
N.B. School profile is derived from the programme with the highest attendance in a given institution (e.g. a school where most pupils attend a general school programme but there is a grade taking secondary school final exam is classified as a general school).

The results of the written test of the school final exam have been converted to entrance test score for each pupil taking the school final exam. The top score of the entrance test was 30 points, with 7 extra points earned by passing a higher level school final exam, which brought the total score to 37 points. (All other points that could be awarded at entrance examinations were ignored.)

8.4. Quality Assurance Policy Toolbox

Since 1990, several assessment tools have been introduced in public education in Hungary to serve as the components of a uniform system of quality assessment. Nevertheless, the system is not yet complete and its operation is not harmonized. The need for a uniform system was also confirmed by a parliamentary decision which served as the basis for the Ministry of Education to launch the development of a uniform strategy of quality assessment in public education.

8.4.1. National Assessment of Basic Competences (NABC)

National Assessment of Basic Competences is one of the most important quality policy tools of public education in Hungary. It is a survey with three objectives: firstly, to identify pupils' performance, secondly, to convey to schools the need to develop new content by new test content and methods of evaluation applied during assessment, and thirdly to promote the development of the culture of evaluation at institutional level and the related supporting methodology.

Administering the assessment under uniform conditions in every general and secondary school of the country is one of the advantages of this form of assessment, which is quite similar to PISA. Schools are requested to forward the test books of 20 pupils to the assessment centre for processing and are sent a separate analysis in return about their performance and expected effectiveness. That enables each school to analyse the achievement of their own pupils based on centrally processed data and to compare their performance to nationwide results and indicators. National Assessment of Basic Competencies supports this by providing schools with the (software) technology for processing results. Using the software, teachers can evaluate results locally by test, by individual and by class.

A nationwide survey suggested that more than three out of four school heads considered the NABC useful both from the perspective of school subject areas and in terms of establishing a culture of assessment and evaluation. The responses of school heads show that their schools mostly used the results of the competence assessment for curriculum development purposes and more than a third of the schools enriched their teaching programme with more practice-oriented tests modelled after the ones used in the assessment.

8.4.2. Evaluating institutions and quality assurance at institutional level

The system of evaluating institutions includes both self-evaluation and evaluation by the maintainer. In Hungary, there is no systematic and professionally controlled independent evaluation. The maintainers of institutions are responsible for evaluating the professional work of their institutions. The Act on Public Education specifically lists the tools that maintainer may use for this purpose. They may use the results of assessments and evaluations of independent experts, may request reports containing self-evaluation from their institutions and may seek the opinion of the school board. The Act on Public Education provides that maintainers must audit their schools regularly (at least once every quarter), which also covers the institutions' financial management, compliance with laws, efficiency and the evaluation of the effectiveness of professional work. Maintainers must disclose to the public their evaluation of the work of their institutions. (See also Chapters 2 and 6.)

The amendment of the Act on Public Education in 2003 required every kindergarten and school to develop its own system of quality assurance and to record it in a separate document approved by the maintainer (quality management program). Institutional quality management programmes must harmonize with the quality management programme of the local government. The latter formulates the expectations and duties of the maintainer regarding all the public education institutions of a municipality, it specifies the relationships between institutions and other branches (such as child and youth prevention, social policy, management of labour, community culture and health) and the procedures for professional, compliance and financial audits scheduled in the framework of the maintainer's management function. The clear identification and the accurate breakdown of the responsibilities of the maintainer are the greatest advantages of the programme. As there have been neither representative surveys nor studies aimed at data collection conducted regarding this area, relatively little is known about professional evaluations by maintainers and municipality level quality management programs. (See also Chapter 2.)

Quality assurance is one of the main objectives of central administration. The regulation issued by the Minister of Education in 2002 set the legal framework for quality assurance and development at the institutional level. The regulation seeks to specify the responsibilities related to quality development and it also established a Prize for Quality in Public Education as a form of recognition of excellence by the state of the outstanding performance of the institutions in this sector. The regulation requires every institution of public education to engage in quality development based on self-evaluation. The National Public Education Evaluation and Examination Centre promote this activity by issuing guidelines. The regulation is not limited to specifying institutional responsibilities; it spells out quality assurance duties primarily as regards professional assistance. The latter covers the development of methodologies, the compilation and maintenance of a list of quality development consultants (institutions may procure the services of quality development consultants for the purposes of performing quality development jobs) and the organization of quality development training and professional upgrading courses. The national assessment and evaluation system of public education must be organized and operated in the framework of nationwide quality development. A ministerial regulation issued about the procedures of an academic year specifies the measurements to be organized that year.