Chapter 8
Quality and Effectiveness of Education

A key objective of the European Union is to ensure the quality and increase the effectiveness of education. In Hungary, this field has received particular political and professional attention since the mid-nineties. The issues concerning quality and effectiveness are closely related to efficiency, which primarily refers to the relation between financial expenditures and result. The outcome of education is a highly debated issue, whether it is the knowledge of students, better opportunities on the labour market, or the value of the social capital increased by education, etc. The definition of quality is also a hot issue. We may observe three definitions that coexist simultaneously and are complementary in Hungarian education: (1) compliance to national standards, (2) adherence to local standards defined by the individual institutions, and their maintainers (3) satisfaction of stakeholders. All these refer to the quality of educational outcomes or ‘products’. The notion of quality is also applied to the entirety of the institutional operation, which may be tied to efficiency in management and to organizational culture. The effectiveness instruments are gaining significance in the quality assurance evaluation processes. Evaluation is no longer focused exclusively on student performance. The assessment of quality and effectiveness now involves a growing number of fields and activities, and there are increasing efforts in finding a link between the process of assessment and school development, educational management and planning.

8.1. Effectiveness in Hungarian public education

Student performance indicators still play a crucial role in the evaluation of educational quality and effectiveness. Hungary took part in several international assessment programmes and also conducted a series of national evaluations. The evaluations based on standardised tests reveal the actual knowledge behind school marks, showing that identical marks cover great differences in knowledge levels. National analyses of the PISA survey found that the performance of urban learners is on the average 8-9% higher than that of students in rural schools with the same marks. Such differences between urban and rural student performance have also been observed in the course of national monitor surveys. This gap had ceased to widen and actually substantially decreased by the millennium. According to the findings of a 2001 monitor survey, the differences between the two categories dropped below the levels measured in 1995.

Figure 8.1.
Deviation from the national average in the mathematics and reading performance of 8th-graders by settlement type (standardized score)

Source: Calculations by Péter Vári based on data from Monitor surveys

8.1.1. Reading literacy

In the PISA 2000 survey, performance of the 15-year-old Hungarian students on the combined reading literacy scale was ranked 22nd. The average performance of Hungarian students was 480 points, which is significantly lower than the 500-point international average. 48% of Hungarian students performed below the third proficiency level, which means that nearly half of them failed to reach the level of reading comprehension necessary for successfully entering the labour market. While 10% of students in OECD countries achieved the highest (5th) level, this rate in Hungary was 5%. To some extent the overall picture of the reading literacy of Hungarian students based on the PISA survey may be complemented with the findings of the 2001 IEA assessment (PIRLS) conducted among 4th-graders (9 to 10-year-olds). The survey aimed to assess the transitional stage in learning, during which students progress from the acquisition of reading skills to the completion of reading tasks as required by their educational needs. According to the initially published international report, the performance of Hungarian 4th-graders far exceeded the international average by attaining the 8th position, out of 35 participating countries. Based upon the PIRLS-survey and earlier national evaluations it seems that primary (ISCED 1) education in Hungary provides for the development of key learning abilities, one of which is the development of the reading comprehension of students. A complex analysis has begun to explore the reason behind the poor results of 15-year-olds in the PISA tests. The unusual nature of evaluation methods and test types, compared to the previously used methods (used for example by the IEA), and the type of curricular attainment targets in Hungarian education are crucial factors of consideration when assessing the results.

8.1.2. Mathematical literacy

According to the IEA-studies (TIMSS 1995, TIMSS-R 1999), the performance of Hungarian 8th-graders was not only above the international average, but even improved from 1995 to 1999. At the same time the gap between the best and worst performance levels increased. This may explain why the lower than average values of the Hungarian students in mathematical performance came as a surprise in the course of the PISA-assessment, producing scores significantly below the international average. Experts believe the main reasons for the poor performance levels may be attributed to the differences between the content and practice of Hungarian education and the approach of the PISA assessment. This view is in line with the opinion of teachers who thought that a large part of the PISA tasks were missing from the list of common tasks used in Hungarian schools. It is worth pointing out that Hungarian students provided a higher than average performance in the same tasks as their Czech, Polish, Korean and Japanese peers did, which goes to show that students with a similar knowledge-structure are capable of levels of performance far above the Hungarian results. However, students from countries such as Sweden, Norway and Finland displayed a higher level of performance in tasks where the Hungarian students performed poorly.

Figure 8.2.
Deviation from the international average in mathematics for various countries in the 1999 TIMSS-R (8th graders) and the 2000 PISA (15-year-old) surveys (standardized score)

Forrás: Vári et al., 2001

8.1.3. Scientific literacy

Scientific literacy of Hungarian students has received feedback from two international surveys, providing a varied picture, similar to the performance feedback of mathematics. According to the results from IEA’s 1995 TIMSS and 1999 TIMSS-R studies, Hungary has shown significant progress in the comparison of the two evaluations, finishing second behind Canada. 15-year-old Hungarian students displayed an average level of performance in the PISA assessment, a level far below the previous performance levels of the IEA surveys. As opposed to the tasks of the PISA assessment, in the Hungarian course of science education and to a large extent in the practice of the TIMSS, students are expected to utilise known algorithms or their combinations to calculate exact results. From the teacher’s point of view, the most unusual feature of the PISA tasks is that, in contrast to the Hungarian practice, students are expected to understand the scientific concepts of phenomena, notice evidences and make decisions based on probability, usually requiring the extrapolation of conditions.

8.1.4. Civic education

In 1999, the IEA conducted a civic education survey in 28 countries to assess the levels of civic knowledge, engagement, and the attitudes of 14-year-olds. Hungary was also involved in this survey. The performance of Hungarian students was on an average level in cognitive questions related to civic knowledge. They performed above average in the field of factual knowledge, yet their ability to interpret was merely average. These results are also supported by the opinions of teachers teaching subjects of civic education. 71% of teachers agree that the dominant form of education is delivering information. Generally speaking, students believe that patriotism has a key place in school education, but feel that insufficient attention is paid to the application of knowledge, to familiarising with other countries, and to understanding different viewpoints.

8.1.5. Information technology

In the past decade the competencies related to information and communication technology have become a key priority in educational policy. According to a survey¹, almost half of all Hungarian students lack the infrastructure for doing homework, which requires the use of information technology equipment at home, while most school libraries lack the sufficient resources and staff in order to provide an appropriate source of knowledge on the matter. Based upon the findings on ICT in the PISA survey, one of the reasons why the situation of Hungarian students differs from the OECD average is that Hungarian students show a higher use of computers at school than at home.

Table 8.1.
Frequency of computer use at home and at school among 15-year-olds in some OECD countries, 2000 (%)

	Computer use at home					Computer use at school
	Almost daily	Once or twice a week	From once a week to once a month	Less than once a month	Never	Almost daily	Once or twice a week	From once a week to once a month	Less than once a month	Never
Belgium	38	26	13	7	17	5	26	32	12	25
Czech Republic	31	15	8	4	41	4	24	34	11	26
Denmark	44	25	14	7	9	23	36	26	11	4
Finland	45	22	10	5	18	6	41	30	16	7
Germany	43	23	14	7	14	4	14	25	20	37
Hungary	29	14	7	4	46	7	58	19	5	10
Ireland	32	23	10	5	30	4	22	25	14	35
Norway	53	22	11	6	9	6	22	33	28	11
Sweden	60	21	9	3	6	16	29	27	17	11
Switzerland	39	25	17	7	12	5	17	37	20	21
USA	49	18	12	6	15	18	19	23	23	17
Average	37	20	11	5	27	10	27	26	16	22

Source: Education at a Glance, 2002

Hungarian students performed well in information handling skill-tests. The majority of students are able to study alone in an environment aided both by IT tools and printed resources. There is no significant difference between the information handling skills of girls and boys. Students performed well in knowledge assessment tests, showing a reliable level of basic knowledge. However, the content of their knowledge is problematic: numerous activities indispensable in the workplace are missing from the school curriculum, and if included, are insufficiently presented.

8.1.6. Foreign language

In 2000, the English and German language skills of 6th, 8th and 10th-graders were assessed, based on a national representative sample. The results indicate that the majority of students have acquired at least one foreign language at a level comparable to the attainment targets of the curriculum for the given age group. However, it is important to stress that the active writing skills in the foreign languages in question are considerably below the levels of reading and listening comprehension skills among students in their final year at general school, while according to all indicators, the results of girls in this age group are better then the results of boys. Similar to performance levels in other fields, the language learner’s command of a foreign language is strongly linked to the educational attainment of the parents.

Table 8.2.
Foreign language test results of 8th-graders by educational attainment of the mother (% of performance)

Educational attainment of the mother	English			German
	Reading	Listening	Writing	Reading	Listening	Writing
Completing general school (8 grades)	45.6	56.1	14.5	52.5	50.6	13.9
Skilled worker qualification	48.2	57.7	17.9	55.2	51.9	14.8
Secondary school-leaving certificate	61.0	64.8	28.4	60.8	53.1	22.6
College degree	71.1	72.5	42.0	67.6	54.7	30.7
University degree	75.7	75.5	49.4	65.2	53.0	28.1
Total	60.8	65.4	30.1	59.6	52.7	21.0

Source: Csapó, 2001

One of the indicators of the success of language education is the number of language exam certificates obtained in the course of secondary education, which also plays a key role in progression to higher education. An average 44.9% of those secondary school graduates admitted to higher education and 29.7% of all applicants possessed a language proficiency certificate in 2001. Most of these were gained by taking intermediate level exams. There are significant differences in the language proficiency of secondary school graduates, based on the type of the language programme in which they have participated and the duration of the study programme. The rate of secondary school graduates without a language proficiency certificate is below 40% in 8-grade general secondary schools, 47.4% in 6-grade programmes, and 83.7% in 4-grade general secondary schools. This rate is even worse among school graduates from secondary vocational schools, while only 10–12% of students participating in economics programmes possess such a certificate before taking the secondary school-leaving exam.

8.2. Factors behind the success of learning

The results of the PISA survey indicated that Hungarian students prefer to study on their own, and compared to students in other OECD countries, they do not prefer co-operative learning. In contrast to the majority of OECD countries, the students in Hungary who adopt learning strategies built on memorisation achieve far better results than their peers. One of the strongest factors behind the different scores of student performance is interest and engagement in the given learning activity. Hungary belongs to a group of countries defined by the PISA survey as one in which student performances are influenced above the international mean by the socio-economic status of the student family and by the presence of classical cultural assets. The success of learning is influenced far above the international average by the school type on the one hand, and by the differences between settlement types on the other, which may also provide an explanation for the differences in performance levels of students attending different school types with different curriculum requirements.

Figure 8.3.
Average reading performance of 15-year-old students by school type, in PISA 2000 survey (standardized score)

Source: Vári (ed.), 2003

8.3. Additional indicators and evaluation instruments for effectiveness

While educational policy-makers have strongly emphasized content development and curricular regulations in the past 15 years, the assessment of student performance and evaluation of institutional operation only began in the last one or two years. The reform of the examination system in public education began in 1997, when the examination regulations for the basic examination and the two-level secondary school-leaving examination were published. Having undergone several amendments, these regulations will come into effect in 2008 and 2005, respectively.

8.3.1. The examination system

Although the primary function of the examination system is to evaluate and certify the achievement of the individual students it is also an important source of feedback on the performance of an institution or the educational system as a whole. Generally speaking, examinations in Hungary either conclude an educational/training phase (secondary school-leaving exam, vocational examination), or form the basis of admission to the next educational level (secondary school and higher education entrance exams).

The role of the secondary school entrance exam gained particular importance in the case of 6- and 8-grade general secondary schools, which are forced to adopt a selective procedure due to the large number of applicants. In 2000, a central entrance examination was introduced in these institutions. The aim of this initiative was to replace the system of heterogeneous and wide-range examinations organized by schools with an instrument capable of revealing the learning abilities and attitudes of students, producing results which are the least dependant on the socio-economic background of the students. The central examination prevents students from having to sit for several entrance exams at different locations even when applying for entrance to more than one school.

The secondary school-leaving examination

The secondary school-leaving examination is the most important exam in the Hungarian system, which concludes the secondary school studies of students and provides the right for admission into higher education. The uniformity of the Hungarian secondary school-leaving examination is guaranteed only by the central assignment of written tasks, and by the standard instructions for correction and marking. As a result, identical marks might cover great differences in the actual performance. The development of a uniform national school-leaving examination as well as the standardization of the requirements and marking criteria has been an oft-expressed need. It was partly because of this demand that the Ministry of Education ordered for the exam papers in mathematics and Hungarian language and literature, previously marked only at school level, to be re-marked on a nation-wide basis. There were hardly any differences shown between the marks given by teachers in the schools and by national experts in the case of mathematics. In the case of the language and literature exam papers, however, the second marking showed a more significant difference. For some of the papers, the appointed national experts gave one mark lower than the school teachers.

The development of the school-leaving examination has been an ongoing process for several years in Hungary. The requirements of the examination were reviewed and amended in 2003 together with the frame curricula, and, as a consequence, the level of requirements has been reduced and skills development has received a greater emphasis. In all exam subjects, there is two-level (intermediate, advanced) specification of requirements. From 2005, the secondary school-leaving examination of levels specified by the given higher education institution will replace entrance examinations. The students themselves will have the right to choose the subject and the level of the exam to be taken. Secondary schools will have to provide opportunities for students to prepare for either of the exams, while all written exams will be uniform and centralised. The assessment of exam results will be based on standardized marking criteria. The intermediate level exam will remain school-based, while the advanced level exam will be organized externally.

8.3.2. Progression into higher education

The number of applicants and admitted students to higher education has been used for years as an indicator for the effectiveness of secondary schools. These schools are ranked according to this indicator. There is a close relationship between admission rates and the social composition of students in a school, therefore such findings provide just as much information on the social characteristics of schools than on the actual effectiveness of education. The expansion of higher education was accompanied by an increasing gap between the group of secondary schools with the best admission results and the group of those with the worst admission rates.

Figure 8.4.
Admission rates from different secondary school programmes to higher education, 1991–2001 (%)

Source: Calculations by Judit Lannert based on Neuwirth, 2002

Inequalities seem to be increasing in many dimensions. The gap between counties, towns and different sized settlements has further increased. While in 2001, for example, 40% of 12th-graders studying in towns with a population over 100 thousand were admitted to higher education, the same rate in settlements with 25 to 50 thousand inhabitants was 33%, in settlements with 5 to 10 thousand inhabitants this amount was 26%, and in settlements with fewer than 5 thousand inhabitants it was 19%.

8.3.3. Value added by educational institutions

The latest studies on the measurement of school effectiveness attempt to assess a kind of added value. In an analysis including 424 secondary schools, the various institutions were ranked according to the background of students (the educational attainment of parents, the rate of unemployed parents, marks in different subjects received in general schools) and then compared to the school’s rank according to the indicators of educational effectiveness (rate of students admitted to higher education/secondary school graduates, rate of students with language proficiency certificates, points scored at entrance exams). Based on the comparison of the two ranking positions, schools that were ranked higher according to their effectiveness indicators rather than by their social composition may be said to have ‘greater added value’. The analyses indicated that approximately 6-10% of schools had significantly better or worse effectiveness indicators than what their social composition might suggest. These value-added analyses provide useful information not only for the school maintainer, but also for educational development. Furthermore, they offer a chance of revealing educational, school management, organizational and economic factors that may lead to exceptionally good or poor performance.

8.4. External evaluation of schools

The external and self-evaluation of schools is an exceptionally important device of assuring quality and effectiveness. In Hungary the task of carrying out regular external evaluation (compliance with the procedural regulations of the operation, management, educational supervision) is the responsibility of the maintainer. The schools and maintainers can choose evaluation specialists from the ‘National Expert List’. Unfortunately, reliably operating methods and procedures for assuring and continuously monitoring the quality of education in the current decentralised institutional structure have not yet been developed. This is equally true concerning the external evaluation of schools: there is no guarantee that there will be regular institutional evaluations conducted in every school according to professional standards, and that, based on its results, the appropriate measures will be taken at the school or maintainer level.

It is characteristic of the nature of evaluation and supervision that according to a 1999 survey, 87% of the evaluation of school maintainers focused on the observance of economic and managerial regulations, 78% on the observance of legal provisions, and only 38% evaluated educational work from a professional point of view. Local governments often use the instruments of evaluation and supervision without real commitment to professional development. It seems that some of the school-maintaining local governments conducted evaluations merely to fulfil their legal obligations. According to a 2001-2002 survey, involving 507 general school maintainers, 390 (76.8%) conducted pedagogical assessments between 1996 and 2001 in at least one of their schools. The proportion of local governments who fail to perform this task is exceedingly high in smaller settlements. As for the evaluation of maintainers, the non-governmental sector deserves special attention. Standardized institutional evaluation techniques have been used by foundational and private schools since the early 1990s.

In 1999, the Ministry of Education created the SZAK system of tenders, which partly served school level evaluations. In the course of three years (from 1999 to 2002), the Ministry produced a financial framework of HUF 3 billion for local governments, in order to employ independent experts for the evaluation of their institutional system and in the development of local managerial skills.

8.5. Involving new stakeholders and tools in developing quality and effectiveness

The increasing number of stakeholders, a widening range of tools, and a growing number of training programmes on the know-how of quality and effectiveness management all indicate that there is an increasing interest in educational quality and effectiveness on the level of educational policy whilst the related activities are becoming increasingly professionalised. In the period under review, a new national centre was established for the co-ordination of the tasks of evaluation (National Public Education Evaluation and Examination Centre – OKÉV), whilst this proved to be the most dynamically developing field of all pedagogical services. The Kiss Árpád National Service Office of Public Education (KÁOKSZI), one of the background institutions of the Ministry of Education has been reinforced and specifically endowed with the task of pedagogical assessment. The Office is responsible for the Monitor programme and other assessments, the organization of the implementation of international research projects (IEA, TIMSS, PISA) and also plays a key role in developing different national diagnostic knowledge assessment tests. The Hungarian universities also play an important role in developing tools for quality assessment and evaluation, especially the University of Sciences in Szeged, where an experienced research group is developing up-to-date assessment tools and conducting national representative surveys.

8.5.1. Quality assurance and the Comenius 2000 quality improvement programme

The most significant measure taken in the field of quality and effectiveness development and evaluation has been the establishment of the Comenius 2000 quality improvement programme. The programme was introduced after the 1999 Amendment to the Public Education Act, following several years of professional preparation.

It is not easy to apply the quality assurance techniques in schools that were employed in the economy. The Comenius 2000 programme resulted in spectacular organizational development in some of the institutions, whereas in many cases strict regulation of the institutional processes remained formal, strengthening the bureaucratic character of the organization. After the 2002 general elections, the Ministry of Education initiated the review of the Comenius programme. The new administration recommends the programme only as an optional quality assurance model for institutions. However, in order to ensure the dissemination of the practice of quality assurance, a law was passed obliging all public education institutions to build their own quality improvement system and to define it in an individual document approved by the maintainer.

8.5.2. The system of national diagnostic knowledge assessment

With the implementation of the new national diagnostic knowledge assessment test, the Ministry of Education wishes to achieve two basic goals. Firstly, the idea is that the content of the tests and methods of the evaluation used in the assessments will convince schools to react to the need for new content development. Secondly, they intend to develop the culture of assessment and its methodological background at the institutional level. Although the assessments naturally yield findings that could be used to evaluate schools and the system of education, these were not intended for external institutional assessment. The aim of the survey was to improve and disseminate the assessment culture, to create a database and establish a method with which self-images become comparable with real images, to make the educational added value measurable and to diagnose student performance and related developments. Based on experiences with tests used in international surveys, a series of up-to-date questionnaires have been developed for these assessments, which were conducted in the first, fifth and ninth grades of all schools. The findings reinforced the earlier results concerning regional inequalities, and showed how this inequality varies in the fifth and ninth grades. On a standard scale with an average of 500, the greatest difference was measured between the mathematical literacy of ninth graders studying in the capital and those attending village schools, showing a difference of 73 points. In contrast, the results of the assessments in the first grade did not show significant regional differences similar to those of the upper grades. In other words, schooling does not diminish initial differences, but rather increases them.

Figure 8.5.
Performance on reading and mathematical literacy in the fifth and ninth grades by settlement type, 2001 (standardized score)

Source: National competence assessment 2001. OM – KÁOKSZI