Arion - Conference on Quality Assurance

by Gábor Pertl

1.Introduction

1.1 Background and context of the report

This Arion Study visit was hosted by the Institute and Methodology Centre for Post-Graduate Teacher Training at Pilisborosjenő near Budapest, Hungary during the period 19-25 March 2000.

Participants came from eight European countries - one each from Finland, France, Ireland, Spain, Sweden and the United Kingdom, three from Germany, and two from Norway. Participants were drawn from various levels and areas of responsibility in their respective educational systems: some were in positions of school leadership, some acted as advisors to schools, others worked in the administration of school systems on a regional or district level, one participant acted as a legal advisor to a ministry of education, some taught in institutions providing teacher initial training or post-graduate in-service training, while others were inspectors of schools. Consequently, the participants brought to the study visit a range of perspectives on many aspects of the notion and process of quality assurance. There were obvious benefits in this diversity of perspective as the discussion and comparisons of national systems were considerably enriched, but it should also be borne in mind that the range of participants was not representative of all European countries nor even of all levels within the countries concerned. As a result the participants would not regard this report or the discussions from which it arises as providing a definitive description of the educational systems under review; rather, the account presented here is influenced by the institutional or system background of the participants and is indicative of their areas of interest. Nevertheless, the participants believe that the report provides useful comparative data and analysis, many aspects of which have direct relevance to the professional tasks facing their organisations.

1.2 Methodologies adopted

The study visit gave the participants a range of opportunities and contexts in which to consider issues concerning quality assurance. These included lectures and talks on approaches to quality assurance in general, and more particularly in the Hungarian educational system, round table seminars in which information from each of the participating countries was exchanged and compared, visits to educational research/support institutes which facilitate quality assurance measures in Hungary, and a number of school visits in which classes were observed and issues concerning quality assurance were discussed with principals and staffs of schools.

1.3 Defining quality assurance

A significant distinction is made between the terms quality assurance and quality control in the literature of management and business studies. Quality assurance generally refers to systems or processes that are designed to encourage all involved in the production or supply of a product or service to review their work constantly so as to assure the quality of the output: in effect, all workers are equally responsible for the quality of the output and it is intended that the maintenance, and if possible, the improvement of quality is a goal shared by all involved in the enterprise. Quality control refers to systems or processes which are designed to check on the quality standard achieved by persons involved in the production or supply of a product or service. It implies a checking process which is additional to the process of production. In general, quality assurance has come to be preferred rather than quality control as the former encompasses a striving for quality on the part of all involved in the undertaking. As the participants discussed the Hungarian school system and those of their own countries, it became apparent that a range of techniques was used to ensure that high educational standards were achieved in educational institutions. These techniques included processes which might be properly described as quality control mechanisms and others which would fit the criteria of quality assurance. For the purposes of the study, it was decided to examine both quality assurance techniques and quality control mechanisms, and unless otherwise indicated the term quality assurance is used in this report to encompass both quality control techniques as well as quality assurance approaches.

2. Key elements in the study programme

2.1 Elements of the programme

The full programme of the study visit is reproduced in Appendix 1 but it is worth noting the key elements which contributed to the success of this Arion visit. Broadly speaking, three types of activity were included within the formal programme: the exchange of system information between the participants, familiarisation with the Hungarian school system and its quality assurance mechanisms, and the discussion of the issues raised in the systems under evaluation.

2.2 Exchange of system information

A significant element of the programme consisted of opportunities for the participants to exchange data on the educational systems and the quality assurance mechanisms of their countries. Some information was also exchanged on aspects of the geographical features, culture, socio-economic development and political concerns of the countries involved. Much of this data was exchanged through semi-formal presentations and question sessions. These proved most valuable as the issues of concern in the quality assurance mechanisms of the various countries were clarified and illuminated through the resulting discussions. Moreover, the discussion of each country in turn encouraged each participant to reconsider aspects of his/her system, a process which was stimulated even further by the questions of fellow participants. Some additional guidance on the nature of the presentations to be made might have assisted participants in the preparation of their contributions. The extent of the time available for this aspect of the course was necessarily limited, but in general participants found these sessions most valuable.

2.3 Familiarisation with the Hungarian school system and its quality assurance mechanisms

This aspect of the programme was delivered through a series of well-chosen presentations from a number of key decision-makers and figures within the Hungarian educational system. The presentations of Mr Károly Pála of the Budapest Institute of Educational Services and Mr Gábor Halász provided excellent overviews of the Hungarian school system and the political context in which it existed. In particular Mr Pála presented a revealingly frank yet perceptive analysis of the strengths and weaknesses of the school system. His input was vital to participants' ability to consider the Hungarian system in an informed manner and it was felt that this presentation might have been even more valuable at an earlier stage of the programme. Information on the methods of quality assurance within the Hungarian school system was presented to the group by Ms Katalin Herneczki who spoke from the perspective of the Ministry of Education and Ms Judith Csapo who outlined the concerns and duties of the maintainers of schools. Ms Herneczki's description of an extensive pilot project on quality assurance within the school system was of particular interest to the participants. These presentations were complemented by a talk from Ms Éva Balázs of the Hungarian Institute of Education who described the Self-developing School Project which had been sponsored by the SOROS foundation in an attempt to foster the development of leadership and management skills within schools in the immediate post-socialist era. The participants also heard from Mr Miklós Csiszár about the adoption and development of quality awards and assurance measures in Hungarian business and on the attempts which were in train to apply these approaches in non-profit making organisations such as elements of the public sector. The participants' understanding of the school system and the effect of existing quality assurance initiatives were enriched by two school visits. On two mornings, the participants visited schools, observed classes and spoke with teachers and other key personnel. One of the schools was a state secondary school or gymnasium - a selective school catering for pupils of above average ability. The second institution provided education at kindergarten, primary and secondary levels. It was an independent school and as such received some state support but was also dependent upon other sources of revenue, the most significant being income from a private educational foundation and fees charged to parents on the basis of their ability to pay. The school visits provided invaluable insights into the operation of the school system and the effects of quality assurance approaches. Hungarian parents are free to seek enrolment for their children in the school of their choice but the most popular schools are heavily oversubscribed while other schools find it difficult to maintain adequate pupil numbers. During the discussions and visits it emerged that, in general, testing of pupils is used frequently throughout the school system. In practice, it is used as a selection device for admission to schools (including admission to many primary schools and in some cases even to kindergarten) and examination results are the key criterion of promotion to subsequent levels within the school system. The positive and negative effects of such selection through examinations and testing were discussed with the teachers, principals and others who accompanied the participants. A number of individuals expressed concern that while selection by popular schools resulted in the admission of highly motivated pupils to these schools it did not ensure good educational standards throughout the system as a whole. The distorting effect on the curriculum of schools was also highlighted. The second school visited was said to be untypical of Hungarian schools as it was founded to foster a specific minority denominational ethos and was in receipt of substantial private funding. However, the attempts it had made to adopt a broadly based child-centred education and its innovative approach to the assessment of pupil progress (utilising pupil portfolios, pupil self-review, and close contact between the school and family) provided interesting insights into the Hungarian system and the possibilities which exist to avoid the disadvantages of an over-emphasis on pencil and paper tests. Participants felt that if time permitted, the visit might have been enhanced by the inclusion of visits to further schools, and especially some in less favourable circumstances than those used on this occasion.

The participants' discussion of the Hungarian context and possible parallels with their own systems was greatly aided by a particularly illuminating analysis of data from national and international surveys of student performance presented by Mr Peter Vári of the National Institute of Public Education. The data referred to included that drawn from Hungarian participation in TIMMS research but it had been analysed in a most innovative way and illustrated some of the effects of existing school provision and quality assurance mechanisms in Hungary. The potential of such an analysis to inform educational debate and policy formation impressed participants but it was ironic that to date the implications of the data for the Hungarian system had not been fully realised.

2.4 Reflection and discussion

Much time was spent by the participants evaluating the information which they acquired during the visit. Discussions formed a significant element of the formal sessions and visits but invaluable work was achieved in informal the reflective discussions which are possible during a study visit such as this. The atmosphere created by the study visit organiser, Ms Gabriella Mózer, contributed enormously to this process and the institute's facilities were readily placed at the disposal of the participants.

2.5 Dissemination of outcomes

The compilation and discussion of this report helped to crystallise the impressions of the participants and it is hoped that it will provide a practical summary for participants as they return to their own countries. Interest in the study visit on the part of the Hungarian educational community and the broader public was indicated by the inclusion of a formal press conference in the programme at which some of the lecturers and a representative of the visitors spoke to journalists about the programme.

3. Comparing systems of quality assurance

3.1 Key characteristics of systems

A significant part of the discussions that took place during this study visit was concerned with the comparison of quality assurance approaches in each of the countries from which participants came and that of Hungary. Data on a range of system characteristics was supplied to the group by each of the participants. It was decided that the number of characteristics to be examined had to be limited, mainly so as to make the comparisons feasible in the time available. The characteristics on which the group chose to concentrate included:

• The definitions of quality adopted by the school system
• The membership of the group or level that defines the indicators of quality
• Information on the processes used to assure quality or evaluate quality
• Information on the output of the quality assurance process or the quality evaluation process
• The methods used to disseminate information on the progress and outcomes of the quality assurance/evaluation process, both within the institution and externally
• The effects of the process and/or the consequences of the process for the institution
• Any key advantages or disadvantages associated with the use of the processes described.

Some of the key data from this analysis is displayed in Table 1. The themes, which emerged from the discussion of this data and other contextual material, are examined in Section 4 of this report.

3.2 Limitations of the analysis

The participants were very aware of the limitations which are inherent in such a comparative analysis. The concentration on a restricted range of criteria necessarily limited the degree to which the analysis could accurately reflect all the complexities of the various educational systems. It may also be said that the aspects of quality assurance considered were skewed towards a quality control model rather than a quality assurance perspective. However, the group believed that the criteria chosen represented valid key indicators of the mechanisms adopted in their countries for the assurance of quality and found that their use, including the problems raised by their limitations, stimulated much worthwhile debate.        

Table 1: Examples of quality assurance approaches in use in participants' systems

Quality assurance technique
Used internally in the school	Used by external agency
School development planning
Development of a school plan, usually by the school staff and in consultation with the school's board of governors/board of management	Monitoring of the school planning process by an external authority such as the local/regional/state education authority
Examples and notes: This technique was in use in many of the countries. In some such as in Scandinavia and in Finland it was the primary technique in use and it was used both internally in the school and through external monitoring of the process. Advantages cited for this approach included the fact that a culture of self-improvement was one of the most powerful influences on quality and that the process could react well to local needs. However, difficulties had been experienced in having this approach adopted in some countries.
Use of questionnaires among pupils, former-pupils, parents and others (including employers of former pupils) to ascertain level of satisfaction with the school
Can be used as part of the school's internal self-development or planning process	Questionnaires may also be used by external agencies, to form a more complete picture of the level of satisfaction with the school.
Examples and notes: Use of such questionnaires was commonplace among systems where school self-development or school development planning was in place. However, the tool was also used by some external inspection systems to ensure that their assessment of schools was based on a broad range of data.
School developed assessment, including testing and examinations
Frequently used by schools and teachers to monitor pupil progress in achieving goals of the curriculum. However, the use of aggregated data by the staffs of schools to analyse their own stage of development was less frequently mentioned.	Some of the external agencies used information from this process in assessing the work of the schools.
Examples and notes: Implicit in the use of this is the notion that the measurement of quality may be wholly or partly based on the measurement of outputs. School-developed tools can be more broadly based and present a more comprehensive picture of student achievement.
Use of a critical friend to stimulate development of culture of self-review in the context of school development planning
This technique could be used to stimulate the internal review process. This was being attempted in several countries, including Hungary. However, the relationship of the critical friend to the school staff was crucial.	In some instances, external agencies, such as the local/regional/state/national authority had appointed ‘critical friends' to designated schools as in the model adopted by UK local education authority school advisors.
Examples and notes: This techniques attempts to enhance the effectiveness of the internal review process while bringing some of the benefits of an external viewpoint.
Use of school inspection system organised at local/regional/state/national level to monitor school effectiveness
Although this approach is almost entirely external to the school, some systems attempted to combine the role of ‘critical friend/adviser and inspector.	In general, participants looked upon inspection as a process external to the school. A range of techniques could be used by inspectors, including classroom observation, review of school plans, and data collected from questionnaires and student achievement results.
Examples and notes: Considerable variation existed in the degree to which the inspection systems were seen to impact positively on school improvement. In some cases, it was believed to distort schoolwork but others cited the advantages of having the school's effectiveness assessed not only by measuring outputs but also by assessing the educational process of the school. Access to the resulting report (e.g. if it was available to the school only or was published) and the degree to which it included data in numerical form were seen as key factors in influencing its impact upon school quality.
School-set scholarship examinations and testing State/national testing and examination
Aggregated data from pupil results could be used as part of the internal review process and this was mentioned in a number of instances.	This is primarily an external assessment mechanism. Many of the systems used it at secondary (and particularly upper secondary) level, but it was also used in some instances at primary level.
Examples and notes: Access to data from these examinations varied. In some systems, the results were published on a school basis but this was rare. The distorting effect on the curriculum of a sole or major reliance on examinations and testing was cited by participants. Participant also cited the inability of this tools to measure the ‘value-added' factor for which an individual school was responsible and the degree to which the school's effectiveness is misrepresented by this approach.
Assessment of pupil achievement through use of assessment instruments in sampling at local/regional/state/national level
Some countries made available to each participating school or class its overall results and its position in relation to other schools in the sample.	This tool is used by several countries and participants noted the growing trend towards the use of trans-national surveys such as TIMMS, PISA, etc.
Examples and notes: Arguments were presented that this form of quality assurance provided valuable and accurate data on the effectiveness of educational systems without causing the distortion brought about by the publication of results from whole-system tests or examinations. Participants were impressed by the degree to which this approach could reveal much information about the effectiveness of the system by using substantial banks of data available to national education systems.
Assuring quality through teacher education
	Some participants pointed to the importance of ensuring that entrants to the teaching profession were of a high standard. In some countries this was administered by teacher preparation institutions, in others the process was monitored by the regional/state/national education authority, in others the teaching profession was self-monitoring or was about to become so.
Examples and notes: Quality of teachers and quality of school leadership were repeatedly stressed as vital factors in assuring quality
Assuring quality through performance-related pay for teachers
	This approach can be used as an external control mechanism, where teachers are assessed by set criteria and their pay, or a part of their pay, or an additional allowance is linked to their performance against these criteria.
Examples and notes: Only one of the countries represented in the study group (the United Kingdom) was attempting to introduce this mechanism at present.

 

4. Themes in quality assurance mechanisms

During the study visit a range of themes emerged during the consideration of quality assurance mechanisms. These included:

4.1 Why quality assurance?

The discussions allowed interesting comparisons to be made concerning the impetus for the development of quality assurance mechanisms within the countries of the participants. A commitment to the provision of quality educational experience was an aspiration shared by each of the countries, and the need to equip the child with the knowledge, skills and competencies necessary for adult life was recognised. However, in many of the countries, debate had arisen on the need for mechanisms to ensure that these aspirations were achieved. This debate had arisen for a variety of reasons. The contribution which high educational standards could make to economic development had prompted the debate in some countries, in others such as the Scandinavian countries, Finland and Hungary, the decentralisation of a previously centralised school system had stimulated a desire to ensure that all children received an adequate educational experience. In other cases, political concerns, sometimes prompted by the use of comparative assessment data, had developed about educational standards and this had led to the establishment of formal quality assurance mechanisms, as in the United Kingdom. At times, these political concerns were reflective of parental demands for improvements in educational systems. The need to justify the expenditure of public funds on education and the desire to ensure ‘value for money' appeared to be a common theme in several countries and it was interesting to note the influence of the development of quality assurance systems in the business context.

4.2 Distinguishing between 'Quality' and 'Quality assurance'

During the discussions, the participants recognised that a distinction had to be made between the notion of quality, which might be defined as the extent to which schools realised the goals to which they were expected to aspire, while quality assurance might be considered as the process which ensured that these goals were met. For example, all of the systems aspired to the provision of a broad and balanced educational process in which each child would develop to his/her full potential, and in many cases, curricular documents defined the range of knowledge, skills and attitudes which would be encapsulated within this process. The degree to which these goals are achieved may be regarded as the level of quality reached by the school. On the other hand the quality assurance mechanisms are those processes which promote and ensure that the level of quality is as high as possible. However, the discussions of the participants demonstrated that, in practice, it is hard to separate the notion of quality and quality assurance because the means by which quality is measured can have a profound impact upon the work of schools and the actual goals to which they work. For example, a curriculum which aspires to provide a broad and balanced curriculum may not be realised if the quality assurance mechanism concentrates solely on pupil performance in written examinations which test only a limited range of knowledge or skills. It was for such reasons that arguments were made in favour of a multiplicity of quality assurance mechanisms.

4.3 Defining and ensuring quality in practice

A most interesting aspect of the discussion concerned the manner in which an educational system might define quality in practice. Here, a wide variety of approaches seem to have been adopted within the educational systems under review. In some, such as in Norway, Sweden and Finland, the quality which was sought in the school system was defined in aspirational terms, closely linked to the desire to provide a rounded and balanced education for each child. In practice, therefore, the quality which was desired in the system was outlined through the articulation of broad educational aims. In sharp contrast to this, the United Kingdom presented a model in which the goals to be achieved by schools were tightly defined using a prescriptive curriculum and published criteria by which schools and pupil outcomes were rigorously measured. Most countries lay between these extremes: in Spain, Germany and in Ireland at primary level, the expectations of the system were defined to a greater or lesser degree in curricula formulated at national, state or regional level, and each of these countries had some form of inspection system which used defined criteria to evaluate the work of the system. These criteria placed an emphasis on both the output of the school and the educational process, and in all cases these inspectorates acted in an advisory capacity as well as performing an evaluation role. At secondary level and particularly in more academically orientated upper secondary schools leading to university or other third level education, the use of examinations became more pronounced in all of the countries, so that in practice the dominant measure of quality became the general level of examination results achieved by pupils. This was true of upper secondary schools in Finland, Sweden, Norway and Germany and at both the upper and lower secondary levels in Spain and Ireland. In the case of the United Kingdom and France, the results of students in nationally set examinations or tests was an important criteria of school quality at primary level and throughout secondary education. This approach was also evident at all levels in the Hungarian system where a free-market approach had been adopted: parents were free to select the school of their choice but access to the desired school at kindergarten, primary or secondary level was frequently determined by success in examinations or assessment tests. Consequently, the quality of a school was defined in practice as the extent to which its pupils were promoted to desirable schools at the succeeding level within the educational system.  

4.4 Quality assurance and quality control

An earlier section of this report has explained that for working purposes the distinction between quality assurance and quality control was set aside during much of the work of the participants. However, consideration of this issue was obviously important in the work of the group and it was an issue to which participants returned on a number of occasions. The participants found it valuable to view quality assurance and quality control as two ends of a continuous spectrum.

4.4.1 Quality control

Mechanisms which encouraged quality control tended to be external to the school. The use of school inspectors in Germany, Spain, France, Ireland and the United Kingdom might be regarded as examples of such control mechanisms as would the reliance on examination and assessment results in Hungary, in the United Kingdom, and at upper secondary level in many of the countries. The discussions revealed that external quality control mechanisms had some advantages: valuable data on the performance of the schools within the system could be collected and this need not necessarily be limited to information on outputs as the process may also be evaluated through the use of trained inspectors. Participants also reported that educationalists and maintainers of schools believed that in many cases, the external quality control mechanism could act as a stimulus to principals and teachers in the improvement of their work. Nevertheless, quality control mechanisms had their limitations: some participants reported that improvements in quality most often occurred in the period immediately prior to the inspection and that a culture of self-improvement was not always created.

4.4.2 Quality assurance

Quality assurance approaches were, of their very nature, internal to the school organisation and were concerned with self-development rather than control. They aimed to foster a culture in which teachers and other members of the school community reflected on current practice, identified strengths and weaknesses, planned for improvement, implemented these plans and continuously reviewed progress. This was the model which had been adopted in the Hungarian pilot project in school quality assurance and variations of this approach had been used or were in pilot test phase in most of the participating countries. In the case of virtually all of the countries, schools were formally or legally required to have school plans but in practice these had generally defined the programme of the school and had not contained specific quality assurance mechanisms or quality improvement procedures. In some of the countries, the lack of a developmental aspect to the plans had prompted the instigation of pilot projects to encourage schools to adopt a self-development approach: such projects were currently underway in Spain, Ireland, France, Germany, Finland, Norway and Sweden. The advantages cited for the adoption of quality assurance mechanisms within these pilot projects included the assertion that the creation of a culture of self-improvement in schools was likely to be more effective than any external and periodic assessment procedure. In Hungary, Spain, France, Germany, Norway and Ireland advisors had been appointed to help schools in developing this self-assessment culture. Sometimes, the term ‘critical friend' was used to describe their role and it was hoped that their existence would bring to the process some of the advantages of external assessment while at the same time ensuring that the culture of school improvement became indigenous to the school.

4.5 Multifaceted nature of quality assurance

One of the most striking impressions from the discussions was the general agreement that the assurance of quality in schools must be achieved using a multifaceted approach.

4.5.1 Danger of over-reliance on one approach to quality assurance

An over reliance on one mechanism was seen as limited in effectiveness and possibly very damaging to the educational system and its overall goals of a generally high standard of education for the full cohort of pupils. The experience of Hungary presented compelling evidence that an over-reliance on quality through examination/test results had a narrowing effect on the curriculum and a tendency to confine high performance to a selected elite of students. In other cases, a heavy or exclusive reliance on school self-evaluation was also seen to have its weaknesses: for example, questions were raised as to the degree to which poorly performing schools could be identified and changed in Sweden and Finland where inspection systems had been dispensed with, or in Norway where school inspection had never been used.

4.5.2 Need to value both educational process as well outcomes

Participants also pointed to the need for educational systems and their quality assurance mechanisms to value both the teaching/learning process and well as the outcomes of that process. In particular, participants mentioned that in an information based society the ability of students to develop skills and adaptability was more important than the accumulation of knowledge. The success of schools in achieving the latter was relatively easy to measure but the development of students' skills was less readily assessed using conventional test instruments. The use of approaches which assessed and helped to improve the development of students' skills would be vital for the improvement of educational standards.

4.5.3 Need to use both quality control and quality assurance mechanisms

Participants were impressed with the value of using both control quality control and quality assurance mechanisms. In some systems, such as the Spanish, Irish and German models a national or state inspectorate attempted to provide both advice and help as in an assurance model and evaluation as in a quality control model. This was seen to be one of the strengths of the inspectorates in these countries. It was interesting to note that in Ireland this model was in use at primary level and was being extended to secondary level at present (where it was intended to temper a previously exclusive reliance on examination results as a mechanism of quality control). The strict separation of school inspection from the provision of advice and support to schools in the United Kingdom (where inspection was conducted under the auspices of a national inspectorate while advice was provided by local education authority advisers) represented a contrasting model for the delivery of a quality assurance and quality control mechanism on a nation-wide basis. It was also interesting for participants to note that the pilot projects in school development planning in Spain and Ireland were introducing the role of advisor in addition to that of the inspector.

4.5.4 Need to ensure ownership of the process

Participants were also impressesed with the importance of genuine involvement of all of the educational partners in the quality assurance process. Mechanisms whereby the views of teachers, principals, boards of management, parents and pupils as well as outside opinions (such as those from an external facilitator, critical friend or inspector and employers of former students) were taken into account during the evaluation of the school were said to be essential if a comprehensive picture of the school was to be formed. In systems where school development planning or self evaluation was in use or in trial phase, the involvement of all members of the school community was seen as a prerequisite. Two groups were focussed upon in the discussion of the difficulties that may be encountered in ensuring the involvement of the full school community in quality assurance. A number of examples of pupil self-assessment and pupil assessment of schools were noted. Su rveys of pupil satisfaction were used at times in the self-evaluation of schools in Norway, Sweden and Finland, for example. During the visit to one of the Hungarian schools, an interesting example of pupil self-assessment was observed: pupils compiled portfolios containing statements of their personal goals for the year as well as samples of their work and the reactions of their teacher and parents as the year progressed. The school authorities argued convincingly that this form of assessment provided a much better indicator of pupil achievement (and indirectly of school effectiveness) than the exclusive on testing which was common in most Hungarian schools. While teachers may initially find the notion of pupil evaluation of the school as somewhat threatening, the experience of participants suggested that pupils were very fair in their assessments and it is interesting to note that this impression is confirmed in the work of experts in school evaluation such as John MacBeath. The inclusion of parents in the process can pose difficulties, particularly if families lack the cultural capital that is needed to engage confidently with the school. Several participants mentioned initiatives which had been taken in their systems to encourage home-school linkages, particularly in areas of educational disadvantage. Efforts had been made within these programmes to equip parents with the skills necessary for them to engage with teachers and the school about their children's education and to contribute in a worthwhile manner to review of the school and school planning. The use of pupil portfolio assessment, as in the Hungarian example cited above, was deliberatley designed to foster on-going links between the school and parents (and grandparents). Participants felt that the degree to which schools managed to achieve this empowerment of parents and an inclusive approach to self-review and improvement should itself be a criterion of quality in assessing work of the school.

4.5.5 Value of mechanisms external to the school

Participants noted that quality assurance in schools could be achieved or at least significantly affected by a range of mechanisms that were not directly related to schools. Three of these mechanisms were discussed: the importance of quality teacher education; the role played by the adequate definition of the curriculum and support for its implementation; and finally, the use of information from sampling and other sources to analyse the performance of the system as a whole or sections of it. The arrangements regarding teacher education varied: the German participants spoke of the importance attached to initial and in-service teacher training in their systems, while in some countries entry to and removal from the teaching profession was controlled by a teaching council made up largely of teachers. Proposals had been made in Ireland to move to this latter model but currently, the national or state authorities in Germany, Ireland, Spain and ... had a direct role in the accreditation of teacher qualifications or probationary service. Within all of the countries, the national, state or regional educational authorities sought to define the knowledge, skills and attitudes to be acquired by pupils through the provision of curricular guidelines or regulations. The level of detail to which the curriculum was prescribed varied: in France a single national curriculum provided a very detailed and prescriptive definition of the courses to be studied, in Spain, the United Kingdom and Ireland curricula were centrally provided but these were adaptable at local school level. In many other countries, such the German states, Sweden, Norway, Finland and Hungary schools were largely responsible for defining the school curriculum but the curriculum contents had to conform to parameters outlined in framework curricula published by the national, state or local authorities. The very useful role which could be played by careful analysis of sampling data from pupil assessments completed in national or state init iatives impressed the participants greatly. Outstanding examples of this were presented by Mr Peter Vári of the Hungarian National Institute of Education. Data generated from surveys of reading competency and the international TIMMS research project on mathematics and science achievement had been analysed by him in a very innovative manner. The aggregate average scores from schools had been plotted on the basis of school type and location: for example schools in Budapest, in county capitals, in provincial towns and in rural villages, demonstrating quite clearly the lower standards of achievement which pertained in smaller and more isolated schools. Longitudinal studies had demonstrated that the discrepancy was growing rather than declining in recent years. Information on the relative performance of individual named schools was not published in this research but information on the average performance and the range of scores obtained by pupils was made available to each school, enabling it to compare its performance relative to other schools. Other analyses drawing on Hungarian data and data from other countries in the TIMMS project, demonstrated the stages in the educational system where most growth had occurred in pupil performance, and where pupil performance declined or was maintained following pupils exit from the compulsory education system. In this regard, findings in the Hungarian system demonstrating that examinations for entry to post-primary schools prompted a spurt in pupil performance which was not maintained in second level education was most telling. So, too, was the finding that early-age high performance in mathematics in countries such as Hungary, the Czech Republic, and the United States did not generally lead to above average competence in maths or science in school leavers whereas some countries having below average student performance during the period of compulsory education had higher than average results among school leavers. Participants in the study visit were most impressed by the potential of such analyses, not only to provide information for policy formers, but also to influence the methodologies used in schools, and where school data was supplied in a meaningful way to schools, to prompt school improvement and development. However for such national sampling to be effective it was clear that public awareness of the research and the existence of public and professional fora at which it could be discussed were essential.

4.6 Access to the outcomes of school evaluation within the quality assurance process

Participants from several countries spoke of the need for all of the educational partners to have access to the outcomes of the evaluation process if these outcomes were to act as an effective catalyst for improvement. In a sense, evaluation that took place without adequate dissemination of the results could fail to deliver the affirmation of good practice and the stimulus for improvement which are necessary if evaluation is to lead to quality assurance. The extent of access to the evaluation results varied considerably, however. In many of the countries, reports on schools were intended primarily for the school staff and the board of governors/management, in others some information might be published in local newspapers and other media (for example in Sweden and Norway), in Finland aggregated data was disseminated on a regional basis, while in the United Kingdom full reports on all inspected schools was publicly available on the internet and in each school. Concerns were expressed in the study group about the detrimental effects of the publication of selective indicators or results, particularly in their effect on parental perceptions of schools. Some participants felt that where statistical data on pupil performance was published on a school basis (either by inspectorates or the schools themselves), this data became the sole or dominant measure of quality used by parents in the selection of schools rather than the valued added dimension of the school's performance. Other participants cited instances where because of the lack of freely available statistical data, parents selected schools using limited information, often information of an informal nature and, at times, little more than rumour and propaganda about the schools in question. The significance of this data or the lack of it, presupposes that parents have the right to select a school for their children: some participants pointed out that effective choice was limited for many parents either because of geography or lack of ability to pay transport costs. The most interesting example in this regard was Norway where, in general, parents had no choice in the selection of school for their children during the period of compulsory education, an arrangement which was justified on grounds of social egalitarianism.

4.7 Contextual factors

All participants agreed that in discussing the quality assurance mechanisms that were in place in any educational system or those which might be profitably introduced to a school system, consideration had to be given to a number of contextual factors. Among those which were raised in discussions were:

4.7.1 The historical context

The influence of the historical development of the school system was a factor that was highlighted by a number of the participants as they described the quality assurance mechanisms currently operating in their countries. Some systems, such as those in Britain, Germany, France, Spain and Ireland (at primary level) had long-established traditions of external evaluation of schools through professional school inspectorates. Unsurprisingly, in these countries, the educational community was more open to the development of mechanisms involving external evaluation and there was little pressure for the elimination of such evaluation. Indeed, in some countries, such as Britain, it had been possible to extend the degree of external evaluation considerably in recent years. In contrast, Norway had no tradition of a school inspectorate and there was no discernible pressure for the introduction of such a system. The influence of the historical context could also be significant in impeding changes within quality assurance mechanisms. In France, for example, the introduction of school self-assessment and development models was impeded by the deeply ingrained desire of teachers and other professionals to retain a strongly centralised school system in which the autonomy of schools was limited and the need for teamwork and self-assessment was less important. Nor did the long-standing existence of an external inspectorate necessarily guarantee the acceptance of external quality assurance mechanisms: Hungary presented an extreme example of this, as the association of the inspectorate with the former communist regime made the use of a centralised inspection process unacceptable to schools and teachers. However, it was also noted that in both Sweden and Finland external inspection systems had been discontinued in the last quarter of the twentieth century.

4.7.2 The political context

The experience of almost all the countries demonstrated that the political context was a significant factor in determining not only the existence of quality assurance mechanisms but also the methodologies deployed. The political reasons that had stimulated or influenced the approach adopted in the various systems varied considerably and could result in widely differing outcomes. For example, a political desire to promote egalitarianism in the Norwegian, Finnish and Swedish societies in the final quarter of the twentieth century had lead to the introduction of comprehensive school systems and, in the case of Finland and Sweden, the simultaneous removal of inspection systems and their replacement with educational advisors. In Germany, a great emphasis was seen to be placed on ensuring the quality of the school system through regulation and delineation of duties/responsibilities because there was a belief within the political culture that compliance with regulation would be high. Moreover, Germany presented a good example of political considerations influencing the locus of decision-making about the quality assurance system: the strongly decentralised political culture ensured that most decisions concerning quality assurance were made at the level of the Bundesländer (state) or at subsidiary levels. In contrast, while Ireland used an external inspectorate of schools as in Germany, the strongly centralising tendencies of the inherited political culture had resulted in a centralised school inspection system. The differing arrangements for the control, provision and inspection of schools which were noted in those areas of Spain having regional political autonomy and those without such status also emphasised the potential and real influences of the pol itical context on important aspects of the quality assurance process.

4.7.3 The political context and the influence of parental and public opinion

A relationship was recognised between the political context and the influence of parents. One of the starkest examples of this was noted in the British system where it had proved feasible for a national government to introduce a radically ‘high-stakes' school assessment process despite teacher objections because of the existence of considerable parental and public concern for standards in education. Moreover, parental satisfaction with the procedures ensured their continuation: although introduced under a right-wing political party, the system was left virtually unchanged by a new left-of-centre government. It was also noted that public pressure could also work against reform of quality assurance mechanisms. Participants were struck by the Hungarian experience in this regard. Here the main effective method of quality assurance was one based on parental choice of schools and the extensive use of selective entrance tests by schools. Visits to the schools and institutes demonstrated to the participants the degree to which this combination of factors had allowed the development and maintenance of high quality schools for high achieving pupils but had also resulted in an undesirable distortion of school curricula and the development of less effective schools in which most pupils were enrolled. The segment of the school population that gained access to the high quality schools achieved high standards but the progress of other pupils, and consequently of the school population as a whole, had been adversely effected. Perhaps the most surprising evidence presented to the participants was that which demonstrated that this quality assurance system has persisted virtually unscathed prior to, during and after the communist era, an outcome that was attributed to general political and parental satisfaction with the status quo.

5. Lessons from the study visit

Participants on the study visit spoke enthusiastically of the way in which the visit had enriched their understanding of quality assurance processes in schools and had facilitated the exchange of ideas and perspectives among the members of the group. The following were among the practical ideas and concepts which participants identified as applicable in their own countries:

• The importance of clarifying the way in which quality in the educational system is defined in any school system
• The close inter-relationship between the mechanisms used to measure or assure quality and the effective goals to which schools will operate
• The value of promoting quality assurance approaches in tandem with quality control approaches rather than an exclusive reliance on one or other of these methods
• The need to use a range of approaches in ensuring that quality is assured within the school system
• The important impact that quality assurance approaches can make to school improvement
• A knowledge of techniques to encourage effective quality assurance in schools, including the use of questionaires among all members of the school community, test data, the advice of a ‘critical friend', school planning, etc
• The importance of giving a voice to pupils in the quality assurance process and techniques which can facilitate this
• The contribution that mechansims external to the school can make to school quality assurance, including the use of national sampling data, framework curricula, and assuring quality in teacher education
• An awareness of some of the problems associated with school quality assurance
• An awareness that contextual factors must influence the adoption and use of any process of school quality assurance.

 

Appendix 1: Programme of study visit

Sunday, 19 March 2000-04-18

	Arrival and check-in until 18.00
18.00-18.30	Welcome Ms Gabriella Mózer, Course Organiser
19.30-22.00	Introduction of members of study visit, election of rapporteur

  Monday, 20 March 2000

09.00-09.30	Opening session Ms Gabriella Mózer, Course Organiser
09.30-10.45	Lecture and discussion Quality improvement programme in school education: Institutional models Ms Katalin Herneczki, Head of Bureau, Comenius 2000 Programme Ms Andrea Lipták, Quality Expert, Comenius 2000 Programme
11.15-12.00	Lecture and discussion Hungarian quality award in the public sector Mr Miklós Csiszár, Hungarian, Quality Development Centre
13.30-15.00	Lecture and discussionThe background to quality assurance in HungaryMr Gábor Halász
20.00-21.30	Presentations and discussions on participants' countries

 

Tuesday, 21 March 2000

09.00-13.00	Visit to secondary school, observation of lessons, discussions with staff Mr Tas Szebedi, Principal
14.00-15.30	Lecture and discussion The structure and transition of the Hungarian school systemMr Károly Pála, Budapest Institute of Educational Services
15.30-19.00	Sightseeing in Budapest

 

Wednesday, 22 March 2000

09.00-12.30	Round table discussionsComparing quality assurance systems in participants' countriesMr Attila Horváth, Mediator
14.30-16.00	Quality assurance from the maintainers' perspectivesMs Judit Csapó
19.00	Cultural programme

Thursday, 23 March 2000

10.00-13.30	Visit to Lauder Javne School: observation of classes, discussions with staffDr Anna Szeszler, Principal
14.30-16.00	The Self-Developing School Project of the SOROS FoundationMs Eva Balázs, Hungarian National Institute of Public Education
16.00-17.00	Press conferenceMs Gabriella Mózer,Course OrganiserMs Katalin Herneczki, Head of Bureau, Comenius 2000 ProgrammeAnd Group Rapporteur on behalf of Participants
19.00	Cultural programme

 

Friday, 24 March 2000

09.00-11.00	Lecture and discussionNational and international surveys of student performanceDr Péter Vári, Hungarian National Institute of Public Education
11.00-12.00	Lecture and discussionMinistry of Education
12.00-15.00	Preparation of group report
15.00-22.00	Sightseeing followed by Conference Dinner

 

Saturday, 25 March 2000

08.00-09.00

Departure

 

Appendix 2: List of participants

Participants

Constanza Falco	Las Palmas de Gran Canaria, Spain
Kjell Halvorsen	Larvik, Norway
Klaus Hanssen	Potsdam, German
Elke Hildebrandt	Badhersfeld, Germany
Harold Hislop	Dublin, Ireland
David Humphrey	Derbyshire, United Kingdom
Hedi Niehl	Trier, Germany
Leif Olin	Helsingborg, Sweden
Nelly Portela	Paris, France
Frank Saether	Molde, Norway
Paavo Tarkiaine	Lieto, Finland
Rapporteur
Harold Hislop	Dublin, Ireland