Proceedings of the AQ 97 Conference
Winchester, 2 December 1997

The purposes of self-evaluation: reconciling the tension between accountability and quality improvement

Jethro Newton
Head of the Academic Office
North East Wales Institute of Higher Education
Wrexham
[email protected]

The purposes of self-evaluation

Progress made at NEWI in developing self-evaluation at subject, programme and institutional levels, in response to the challenge of external quality audit and quality assessment, has been outlined elsewhere [9,10,11]. Underpinning this progress has been an attempt to encourage involvement and teamwork, and a focus on action planning.

Brennan, Frazer and Williams [1] have developed guidelines to assist departments and institutions in clarifying and developing their own approach to self-evaluation. They point to the various phases and stages involved and remind us that self-evaluation or self-assessment can be undertaken for a variety of purposes and, as has been the case at NEWI, there may be both external and internal drivers. A cautionary note is added when they suggest that such guidelines "need to be interpreted in the light of local circumstances and constraints" [1: Preface]. They also draw attention to some of the dangers and pitfalls of self-evaluation. Firstly, they argue, if motivated exclusively by public-relations concerns, self-evaluation can develop into self-deception. Secondly, where evaluation is externally imposed, this may encourage compliance or concealment. Thirdly, "there are dangers of producing an evaluative process which is paper-driven and bureaucratic" [1: p.5]. To this we might add that there is a danger that use of any quality procedures and tools can result in 'ritualism' and 'tokenism', with participants primarily engaged in learning and following 'the rules of the game'.

For Brennan and his colleagues, self-evaluation is primarily an academic rather than an administrative task:

Self-evaluation is about whether educational objectives are being achieved and whether current practice can be improved upon.

[1: p.5]

They argue that there are two underlying purposes of self-evaluation, 'being accountable' and 'seeking improvement'.

By identifying strengths and weaknesses, the institution, or department or course/programme team, are able to see themselves as they 'really are', and this is a step towards being accountable to a range of internal and external authorities; and by building on strengths, and taking remedial action on weaknesses, improvements are intended to ensue.

[1: p.5]

The challenge, they argue, is to achieve clarity and agreement as to the balance which is sought between accountability and improvement. This has become an increasingly familiar theme in the 'quality in education' debate [2,12].

In the context of this case study, two questions arise in relation to the 'tension' or balance' between accountability and improvement. Firstly, how far has the 'accountability challenge' been met by NEWI, at both subject/programme and institutional levels? Secondly, in seeking to strike an appropriate balance between accountability and improvement, how far can the prevailing situation at NEWI be described as 'improvement-led' in the sense which Harvey has described?

Control of the quality improvement process is by those who can really implement changes and make an impact (staff and students).

[2: p.68]

Identifying internal and external performance measures

One means of addressing such questions is to identify performance measures which can then be used to assess the effectiveness of the implementation of the quality assurance system and procedures at NEWI. Various possible internal and external reference points, for use in measuring performance, suggest themselves. These include, external 'quality reports' of monitoring bodies, professional bodies, external examiners or employers. Internal measures, quantitative and qualitative, include internal quality monitoring reports, student feedback, validation and review event reports, staff views and feedback and so on.

For present purposes, we shall briefly consider evidence from three sources, two external and one internal. The two external measures are: the HEFCW reports on assessments of the quality of teaching and learning at NEWI between 1993 and 1996 [5,6,7]; and the HEQC report on the institutional audit of NEWI's quality assurance systems in May 1996 [8]. The reports contain comments and profiles derived from the professional judgement of teams of academic peers. An internal measure of performance is drawn from the results of a questionnaire distributed to 109 members of staff (academic staff, academic support staff and academic managers) at NEWI during January 1997. The survey population included 74% of all full-time academic staff, together with a sample of staff in central academic service areas.

External performance measures

When the HEQC institutional audit report, along with its conclusions and recommendations, is compared with NEWI's own self-assessment, it becomes evident that NEWI scored well on the 'index of surprise'. Strengths and shortcomings requiring action, as identified in the external monitoring report, more or less matched the profile set out in the institutional self-study. There were no 'unpleasant surprises', though there were some welcome ones. The self-assessment was proved accurate.

The positive outcome of the institutional audit, in terms of satisfying external demands for accountability through having appropriate quality systems, is formally confirmed in the HEQC Quality Audit Report on NEWI in which it is stated that:

From its review of the Institute's quality assurance activities, the audit team would wish to commend in particular ... the success with which the developing quality assurance systems have been embedded in the Institute's processes and become fully owned by the staff as a whole.

[8: p.27]

In turning to the second external reference point, external assessment by HEFCW at subject/programme level, we can note that successive reports on the quality of provision commented favourably on positive features of the quality system and its implementation. For example, the report on Art and Design indicated that:

There is clear evidence that input from course team meetings in the department, which are held every two weeks, is used to detect shortcomings and then to improve the quality of provision. The use of the quality systems to improve the student learning experience is a major strength.

[6: p.7]

An earlier report, on Environmental Studies provision, is similarly positive in its comments on the use of action planning:

The self-assessment report identifies a number of strengths and weaknesses, and an action plan to resolve the latter is also included. There is evidence that concerted effort is being made to rectify weaknesses.

[5: p.7]

In the next example, drawn from the report on Electronic and Electrical Engineering provision [7: p.7], the assessment team commends the involvement of student representatives in quality assurance procedures and again points to 'ownership':

There is a continuing process of self-evaluation in place to rectify shortcomings identified from a variety of sources, within a well established institutional infrastructure. ... The quality system has been implemented effectively by the course team .., with staff taking ownership of the system.

[7: p.7]

Again the demands of external accountability appear to have been satisfied. Moreover, it can be argued that there is evidence, in the context of 'matching the external test', that progress has been made towards what Harvey [2: p.40] refers to as:

The development of operational teams, who take responsibility for quality, set their own agendas for action, report their intentions, actions and consequent achievements.

Internal performnace measures

The questionnaire survey was designed as part of the present author's ongoing research into the development and implementation of quality assurance systems in higher education. It is being supported by in-depth follow-up interviews with key individuals and focus groups at NEWI. Detailed results and findings will be reported in forthcoming work. Some aspects of interim findings are reported here on a selective basis, for illustrative purposes.

The survey was structured into three parts. The first part contained twenty-three questions with respondents invited to place their responses on to a six-point Likert-style scale as shown below in Table 1. The questions were focused on the aims, purposes and mechanics of NEWI's quality system; factors which affect implementation; outcomes and improvements associated with the quality procedures and system; and the effects of changes in the higher education system. The second part invited respondents to indicate, from a list of sixteen items, which of the items best described the achievements of the system. Part three invited any additional comments. 

The underlying theme of this research connects well with Brennan, Frazer and Williams's description [1] of the two purposes of self-evaluative approaches in quality systems - 'being accountable' and 'seeking improvement' - since the questionnaire and interview schedules have been designed to explore the extent to which NEWI's quality assurance system has met an essential requirement of an effective quality assurance system: that of reconciling the demands for accountability (internal and external) and quality improvement. Table 1 sets out responses to five questions covering areas which are of particular interest for present purposes. These relate to the 'aims and purposes' of the quality system and the 'outcomes and improvements' associated with it. Similarly, Table 2 identifies responses in the sixteen areas where respondents were invited to describe what, in their view, were the achievements of the quality system.

Results

From Table 1, we can deduce from items 1 and 2 that there is an overwhelming consensus of agreement that both internal and external requirements for accountability have been satisfied. Within this, 81% either 'agree' or 'strongly agree' with respect to external expectations while the equivalent figure for internal expectations is 71%. When the category 'tend to agree' is included the figures are 90% and 92% respectively. However, when we turn to the issue of improvement for students and for staff (items 2 and 3), there is a greater spread of opinion. In the case of improvements for staff there is more or less an equal split between those who express a degree of 'agreement' and those who tend towards some measure of 'disagreement'.

This picture is confirmed in the results set out in Table 2. Items 2 to 5 together cover the range of external monitoring bodies which set out requirements for NEWI as a higher education institution. The results in respect of HEQC institutional audit and HEFCW quality assessment are particularly striking since, respectively, 93% and 90% agree that the quality system has helped to ensure successful preparation and outcomes to monitoring by these bodies. It is also evident that most respondents view the quality system's 'technology' as having been successful, since the procedures for course monitoring, validation and review are viewed in terms of 'achievement' by 60%, 74% and 68% of respondents respectively. Almost a third register the view that the quality system and procedures can be described as 'just better bureaucracy'.

When we turn to items 7, 12, 13 and 16 in Table 2, there is even less of a tendency than in Table 1 to identify the quality system with the delivery of quality improvements and quality development. The general category of 'real quality development' is elected by only 26% of respondents, with a similarly low percentage (23%) acknowledging 'improvements for staff' as being a perceived achievement. This increases to 34% for the category 'improvements for students'.

Discussion

Although there is some acknowledgement, by a proportion of staff, of the capabilities of the quality system in delivering 'improvement' and 'development', support for this view is relatively low. As noted already, external evidence portrayed a more positive picture. From this it seems that there is a degree of imbalance between the perceived ability to meet the requirements of accountability on the one hand, and of achieving quality improvement on the other. Within this, an apparent 'gap' between the external and internal measures needs to be explained. As noted, one external source cites use of the quality procedures to improve the student learning experience as a 'major strength'; yet internal evidence drawn from the views of staff does not fully support this assessment. It may point to the possibility, referred to earlier, that staff have become well versed in 'the rules of the game' and that, in effect, procedures are used primarily to satisfy external requirements. Or, it could signal that if a quality system is used as a matter of routine practice then staff may not ascribe improvements to its use per se. Another possibility is that there may be a reluctance, on the part of some staff, to admit that 'quality', which has been perceived by many in UK higher education as an extra burden, can have a positive effect.

There is a further possible avenue of explanation. Neither of the external quality monitoring bodies penetrates what might be termed the wider aspects of quality in institutions, or 'total quality' issues. While some attention is paid to library and student support services, there is no real attempt to consider, holistically, the full range of services and management processes which impact upon the student and staff learning and working environments in the widest sense.

Evidence to support this line of explanation for the 'gap' between internal and external measures of success, may be deduced from the responses to item 5 in Table 1. There it can be seen that 94% of respondents regard it as necessary for NEWI's quality assurance system to be extended to cover all service areas as well as academic units. Of these, 65% 'strongly agree'. In a similar vein, in Table 2, item 16, only 15% indicate support for the view that the quality system has delivered an 'improved capability to address quality issues in non-academic service areas'. This points towards the possible significance for staff of both the 'total quality' dimension, and of wider issues around quality management processes. It may also indicate the significance, for staff, of resource-related issues. Amongst the commonest factors which can bring a quality system into disrepute are that there is no effective means of 'closing the loop', or that some areas of activity or service provision, which may impact upon the totality of the student and staff experience, are excluded from, or are not given sufficient prominence in, the formal system for managing and reporting on quality.

This assessment of the wider dimensions of quality also needs to be viewed against the context, described earlier, of a sharply reducing unit of resource, resulting in higher student-staff ratios, less funding for equipment and so on. A shift in emphasis from a 'resource-led' culture to a 'problem-solving', 'improvement-led' culture makes considerable demands on staff. Such pressures may be compounded if the quality assurance system is not perceived as providing a sufficient basis for ensuring transparency in quality management across all of an institutions' activities.

Conclusions

In this paper we have noted that the movement towards a mass higher education system in Wales has been accompanied, as elsewhere in the UK, by increasing demands for accountability. In terms of quality assurance in higher education, this has been accompanied by an evolving emphasis, at sector, institutional and departmental levels, on self-evaluation and self-assessment. At NEWI, changes in the external monitoring environment have been responded to positively and the Institute's obligations in the areas of both external and internal accountability have been fully addressed. NEWI has sought to align its internal quality assurance arrangements with external arrangements and the means of achieving this have been described. Self-evaluation and teamwork have been encouraged in an attempt to achieve ownership of systems and improvements in quality.

It has been shown that NEWI has performed well in the demanding 'fitness for purpose' tests as represented by HEQC quality audit and HEFCW quality assessment. The views of successive teams of external peers have been favourable and staff have exhibited an increasingly high level of confidence in their own and the institution's ability to satisfy the demanding external criteria. This has been supported by evidence drawn from ongoing research at NEWI which has indicated that, on the basis of both external and internal 'performance measures', requirements for accountability are being met. However, although external monitoring reports confirm that systems are embedded and owned, that self-evaluation and action planning are used routinely, and that the quality system is being used by staff for the purposes of improving the students' learning experience, the internal evidence regarding 'quality improvement' and 'quality development' indicates a degree of scepticism on the part of staff.

While the evidence contains some acknowledgement of 'improvement' and 'development' by a proportion of staff, support for this view is generally low. This is in marked contrast to the external evidence. What this appears to show is an imbalance between the need to demonstrate an ability to meet the requirements of accountability on the one hand, and the requirements of quality improvement on the other. Despite attempts by the Institute to develop an 'improvement-led' approach, it seems that the strengths of the quality system tend to be interpreted, internally at least, in terms of an 'accountability-led' approach. 

There is a challenge here for national higher education systems, for individual institutions and for external monitoring agencies. That challenge can best be expressed in terms of the need to reconcile the needs and requirements associated with meeting demands for accountability, with the need to demonstrate achievement of quality improvements in the student and staff experience, and in their learning and working environments. This is particularly timely and pertinent in the UK given the recent establishment of a Quality Assurance Agency for Higher Education, with responsibility for both quality audit and quality assessment.

Acknowledgement

I should like to thank Dr Jon Owen, Quality Enhancement Unit, University of Newcastle upon Tyne, for his helpful comments on an earlier draft of this paper.

References

1. J. Brennan, M. Frazer and R. Williams, Guidelines on self-evaluation, London, Open University Validation Services, 1995.

2. L. Harvey, "Continuous quality improvement: a system-wide view of quality in higher education", in University-wide Change, Staff and Curriculum Development, P.T. Knight (ed.), Birmingham, SEDA Paper 83, 1994.

3. L. Harvey, Quality assurance systems, TQM and the new collegialism, Birmingham, QHE Project, 1995.

4. HEFCW, Quality assessment report on environmental studies provision at the North East Wales Institute, December 1994, Higher Education Funding Council for Wales, 1995.

5. HEFCW, Quality assessment report on art and design provision at the North East Wales Institute, November 1995, Higher Education Funding Council for Wales, 1996.

6. HEFCW, Quality assessment report on electrical and electronic engineering provision at the North East Wales Institute, March 1996, Higher Education Funding Council for Wales, 1996.

7. HEQC, North East Wales Institute of Higher Education, quality audit report, London, Higher Education Quality Council, 1996.

8. J. Newton, "Opportunities for partnership in quality improvement: responding to the challenge of teaching quality assessment in Wales", Quality in Higher Education, 3,1, pp.37-50, 1997.

9. J. Newton, "Collaboration and partnership in the development and implementation of a quality assurance system," presented at the Fourth Biennial International Conference of the International Network for Quality Assurance Agencies in Higher Education, Berg-en-Dal, South Africa, 1997.

10. J. Newton, "Institutional self-evaluation: responding to the challenge of external quality monitoring in a higher education institution in Wales", in A.H. Strydom, L. Lategan and A. Muller, Enhancing Institutional Self-evaluation in the South African University System, Bloemfontein, South Africa, University of the Free State Press, 1997.

11. A.I. Vroesijenstijn, Improvement and Accountability: Navigating between Scylla and Charybdis, London, Jessica Kingsley, HE Policy Series 30, 1995.

© Jethro Newton 1997 Published by Information Geometers Ltd

Back to the AQ 97 contents list