Australian Journal of Educational Technology
1990, 6(2), 92-98. 		AJET 6

Validation: Cost effective external evaluation

Peter Brown and Michael Hickey
Department of Defence (Navy)
The purpose of any training system or organisation is to produce a trained person who can successfully perform specific tasks in the workplace. If these tasks cannot be performed to a required standard, then the training has failed. In fact, the quality control process of ensuring that a particular course meets the job requirements, by investigating the trained person on the job, is arguably the most important phase in a training system. After all, if a person performs well in the workplace there is probably nothing significantly wrong with the training course, and any necessary corrective measures will be concerned more with the efficiency of the training.

Despite the importance of this process, external evaluation (or 'validation') has been neglected in the past and only now is gaining the recognition it deserves. Current training legislation (NSW, 1989) attempts to ensure quality training in a cooperative industrial context. For the future, much will depend on the emphasis policy makers place on quality control, and in particular on validation (Bright, 1990). This paper describes the validation process and demonstrates that a relatively simple and cost-effective validation unit can be of considerable benefit to any organisation involved in training.

The validation component of a training system

It is generally recognised that a systems approach to training provides the most suitable framework to effectively manage and implement training (Dick and Carey, 1978). The system model illustrated in Figure 1 has been successfully used in the Royal Australian Navy since 1971. A job is analysed, a course designed and conducted and then the course graduates perform the instructed tasks, in the job. Quality control of this 'training loop' is essential and the control (evaluation and validation) is illustrated in Figure 1. Evaluation is primarily the internal evaluation process concerned with the efficiency of a training course. Evaluation instruments employed may include instructor evaluation reports, course reports and student feedback during a course (Romiszowski, 1981).

Validation of a particular course is the external evaluation process which concentrates on information concerning the trained person in the job. A number of terms have been used to describe this process and they include 'external evaluation', 'external validation', 'summative evaluation' and long term follow-up evaluation'.

Figure 1

Figure 1: The Royal Australian Navy Training System Model.

The differences between 'evaluation' and 'validation' are illustrated in Figure 2 where it can be seen that evaluation is concerned with the efficiency of a course or on-the-job training and validation investigates the effectiveness of these processes by looking at the trained person (either fully or partly trained) on the job.

Figure 2

Figure 2: Quality control of the training process.

Resources necessary to form a validation unit

In forming a validation unit, resource requirements will necessarily depend on the current and future organisational structure, manpower and budget limitations. If the organisation already includes experienced course evaluators. then they may be available to assist in all phases of the validation process. The size of the unit will depend on available manpower (either full or part time) and budget limitations will determine such decisions as whether manual or computer analysis should be performed. However, this paper is concerned with the formation of a relatively simple and cost-effective unit and will therefore assume that limited resources are available.

The use of assistance from other departments should be considered. This may include assistance from occupational analysts, training developers and training consultants. These personnel may already be available within an organisation and part-time assistance readily available. The required manpower can then be determined. It may be possible to allocate validation responsibilities to existing staff although it would obviously be preferable to have staff solely dedicated to the validation function. Costs may involve travel (the validation team will need to visit graduates in the workplace), duplicating and typing (production of questionnaires and reports) and analysis instruments (computers and optical mark readers). These costs, however, may be minimised if the validation team is co-located with the workforce and if there are only a few course graduates to interview and therefore only a few questionnaires to analyse. The number of course graduates should not influence the amount of validation effort. Only a handful of astronauts are trained each year, but who would question the necessity for careful validation of the associated training course?

Methodology

The results of resource analysis will determine the number of courses to be validated annually. As a guideline, three experienced personnel can evaluate ten courses per year. For larger organisations this will require a prioritised list of validation studies which should be given by the organisation's national training authority. At the macro planning level, using the guideline of three people validating ten courses per year, three courses can be investigated at any one time and it is advisable to have a project leader for each validation study.

Planning at the micro level is relatively straight forward. Most courses can be validated over a four month period although allowances should be made for particularly complicated courses, remote and varied work locations and the availability of interviewees. The four stages of a validation study are illustrated in Figure 3, and each stage takes approximately one month to complete, assuming three studies are being undertaken at any one time.

Questionnaire
Design		InterviewsDraft
Report		 Final
Report

Figure 3: The validation study processes

The instruments used to collect information include questionnaires, interviews, expert panel input, observations on the job, course results, evaluation reports and company reports (including safety reports). Data is collected, collated and analysed and the draft report is checked with interested parties before the final report is written. The report recommendations are followed-up after a pre-determined time.

Sampling

With three studies running simultaneously, a team of three can generally interview fifty people over a one-month period. Ideally, ex-trainees who completed the course from between three and twelve months previously should be considered. These job-holders would have completed the instructed tasks and yet not have forgotten details of the training course. In many cases it may not be necessary to sample from this population. If sampling is necessary, then at least thirty graduates should be chosen. Stratified sampling techniques may need to be employed where differences in status or environment are assumed to be significant. The full population should, however, be interviewed whenever possible (Gay, 1987).

Questionnaires

Two standard questionnaire types are used, and it is preferable that these are administered by a member of the validation unit. The first type is completed by the ex-trainees and is derived from the course training and enabling objectives which are rewritten in performance terms. The second type is derived from the training objectives and is administered to the immediate supervisors of the ex-trainees. Obviously, much time can be saved if the objectives are available (Briggs, 1977). The content of the draft questionnaires is evaluated by a panel of experts involved in the field of the subject course, trialled to determine the suitability of the wording and the time necessary to complete. Both questionnaires include closed and open-ended questions. The ex-trainee's questionnaire includes questions concerning the usefulness and importance of all instructed tasks and the considered effectiveness of instruction. The supervisor's questionnaire is a much shorter one and includes questions concerning the ex-trainees performance. Questionnaires suitable for optical mark reading are also available where large populations are involved, and programs such as 'CODAP', 'EXVAL' and 'MARVAL' will accommodate these.

Interviews

Whenever possible, questionnaires are administered by a member of the validation team. This affords the opportunity to undertake personal interviews which usually result in additional valuable information. Consequently, the validation team members should have the necessary interview and interpersonal skills. The rank or grade of the interviewer is important and the ideal status is that of an experienced supervisor. Knowledge of the course related job, although desirable, is not essential as question content will have been confirmed by an expert panel and training staff (Poulter, 1982).

A combination of informal and structured interviews produces the best results despite the difficulties in recording responses, and are particularly useful in discovering attitudes associated with the course. The use of audio tape, overt or covert, should be avoided. Telephone interviews may be used when personal interviews are not possible (Crockett, 1989).

Analysis of data

Computer software packages and optical mark readers are available to analyse validation data. For the analysis of the standard fifty questionnaires, however, collation may be performed manually. Manual collation is performed by enlarging a blank copy of the distributed questionnaire and the collected responses are collated on this copy.

Areas of concern are then identified. These would include those tasks that are not considered to be useful by more than half the respondents and those that have never been used by more than eighty per cent of the job holders. Interview data is analysed to form additional conclusions.

Reports

Draft reports include background information on the course and results indicating the areas of concern are given in annexes. Recommendations are restricted to those directly concerning the course undergoing validation. The draft is then checked with the original panel of experts and agreement reached whenever possible. This consultation is vital if any proposed changes to the course are to be successfully implemented. Conclusions include both the strengths and the weaknesses of the established course.

The final report is then produced in a high quality format and issued from the highest suitable authority to trainers and managers. The report, however, does not mark the conclusion of the validation study.

Follow up

The recommendations are followed up at between three and six months following production of the report. Changes will be made at any phase in the training loop (Figure 1). The process of implementing the changes is the cornerstone of the validation phase and yet is the weakest point in the structure. It is therefore vital that the report is issued by a high authority within the organisation and follow-up is vigorously pursued by that authority.

Measures of effectiveness

The effectiveness of a validation unit can be measured in three ways - report production, implementation of recommendations and cost saving.

Report production is an indicator of performance at the internal office management level and should meet a pre-determined annual output of quality reports. The number of recommendations implemented is the performance indicator at the second level (training department). Dollar savings as a result of creating a cost effective training environment is a third performance indicator and affects the organisation at the third and highest level.

Conclusions

Formation of a validation unit is feasible within any organisation concerned with training. This paper provides guidelines for the formation of a small cost - effective unit and describes the methods and resources necessary to set up and run such a unit.

The systematic approach to training is the most effective method of training for purpose. The validation phase provides the vital feedback from the job to ensure that effective training occurs. If quality control of training is to be given the attention it warrants, then validation will become an increasingly important function.

References

Briggs, L.J. (Ed) (1977). Instructional Design - Principles and Applications. Educational Technology Publication Inc; Englewood Cliffs, New Jersey, USA.

Bright, J.C. (1990). Pre-conditions for Flexible Training Systems in Education and Industry in Australia in the 1990s. Industry Training for the 1990s Conference Paper 4.

Crockett, R.A. (1989). An Introduction to Sample Surveys - A User's Guide. Australian Bureau of Statistics, Victoria, Australia.

Dick,W. and Carey, L. (1978). The Systematic Design of Instruction. Scott, Foresman and Company, Glenview, Illinois, USA.

Gay, L.R. (1987). Educational Research - Competencies for Analysis and Application. Merrill Publishing Company, Columbus, Ohio, USA.

NSW Parliament Legislative Assembly (1989). Industry and Commercial Training Bill. Government Printer Sydney, NSW, Australia.

Poulter, B (1982). Training and Development. CCH Australia Ltd, Sydney, NSW, Australia.

Romiszowski, A.J. (1981). Designing Instructional Systems. Kogan Page, London, UK.

Authors: Peter Brown and Michael Hickey are Training Validation Officers with
Department of Defence (Navy), PO Box 706, Darlinghurst NSW 2010.

Please cite as: Brown, P. and Hickey, M. (1990). Validation: Cost effective external evaluation. Australian Journal of Educational Technology, 6(2), 92-98. http://www.ascilite.org.au/ajet/ajet6/brown.html

[ AJET 6 ] [ AJET home ]
HTML Editor: Roger Atkinson [rjatkinson@bigpond.com]
This URL: http://www.ascilite.org.au/ajet/ajet6/brown.html Last revision: 23 Sep 2002.
Previous URL 13 Dec 1996 to 23 Sep 2002: http://cleo.murdoch.edu.au/gen/aset/ajet/ajet6/su90p92.html