| Australian Journal of Educational Technology 1992, 8(1), 1-12. |
AJET 8 |
Post course evaluation is equated with the 'Smile Sheet' in the minds of most trainers, and as such it is not held in high repute but rather regarded as something of a joke. In this article the author demonstrates how post course evaluation is an excellent means of obtaining in depth data about course participants' learning, skill development and behaviour change. The difference between the 'Smile Sheet' and effective post course evaluation is in the systematic approach to the design of the questions. A methodology to guide this design is outlined, and sample questions from post course evaluations are given.
This article takes up the challenge to prove her wrong! Many trainers prepare post course evaluation forms as a matter of habit - after all, evaluation is important, we need some statistics to give management if they ask us how the course is going, and since most trainers do not know how to develop anything more sophisticated nor have the time, this type of survey is the standard approach. Even the terminology used to describe such forms is telling - "Happy Sheet" or "Smile Sheet" or "Reaction Sheet" ...it suggests there is no serious intention to identify evidence of learning for example, let alone "unpleasant news". What happens to such forms? Many are not even analysed, and those that are usually get the smile and file treatment. Feedback is generally only in response to a higher level request for information about how well the course went, or as a not so subtle means of proving what a good course the Training and Development department has produced and delivered.
It is the author's view, based not only on the literature, but also on her extensive experience in the design of such forms, that post course evaluation is a much maligned and little understood form of evaluation. Yet it can yield considerable data about learning, attitude change, skill development, and course effectiveness in relation to the learning objectives. This article argues for a rethink of the training fraternity's attitude to post course evaluation, proposes a methodology for question design, and provides examples from evaluation forms the author has used.
The decision about what questions to ask needs to be approached thoughtfully. A good form is not designed in a hurry, as a last minute exercise before the course is presented. The methodology developed by the author has been written up in detail elsewhere (Foxon, 1990). In brief, it involves posing the following questions as a prelude to the design of the evaluation strategy, in which the post course reactions form is but one of the tools used:
| PURPOSE | What do I want to know? Why? |
| FOCUS | Where will I source the data? |
| TOOLS | How will I collect the data? |
| FEEDBACK | Whom will I tell and how? |
The purpose of the evaluation must be clarified before proceeding with question design. This phase appears deceptively simple but evaluators are often hazy about what they want to know and why, and this lack of clarity makes question design problematic. Brainstorm the things you want to know - they can include design issues such as timing, sequence, and effectiveness of media; content relevance; instructor effectiveness; likelihood of transfer to the job; how enjoyable the learning experience was; satisfaction with venue (food, rooms, leisure facilities etc), and so on. Now ask yourself why you want to know those things - what difference will it make and what will you do with what you know? Immediately the list will shorten.
There is a close relationship between the evaluation focus and the evaluation tools. The evaluator needs to be clear about where or whom the evaluation data will come from (participant reactions, instructor reactions, on the job behaviour after some months etc) before the tools can be decided on. In my experience trainers tend to start with a decision about tools ("we'll have an end of course reaction sheet and a test") before clarifying the purpose and focus of the exercise. The tools can involve questionnaires, observations, interviews, testing, action plans and so on, and need to be appropriate to the data source (focus) and purpose.
Take a common "smile sheet" question - rating the venue. If you really want to know about the venue, do you need to waste an evaluation question on it? If it's that bad, instructors will have already formed their own opinion before the evaluation form is distributed and participants usually let instructors know as the course progresses. Ask around during breaks if you need to know and are not sure of the general opinion. But if your company holds every residential course at that particular venue, the management is generally efficient, and no complaints are usually received, why ask the question? A quick (verbal) question from the instructor at the end of the course asking for any problems experienced will give you the information, if there is any, and save processing the same bland answer on 20 forms. Use your evaluation form questions to gather data that will be more useful.
The fourth phase, evaluation feedback, is not the subject of this article, but suffice it to say the nature and timing of the feedback, as well as those to whom it will be sent and why, needs careful consideration.
Figure 1
Most courses have several objectives. These need to be clearly stated and each one commented on, if that is the type of data you want. Remember also that the learning objectives of the participants, and the objectives of the course may not always be congruent. And finally, there are better ways of assessing whether course objectives have been met than by asking that vague question.
This article is not addressing techniques of question writing, but one good rule of thumb is to think carefully about what information you want and write a question that will generate that class of data. And always try questions out on a few people before presenting them at the end of the course.
Give careful consideration to the form layout also. Is it "user friendly" and easy to read and fill in? Could the number of questions be reduced or redesigned in order to yield more than one type of data from a single question?
What data can be obtained from post course evaluations? The list is unlimited, but the more common and most useful data are obtained by evaluating at Kirkpatrick's learning and behaviour levels. Examples are given below from some of the author's post course evaluation forms to illustrate the wealth of information that can be obtained. It should be stated that the post course evaluation form is only one aspect of an overall evaluation strategy that is designed by the author for all courses, and which involves long term follow up and evaluation of skill transfer.
Sometimes not every skill can be tested during a course but it may be useful to ask participants to assess their own competence in non-complex skill or knowledge areas. Figure 2 refers to a course teaching a new audit methodology where the new documentation was considerably different from that initially learnt. Consistently low confidence ratings for a particular document alerted course developers and instructors to the need for more in-depth teaching and additional exercises. In addition, some participants had already had informal exposure to the documentation on the job, so the developers wanted to know whether these people were entering the course confident they had a good working knowledge of the subject area, and if that therefore indicated a need for streaming participants on later courses.
Figure 2
Figure 3 also seeks information on entry level knowledge. When this course was being designed it was not possible to determine with any accuracy the entry level knowledge and skills of the likely participants. This question enabled the designers to get feedback on the skills and knowledge participants were bringing to the course. Where necessary they could follow up with individual participants and discuss further how effectively the knowledge/ skill lacks had been met, since all evaluations were signed.
Figure 3
An evaluation of skills in a general sense is demonstrated in Figure 4. The full range of skill areas were listed, and an analysis of the responses indicated where the course content was inadequate and how significantly.
Figure 4
A more specific skill evaluation is shown in Figure 5 which asks participants to assess their competence in relation to taxation matters. The various skills listed are based on specific learning objectives which have been taught as skills (and not just covered with a knowledge based presentation). Unless the course has been skills based in its delivery, such evaluation questions cannot be asked, as participants are making guesses about what they think they will be able to do.
Figure 5
It is often useful to focus on the problem areas that participants anticipate back on the job. This may reveal weaknesses in the course content or depth of coverage. It may also give an insight into aspects of the participant group which had been overlooked in the course design. Fig. 6 is from a course teaching on the job coaching skills. Consistent mention of the same problem area tells designers and instructors a great deal about the course content's relevance and effectiveness, as well as about the participants.
Figure 6
Such questions may also reveal organisational issues which can undermine the effectiveness of the training. Figure 7 is an example from the audit course teaching the new methodology. There was concern by the developer and instructors that some of the senior staff did not support the changes and might in fact attempt to sabotage the implementation of the changes by not giving the documentation jobs to the staff trained in the new methodology.
Figure 7
In order to assess changed on the job behaviour, the evaluator will need to go to the workplace and do some form of evaluation at a later date. However, post course evaluation can in fact indicate likely behaviour change, or lack of it, by asking the right questions. Consider Figure 8. These ask participants on managerial skills courses to comment on their attitudes to managing/supervising. These questions can provide a gold mine of information. They reveal the aspects of the course that made the greatest impact (and that in turn may say something about the particular presenter if there was more than one), what attitudes to managing people the participants held prior to attending this course (often amazing!), what they see as important, their perception of how it can be implemented, and so on.
Figure 8
Figure 9 is seeking slightly different information but embodies the hidden agenda of being a form of action plan and forcing participants to think in terms of specific applications before leaving the course. If the form is signed, it allows for later follow up and assessment of on the job application.
Figure 9
A good general question which can generate considerable data of interest is Figure 10; it also forces participants to look at how valuable the course was in terms of their job, irrespective of whether or not they enjoyed it, liked the venue, thought the video was 'old hat', hated the instructor's style or whatever.
Figure 10
If post course evaluation is taken seriously, adequate time needs to be slotted into the agenda for its completion. The form should be distributed to participants after the course review, and at least 10 to 15 minutes allocated on the timetable for its completion. Moreover, all participants should remain in the course until the evaluation is completed. The course is then closed, and participants are free to leave. Failure to treat it as an integral part of the course does not encourage participants to take the evaluation seriously. If participants know they can leave as soon as the form is filled in, they are less likely to provide the evaluator with thoughtful responses and in-depth data.
The subjective nature of post course evaluation need not be a problem unless that is the only form of evaluation being done. Where post course evaluation is but one segment of an evaluation strategy, it provides relevant and insightful data about various aspects of the training. The need for such forms to be focussed, carefully designed, and piloted before administering them has already been highlighted.
No doubt trainers will go on preparing "Smile Sheets". But for those who know there has to be a better way, a rethink about the post course evaluation to assess reactions, learning and changed behaviour is well worthwhile.
Easterby-Smith, M. (1986). Evaluation of Management Education, Training and Development. UK: Gower.
Foxon, M. (1989). Evaluation of training and development programs: A Review of the literature. Australian Journal of Educational Technology, 5(2), 89-104. http://www.aset.org.au/ajet/ajet5/foxon.html
Foxon, M. (1990). Demystifying Evaluation: A Strategy to Get Started. Performance Technology - 1990. Washington DC: NSPI.
Kirkpatrick, D. (1983). Four Steps To Measuring Training Effectiveness. Personnel Administrator, 28(11), 19-25.
Marsden, J. (1991). Evaluation: Towards a definition and statement of purpose. Australian Journal of Educational Technology, 7(1), 31-38. http://www.aset.org.au/ajet/ajet7/marsden.html
Phillips, J. (1983). Handbook of Training Evaluation and Measurement Methods. London: Gulf.
Warr, P., Bird, M. & Rackham, N. (1970). Evaluation of Management Training. UK: Gower.
| Author: Marguerite Foxon holds post graduate qualifications in instructional design and adult education and has made the subject of training evaluation a focus of interest and research for many years. She is well known for her work in establishing NSPI in Australia, and at the time of writing was National Director of Education for Coopers & Lybrand.
Please cite as: Foxon, M. (1992). In defence of post-course evaluations: Going beyond the smile sheet. Australian Journal of Educational Technology, 8(1), 1-12. http://www.ascilite.org.au/ajet/ajet8/foxon.html |