Figure 1: Steps in the Selection of a CBT System
| Australian Journal of Educational Technology 1992, 8(2), 188-202. |
AJET 8 |
Peter Hosie
Instructional Design and Evaluation Australia
Choosing a Computer Based Training system is a daunting task. With in excess of one hundred major packages on offer, errors are costly in terms of scarce capital, opportunities foregone, and human resources wasted. Selection criteria must start with, and be matched to, learner needs and the training requirements of the organisation, and not be technology driven. This article will outline a process for the selection of a CBT package in line with these needs.
CBT authoring systems can not be all things to all people. Conversely, unless the tasks requiring training are compelling enough to warrant the selection of a dedicated system with specialist capabilities, a selected CBT system may be used in a number of training circumstances. Undoubtedly tomorrow's authoring systems will offer exciting possibilities. As enchanting as so-called artificial intelligence languages and such are, or will be, any evaluation is only concerned with what can be purchased and implemented now, not in a decade.
For a CBT authoring system to assist designers produce courseware akin to learner preferences it must have certain technological capabilities. Quality courseware is most effectively developed by providing courseware designers with the best tools available.
As Kearsley (1983, p 14) observes computers have certain advantages in instructional settings, such as: permitting "students to learn at their own pace, individual learning styles are considered, resulting in increased student satisfaction. Most importantly, there is more control over learning materials and learning process".
Parallelling this high level of activity has been the number of courseware vendors; the nine courseware vendors surveyed in 1982 reached 114 in 1986 and 150 in 1987. "The 1993 CBT Buyer's Guide" list 124 vendors providing a wide range of CBT courseware and services. The offerings continue to expand which is either paradise or a prospective client's nightmare.
According to Data Training (1987, p 5), "It isn't just the number of systems that keeps growing either, it's their capabilities ... the variety of features and options is fast outstripping our ability to report on them succinctly". Witness the growth in multimedia and electronic support features being incorporated into the latest versions of authoring systems. If the experts can't keep pace what hope is there for neophytes to the game?
Weingarten Publications' has been publishing an annual "Guide to Computer-Aided Training" since 1982. These are of inestimable value for initially sorting which system or systems to choose. But such a publication does, by its own admission, have shortcomings. There is no attempt to rate the quality of the features mentioned, only their presence or absence - essentially a binary analysis. Despite this limitation such information saves considerable brain and legwork - permitting unsuitable products to be eliminated at an initial screening.
The process outlined in Figure 1 will filter out those systems that do not meet the organisation's needs, and provides a mechanism to rank order candidate CBT systems. The cornerstone of the process is the clear identification of learning needs, and the necessary system capabilities to meet these needs. This is achieved through the formulation of a model CBT system, against which candidate systems can be assessed.
Figure 1: Steps in the Selection of a CBT System
There are a number of considerations when it comes to selecting an authoring system. The point to be stressed is that consideration must be given to a range of influences, not just the technology, viz:
Bearing in mind the probable instructional design strategies, probable author capabilities, and other critical aspects of the development environment, the selection of authoring features and functions is a daunting task. Choosing an authoring features always requires trade-offs. Gery (1987, pp 81-87) offers some sound advice when is comes to the selection of authoring features. There are three interrelated trade-off dimensions in the selection of authoring systems:
Productivity vs CreativityIn addition to identifying the features and functions, these considerations will identify those features which are essential and those which are desirable. The essential features should constitute a minimum set of features and functionality to meet organisational and the learner's needs. Any system not meeting these requirements will be excluded from further consideration.
Power vs Simplicity
Structure vs Freedom
These features must be derived from the earlier considerations and that they cover a number of aspects of the authoring system, not only authoring features.
AuthoringA detailed instrument is at Appendix 1.
Presentation
Management
Vendor Support
There may be other considerations that an organisation may wish included in an evaluation instrument. However, the impetus for the inclusion of any feature or function must come from the earlier considerations, and not an arbitrary decision.
In determining the suitability of any feature or function, two five point rating scales are employed. The "desirability' scale measured perceived learner and organisation features, with the rating scale providing a method of evaluating the quality of particular features. The two scores were multiplied to give a raw weighted score. The scales are:
| Desirability | ||
| 5 | = | Very important |
| 4 | = | Highly desirable |
| 3 | = | Desired |
| 2 | = | Desired but not required |
| 1 | = | Present but not required |
| Rating | ||
| 5 | = | 100% match required |
| 4 | = | 75% match required |
| 3 | = | 25% match required |
| 2 | = | 25% match required |
| 1 | = | 0% match required |
The earlier decisions on features and functionality for the organisation and the learner must be reflected in the selection of the appropriate point on the item's desirability scale. In this example, maximum possible scores are:
| Authoring | 410 points | (40% of total possible score) |
| Presentation | 220 points | (22% of total possible score) |
| Management | 135 points | (13% of total possible score) |
| Vendor Support | 155 points | (15% of total possible score) |
These scores reflect the relative importance of each section in the selection of a CBT system. It must be stressed however, that these weightings are by way of illustration only. The selection of a system is influenced by different factors for every organisation. Accordingly, these factors will drive the inclusion of system features as well as the desirability rating for each feature.
Gery (1987, p 76) makes the point to:
A section for a "qualitative response" has been added (100 points or 10% of total possible score), consisting of:
| Subjective analysis | 50 possible points |
| Colleagues evaluation | 20 possible points |
| Literature comments | 30 possible points |
Naturally this could be criticised, quite reasonably, for introducing too subjective a criteria into the evaluation. This claim can be readily countered by the observation that many CBT systems, as selected, use purely subjective evaluation methods. After all, an affinity with what learners find engaging should be part of the repartee of a CBT designer.
The opinions of colleagues using the software should be sought where possible. However, because these opinions are subjective as well as secondhand, judicious weighting must given to this category.
Reviews of the software under consideration should also be examined. The quality and quantity of these reviews varies, but on balance, they remain a valuable source of information and therefore should be included in the rating considerations.
From this process, candidate systems can be rank ordered, with the systems with the highest score best suiting the needs of the organisation.
The intention of the cost analysis is to determine the system with least cost for producing specifically defined outputs. The analysis will identify variations in input for a common level of output. Systems not attaining this common level of output will be normalised with the costs involved in raising the solution to the required output level included in the costing for that solution.
The analysis should be for a defined period from a base year, with costs expressed in terms of their discounted, or base year, values. This allows for systems with minimal initial expenditure and increasing recurring expenditures to be compared to systems with high initial capital investment but lower operating costs.
The economic life is the length of time for which the system can be expected to yield benefits. In the context of CBT systems it is unwise to predict a very long economic life. Systems and technology are changing so rapidly, that even the most up to date system can be expected to be overtaken by new developments in a short time. Five years is probably a reasonable time for mainframe and mini based systems, or systems with a high degree of specialised application, and three years being a realistic figure for PC based systems.
It is important to recall that the evaluation is for the most appropriate training tool to meet the needs of the organisation (including trainers and developers) and the learner, not necessarily the best computer system. As such, training considerations should predominate. The analysis, apart from considering capital and recurring costs, should deal not only with the basic software purchase, but should include:
Results gained during this costing stage may cause a change in the rank ordering of the desirable systems.
The purpose of a prototype is to determine if the design of the system is appropriate: before progressing. In this case. the purpose of the prototype is to validate the system characteristics and capabilities identified earlier. Thus, a prototype represents some small portion of the system, usually selected as exemplary of the full-scale effort. A pilot test would involve trying out the system with a small group of "students". Such "students" are typically colleagues or subject matter experts who am not directly involved in the development of the system. The purpose of a pilot is to detect any major problems in the hardware, software, courseware, courseware development and project management process.
The final stage of a trial is field testing in which the system is tried out in the actual training setting for which it is being selected. It may be necessary to review some of the earlier decisions (performance, system feature desirability, costing), based on the information gained. This review may alter the rank ordering of the systems.
The outcome of this filtering process will be the identification of the most suitable authoring system, based on the organisation's and learners' needs.
The critical success factors in Selecting a CBT system are:
Not all features will be of equal importance in the selection of a CBT System. While suppliers strive to increase the complexity of their offerings and the variety of the features offered, few systems offer all features and functionality. This leaves evaluators with a dilemma. On what basis is the most appropriate CBT system selected?
Choosing an authoring system always involves trade-offs, whether this is realised or not. However, the foundation for a successful selection process is ensuring that all decisions on selection are based on the learning needs of the organisation and the learner. In making these trade offs, features and functions may be omitted, or their desirability changed, but this must be a conscious, considered action, rather than by oversight. In an process of selecting a CBT system, comparisons between systems must be as objective as possible. This can be achieved by measuring candidate systems against a definability/performance model derived from organisation and learner needs.
Burke, R. (1982). CAI Sourcebook. New Jersey: Prentice-Hall.
CBT Directions (1992). Comparing Authoring Systems: Where Do You Start? CBT Directions, May/June, 15-29.
CBT Directions (1992). 1993 CBT Buyer's Guide. Boston, MA: Weingarten Publications.
Data training. (1987). The 1987 Guide to Computer Based Training. Boston MA: Weingarten Publications.
Data Training (1991). The 1991 Guide to Computer Based Training. Boston, MA: Weingarten Publications.
Gery, G. (1987). Making CBT Happen: Prescription for Successful implementation of Computer Based Training in Your Organisation. Boston, MA: Weingarten Publications.
Hosie, P. (1987). Adopting Interactive Videodisc Technology for Education. Educational Technology, July, pp. 5-10.
Jamieson, D. (1983). Selecting a CBT System. Proceedings of the information Technology Month Seminar. Perth, Australia.
Kearsley, G. (1983). Computer Based Training: A Guide to Selection and Implementation. Reading, MA: Addison-Wesley Publishing Company.
| Desirability | Rating | |
|---|---|---|
| 1 2 3 4 5 | Menus or prompts for creating lessons | 1 2 3 4 5 |
| 1 2 3 4 5 | Authoring templates to support a full range of learning styles | 1 2 3 4 5 |
| 1 2 3 4 5 | Ability to bypass menus, prompts and templates for command level authoring | 1 2 3 4 5 |
| 1 2 3 4 5 | Unlimited use of the screen by author | 1 2 3 4 5 |
| 1 2 3 4 5 | Flexible branching under author control | 1 2 3 4 5 |
| 1 2 3 4 5 | Logical command structure | 1 2 3 4 5 |
| 1 2 3 4 5 | Full screen text editor (internal or external) | 1 2 3 4 5 |
| 1 2 3 4 5 | Import external text and graphics | 1 2 3 4 5 |
| 1 2 3 4 5 | Interface with other software for authoring | 1 2 3 4 5 |
| 1 2 3 4 5 | Use of non keyboard devices for inputting | 1 2 3 4 5 |
| 1 2 3 4 5 | Editor for unresolved branches | 1 2 3 4 5 |
| 1 2 3 4 5 | Easy screen partitioning and windowing | 1 2 3 4 5 |
| 1 2 3 4 5 | Spelling Checker | 1 2 3 4 5 |
| 1 2 3 4 5 | Multiple fonts and character sizes | 1 2 3 4 5 |
| 1 2 3 4 5 | Range of colour palettes | 1 2 3 4 5 |
| 1 2 3 4 5 | High resolution graphics | 1 2 3 4 5 |
| 1 2 3 4 5 | Ease of controlling video disc | 1 2 3 4 5 |
| 1 2 3 4 5 | Ease of controlling video tape | 1 2 3 4 5 |
| 1 2 3 4 5 | Ease of frame (video) and screen (text) integration | 1 2 3 4 5 |
| 1 2 3 4 5 | Ability for author to test lesson without leaving author system | 1 2 3 4 5 |
| 1 2 3 4 5 | Capable of creating lessons up to a maximum size of 360 kbytes | 1 2 3 4 5 |
| 1 2 3 4 5 | Ability to link several modules | 1 2 3 4 5 |
Other Comments
Presentation
| Desirability | Rating | |
|---|---|---|
| 1 2 3 4 5 | Flexibility in handling input from student | 1 2 3 4 5 |
| 1 2 3 4 5 | Range of response types | 1 2 3 4 5 |
| 1 2 3 4 5 | Use of non keyboard devices for input, selection and progression | 1 2 3 4 5 |
| 1 2 3 4 5 | Ability to use external devices other than video tape and disc | 1 2 3 4 5 |
| 1 2 3 4 5 | Ability to communicate with other software at runtime | 1 2 3 4 5 |
| 1 2 3 4 5 | Student resume facility | 1 2 3 4 5 |
| 1 2 3 4 5 | Easy back paging and progression | 1 2 3 4 5 |
| 1 2 3 4 5 | Student controlled branching Hard copy of screen displays | 1 2 3 4 5 |
| 1 2 3 4 5 | Author/student communications facility | 1 2 3 4 5 |
| 1 2 3 4 5 | Help facility | 1 2 3 4 5 |
| 1 2 3 4 5 | Random test generation | 1 2 3 4 5 |
| 1 2 3 4 5 | Lesson progression and completion display | 1 2 3 4 5 |
Other Comments
Management
| Desirability | Rating | |
|---|---|---|
| 1 2 3 4 5 | Student registration facility | 1 2 3 4 5 |
| 1 2 3 4 5 | Author controlled student tracking | 1 2 3 4 5 |
| 1 2 3 4 5 | Ability to record individual responses | 1 2 3 4 5 |
| 1 2 3 4 5 | Ability to record the number of help requests | 1 2 3 4 5 |
| 1 2 3 4 5 | Security facility that restricts access to performance data and courses | 1 2 3 4 5 |
| 1 2 3 4 5 | Ability to analyse individual student performance data | 1 2 3 4 5 |
| 1 2 3 4 5 | Ability to compare student performance | 1 2 3 4 5 |
| 1 2 3 4 5 | Ability to export student data for analysis | 1 2 3 4 5 |
Other Comments
Vendor Support
| Desirability | Rating | |
|---|---|---|
| 1 2 3 4 5 | Concise and easy to use documentation | 1 2 3 4 5 |
| 1 2 3 4 5 | On site author training | 1 2 3 4 5 |
| 1 2 3 4 5 | CBT based author training | 1 2 3 4 5 |
| 1 2 3 4 5 | Locally based technical support | 1 2 3 4 5 |
| 1 2 3 4 5 | Telephone support | 1 2 3 4 5 |
| 1 2 3 4 5 | Availability of local user groups | 1 2 3 4 5 |
| 1 2 3 4 5 | Implemented for a range of computer video configurations | 1 2 3 4 5 |
| 1 2 3 4 5 | Planned development cycle for system | 1 2 3 4 5 |
| 1 2 3 4 5 | License agreement non restrictive | 1 2 3 4 5 |
Other Comments
Qualitative Analysis
Subjective Evaluation (50 points)
Colleagues' Evaluation (20 points)
Literature Review (30 points)
| Authors: Duncan Jamieson is a Training Consultant with Applied MultiMedia, currently based in Canberra. Duncan develops CBT and paper based self study materials, as well as advising on the identification, formulation, design and evaluation of Training in the fields of information Technology and Computing. Duncan has worked for the WA Department of Computing and Information Technology as a Human Resource Development Consultant, and has been the National Training Manager for the Australian Federal Department of Social Security and a CBT adviser. He is a member of the Australian institute of Training and Development (AITD), the Australian Computer Society (ACS), the Australian Society for Educational Technology (ASET) and the National Society for Performance and instruction (NSPI).
Peter Hosie is currently a Senior Educational Development Officer with the Ngee Ann Polytechnic in Singapore. Peter has been the Executive Officer for the Telecommunications for Education and Training Taskforce, formed by the Western Australian Government to establish an enduring mechanism for the delivery of education and training using telecommunications. As an Education Officer with the WA Ministry of Education, Peter was involved in the development of an interactive videodisc package for career counselling. Please cite as: Jamieson, D. and Hosie, P. (1992). Selecting computer based authoring packages for training. Australian Journal of Educational Technology, 8(2), 188-202. http://www.ascilite.org.au/ajet/ajet8/jamieson.html |