AJET 5(2): Watt (1989) - exchange of information on instructional materials

Australian Journal of Educational Technology
1989, 5(2), 77-88.

The exchange of information on instructional materials: An evaluation for planned change in Australian education

Michael G Watt
Division of Educational Programs
Tasmanian Department of Education and the Arts

This article is a summary of a final report on an evaluation project. This evaluation project presented a plan for a program to establish a clearinghouse for exchanging qualitative and quantitative data on instructional materials used in Australian schools. A range of elements necessary to plan a program suited to Australian requirements and conditions was investigated.

The collection, the synthesis and the dissemination of qualitative information on instructional materials to teachers has represented an intractable problem in most contexts of the educational setting. The few, successful programs in education provide the evidence to substantiate this claim. The purpose of this article is to report a planned solution to this problem for Australian education by basing that solution upon such estimable and successful programs operating in foreign contexts.

The rationale for this planned solution arose from communications during 1986 with an American agency, the Educational Products Information Exchange (EPIE) Institute, which operates such a program by collecting, analysing and disseminating descriptive information on educational products to all sections of the American educational community on a national basis. The outcome of these communications was conveyed to the board of directors of the Australian Schools Catalogue Information Service (ASCIS), which then recommended at a meeting during April 1988 with representatives from the Curriculum Development Centre (CDC) that the methods, techniques and practices applied by the EPIE Institute should be considered for adoption. However, the proposal, stated at the sixtieth meeting of the Australian Education Council during April 1989, to amalgamate both these organisations into a single agency, to be known as the Curriculum Corporation of Australia, has meant that the recommendation must be deferred until after the new agency is established.

During this period, an evaluation project was undertaken in order to verify the propositions proposed in the recommendation. The project had three main aims: to explicate the objectives of a prospective program to collect, to synthesise and to disseminate qualitative and quantitative data on instructional materials to Australian schools; to investigate the potential adaptation of innovative methods, techniques and practices to analyse qualitative and quantitative data on instructional materials; and to determine the resources that both Australian and foreign educational organisations may contribute to planning, structuring, implementing and recycling the procedures of the prospective program. The design and findings of the evaluation project, together with the implications for future action, are described in the remainder of this article.

The Evaluation Design

The Context-Input-Process-Product (CIPP) Model, postulated by Stufflebeam et al. (1971), was applied in the evaluation project through the use of two types - context and input - because the project was limited to that of a feasibility study. A context evaluation in the contingency mode was used to assess the attributes of the methods, techniques and practices used in the problem area, and then an input evaluation for neomobilistic change was used to assess both the relevant capabilities of Australian educational agencies to assume decision-making responsibility for the prospective program and the strategies that foreign educational agencies might employ to achieve the goals of the prospective program.

Stufflebeam et al. define the purpose of context evaluation as serving planning decisions in order to determine objectives. Two modes of context evaluation are identifiable: contingency and congruence. Because the contingency mode involves identifying opportunities and forces beyond the boundaries of the immediate system to promote improvements in it, characteristic techniques of this type were employed to probe external forces and to predict into the future.

Stufflebeam et al. define the purpose of input evaluation as providing information for determining how to use resources to meet program goals. This is accomplished through identifying and assessing three characteristics: firstly, the relevant capabilities of a responsible agency; secondly, strategies for achieving program goals; and thirdly, designs for implementing a selected strategy. In this project, such a process assumed a particular form. In the first instance, the relevant capabilities of the two Australian educational agencies were identified and assessed, in the second instance, a technique was administered to probe strategies that foreign educational agencies may apply to achieve program goals, and in the third instance, the design for implementing the program was determined through establishing a congruence between existing capabilities of Australian educational agencies and strategies that foreign educational agencies can provide.

The CIPP Model ascribes a variety of methodologies for input evaluation, depending upon whether large or small change is involved and whether high or low information grasp is available to support the change. The conceptualisation of the decision-making process, described by Stufflebeam et al., was used to determine that a neomobilistic decision setting pertained to this problem. The advanced attributes of methods, techniques and practices used in foreign contexts determine that innovative activity for inventing, testing and diffusing new solutions should be applied. Such a large change, however, is supported by little information within the Australian context. The planned change model, recommended by Stufflebeam et al. for neomobilistic settings, was applied within this study in two ways: firstly, to assess the capabilities of the relevant Australian educational organisations; and secondly, to specify the design for the prospective program, in which innovative methods, techniques and practices derived from foreign contexts are superimposed upon existing practices in Australian education.

Target Populations

The target population for the context evaluation comprised service agencies of the Department of Education and the Arts, Tasmania and the education departments of other Australian states and territories. The target population for the input evaluation comprised those foreign educational agencies likely to participate in the prospective program. Methods of sampling were not employed in the study.

Description of the Designs

The design for the context evaluation employed both empirical methods to collect baseline data on the methods, techniques and practices used in the Australian context and archival research to survey the American context. In the former case, an instrument was designed to measure the validity and reliability of the methods, techniques and practices that staff members at the service agencies used for five functions: learner verification and revision of instructional materials; screening instructional materials for adoption; descriptive analysis and evaluation of instructional materials; decision-making for selection of instructional materials; and decision-making for implementation of instructional materials.

The design for the input evaluation employed a planned change model to predict a plan for a prospective program in Australian education. This design consisted of four stages: firstly, defining the problem through the four elements - awareness, design, choice and action - of the decision-making process; secondly, determining the decision-setting, that is, whether the problem is set in a homeostatic, incremental, neomobilistic, or metamorphic decision-setting; thirdly, applying the appropriate decision model - whether it is the synoptic ideal model, the disjointed incremental model, or the planned change model - following determination of the type of the decision-setting; and fourthly, accounting for the types of decisions applied, whether planning, structuring, implementing and recycling decisions.

Once these procedures had been applied, a flow chart reproduced as Figure 1, was described for planning and structuring decisions. The flow chart, to be read from top to bottom, shows that change from the existing program to the prospective program is to be met by application of innovative methods, techniques and practices, derived from foreign contexts, to data on instructional materials.

At this point, the relevant capabilities of the responsible Australian educational agencies were assessed. The relevant capabilities of the Curriculum Development Centre (CDC) were assessed on the criteria that relate to research and development activities, the relevant capabilities of the Australian Schools Catalogue Information Service (ASCIS) were assessed on the criteria that relate to diffusion activities, whilst both the Curriculum Development Centre (CDC) and the Australian Schools Catalogue Information Service (ASCIS) were assessed on criteria for adoption activities.

Secondly, a questionnaire, derived from a technique termed Policy Implications Analysis (Madey and Stenner, 1981), was administered to assess the strategies that foreign educational organisations could employ to achieve the goals of the prospective program. Policy Implications Analysis is a procedure designed to enhance the likelihood that an evaluation will have an impact upon decision-making. The method employs six steps: firstly, generation of hypothetical findings; secondly, the preparation of a questionnaire to be administered to a selected panel of respondents; thirdly, the selection of a panel of respondents; fourthly, the administration of the questionnaire to the panel; fifthly, the analysis of the responses; and lastly, the use of the analysed responses to develop a set of policy-relevant hypotheses.

Neomobilistic Decision Setting

Figure 1: A flow chart of attributes in decision-making for the prospective program.

Lastly, the congruence between the findings of the two previous stages was assessed to determine the procedural design, in terms of the collaborative relations between both the Australian and foreign educational organisations. The planned change model was applied to test the problem at each of two steps: at the first, the model was used to provide an action plan, whereby the Australian educational organisations would structure the processes of the prospective program; and at the second, the model was used to present the optimal procedural design, whereby both the Australian and foreign educational organisations would implement and recycle the processes of the prospective program.

Types of Analyses Performed

Only qualitative data were analysed in both the context and input evaluations. The use of qualitative data meant that criteria were non-numerical. Intuitive processes, based upon the researcher's knowledge of the topic, were used to form judgements. Statistical methods were not applied to analyse data.

Major Findings on Questions of Context

What do the findings inform about Australian practices?

The study identified that research findings (Marsh et al., 1985) on Australian practices are not extensive. These findings were consistent with the findings of this evaluation project, however, in identifying deficiencies in the validity and reliability of practices, techniques and methods used in the Australian context. The techniques used to provide information for learner verification and adoption, relied upon intuitive processes, those used for descriptive analysis were performed on criteria which do not relate to the principles of curriculum design, and those applied to decision-making for both selection and implementation were usually based upon informal, consultative processes.

This work is conducted almost entirely by state departments of education and there is no co-ordination of these activities at a national level. Research and development activities are funded inadequately and appear to have contributed little to the improvement of practice. Although the means are now available to diffuse qualitative information on instructional materials to users, such means are not being used at present to diffuse this type of information. There is no evidence that people undertaking this work are trained formally, that the activities of this work have been trialed in the field, and that the characteristics of the activities fit or have been assimilated successfully to the programs of state departments of education. In summary, work in the problem area appears to be ineffective because the coverage of some activities lacks inventiveness, and the separation of programs at the state level has caused inefficiency and a lack of co-ordination.

What do the findings inform about foreign practices?

The evidence suggested that the widespread use of valid and reliable techniques occurs only in American and Canadian contexts. There is an extensive history of the use of committees to adopt textbooks in the United States, and whilst the activities of committees have been criticised in the reform reports on American education (National Commission on Excellence in Education, 1983; Bennett, 1988), training activities to improve teachers' skills in adoption practices have been implemented recently. The Educational Products Information Exchange (EPIE) Institute has implemented techniques for learner verification and descriptive analysis at a national level, and it is applying processes for curriculum alignment to this field of work through an Integrated Instructional Information Resource (Komoski, 1987).

The evidence suggested that Canadian practices have been derived and adapted from the American context (Wright, 1983). The provincial education departments in British Columbia, Alberta and Manitoba have formed a consortium which applies practices, techniques and methods adopted from the EPIE Institute. Through this consortium, the Canadian Exchange for Instructional Materials Analysis (CEIMA), Canadian educators coordinate the exchange of descriptive analyses of instructional materials between provincial departments of education.

There was evidence in research literature that similar practices, techniques and methods have been developed in some European countries, but current work in Europe appears to be restricted to that taking place at the University of Sussex in the United Kingdom (Eraut et al., 1975). Practices, techniques and methods from this work are being adapted for the use in Iceland (Sigurgeirsson, 1986), and being considered for use in the educational system of the People's Republic of China.

Major Findings on Questions of Input

The findings on the questions of input identified that change from the existing to a preferred situation in Australian education is governed by two main attributes: first, its degree is large because of the extensive deficiencies in Australian practice; and second, the knowledge and skills of Australian practitioners about the innovative solutions to be introduced are low. Therefore, the change process will occur within the decision setting in which innovative solutions must be introduced. A model for planned change, involving many steps and agencies over a long duration, is prescribed to achieve the change process.

Through analysis of the findings, the relevant capabilities of Australian educational agencies are diagnosed to be deficient in three sub activities of the change process: research; invention; and training. Furthermore, foreign educational agencies were identified as being capable of employing a range of those strategies that would achieve the goals of the prospective program in those sub-activities diagnosed to be deficient.

Limitations of the Study

The limitations of the study that may affect the validity of conclusions drawn from the data are described for each component of the findings: the context evaluation of Australian education; the context evaluation of American education; and the input evaluation.

Although the coverage of activities in this field within Australian education was extensive, its balance was inequitable. Whilst the practices, techniques and methods used in Tasmanian education were examined in detail, those used by other state departments of education in Australia were not examined with the same rigour. The scope of this coverage was governed by the use of empirical approaches, which meant that the parameters of the study were limited by practicality.

Although the coverage of activities in this field within American education was extensive, it was not fully comprehensive. The examination concentrated upon the practices, techniques and methods used for descriptive analysis and evaluation of instructional materials. The treatment of practices, techniques and methods for learner verification and adoption was superficial. Practices, techniques and methods, used by American teachers in decision making for either selection or implementation of instructional materials in classrooms were not examined. The coverage of activities in the American context was restricted by two attributes: first, the aim of the study; and secondly, the availability of information.

Furthermore, activities in this field within both Australian and American education were examined from the perspective of the practices, techniques and methods for analysis and not for use of the subsequent products.

Although the decision-setting for change in Australian education was diagnosed by the use of the model for planned change, empirical approaches needed to be applied to analyse data to gain objectivity rather than relying upon the process of subjective judgement. Furthermore, the survey of strategies that foreign educational agencies could employ to achieve the program goals was limited by the incompleteness of the data. The evaluation of input was governed by practicality, which limited the range of approaches that could be applied.

Implications of the Findings for Future Action

On the basis of the diagnosis for input, a set of steps should be prescribed to introduce procedures for the research, development, diffusion and adoption phases of the change process to achieve the goals of the prospective program. This task is predicted to be complex, requiring the application of a great deal of information about the process and the products of the change steps. Because there is little information available within the Australian context on both innovative solutions and the processes and products of the change steps, this information needs to be sought from foreign contexts.

An action plan for the prospective program is presented below whereby the Curriculum Corporation of Australia will be able to structure decisions for the optimal procedural design. The plan describes a process whereby the Curriculum Corporation of Australia reviews existent research on the problem area, implements a set of recommendations and collaborates with the EPIE Institute to design, construct, assemble, disseminate and demonstrate the plan for the prospective program to client groups and key audiences, and then, together with the EPIE Institute and the CEIMA, train local personnel, trial, install, and institutionalise the prospective program in the Australian context. Eleven steps for future action would follow the sequence prescribed in the model for planned change: research; invention; design; construction; assembly; dissemination; demonstration; training; trial; installation; and institutionalisation.

Research: Research should be initiated by the Curriculum Corporation of Australia, through a review of the final report by Watt (1988). It should then extend attention to the examination of reports published by the EPIE Institute and the CEIMA. The Curriculum Corporation of Australia should implement the recommendations presented by Watt to review existing activities in the Australian context. The Curriculum Corporation of Australia should commission projects to research the possibilities of adapting innovative solutions derived from the EPIE Institute and the CEIMA.
Invention: The Curriculum Corporation of Australia should sponsor the EPIE Institute to conduct a workshop for Australian educators. The workshop would have two aims: first, to introduce knowledge and skills of the practices, techniques and methods that the EPIE Institute uses; and second, to assist the Curriculum Corporation of Australia to structure decisions for the prospective program for Australian education. Thereafter, invention, resulting from collaboration between the Australian agencies and the EPIE Institute, would be a prominent constituent in the program. The Fulbright Awards of the Australian-American Educational Foundation may form a source to fund the visit of a staff member from the EPIE Institute.
Design: In collaboration with the EPIE Institute, the Curriculum Corporation of Australia should design a blue-print to be followed in engineering the prospective program. The parameters of the blueprint should encompass the six short-term objectives for the program: to determine the types of information to be collected, whether producer, analyst, and user information; to apply a technique of instructional design analysis for the descriptive analysis and evaluation of curriculum materials that also incorporates criteria to identify biases and to match readability levels of curriculum materials and students' reading levels; to develop a data collection instrument; to implement a program to train and to certificate analysts in the adopted technique for descriptive analysis and evaluation of curriculum materials; to develop software for storage and retrieval of information; and to develop the products for dissemination of information to users.
Construction: In collaboration with the EPIE Institute, the Curriculum Corporation of Australia should construct the components for the six short-term objectives for the prospective program. Once the short-term objectives have been implemented, components for each of the seven long-term objectives, and for the incorporation of the National Software Clearinghouse Unit, should be constructed. Listed in order of priority are seven long-term objectives for the program: to develop the practical means for publicising the program, and for informing users of significant matters concerning educational products, and for collecting information from users; to develop and to implement a teacher education program; to implement guidelines for Australian publishers of instructional materials that comply with learner verification and revision requirements; to implement standardised techniques and processes for adopting and selecting instructional materials through a utilisation policy; to initiate research on the uses of instructional materials in the Australian educational context; to investigate the need to provide a consumer protection service; and to initiate the introduction of an integrated instructional information resource and a curriculum alignment service.
Assembly: In collaboration with the EPIE Institute, the Curriculum Corporation of Australia should assemble the components of the six short-term objectives into an operating program. Once the six short term objectives are implemented, the components of the seven long term objectives and for the incorporation of the National Software Clearinghouse Unit should be incorporated into the operating program.
Dissemination: In collaboration with the EPIE Institute the Curriculum Corporation of Australia should disseminate through a newsletter information on the components of the operating program to create awareness among teachers, teachers educators, curriculum specialists and school administrators. The newsletter should also function as the main means to collect data from users on a range of research and consumer issues concerning their uses of instructional materials.
Demonstration: In collaboration with EPIE Institute, the Curriculum Corporation of Australia should provide opportunities for client groups to examine and assess the operating program. Because diffusion is to occur at a national level, this activity would be best achieved through workshops and seminars conducted in cooperation with state territorial departments of education.
Training: The Curriculum Corporation of Australia should sponsor the EPIE Institute and the CEIMA to conduct workshops in Australia on the procedures that these agencies use to train analysts of instructional materials. Following the adoption and modification of the procedures to meet Australian requirements, the Curriculum Corporation of Australia could gain responsibility for conducting this activity. Once the program has been implemented, the Curriculum Corporation of Australia should design professional materials to improve teachers' practices for selecting and implementing instructional materials.
Trial: In collaboration with the EPIE Institute, the Curriculum Corporation of Australia should trial the operating program both to build familiarity with its components and to test its quality, value, fit and utility in the adopting agencies. The trial should be conducted within the system of a single state education department in Australia. The program should be revised according to the findings of the trial.
Installation: In collaboration with the EPIE Institute, the Curriculum Corporation of Australia should install the program, comprising the components of the six short-term objectives. Subsequently the components of the seven long-term objectives should be installed.
Institutionalisation: In collaboration with the EPIE Institute, the Curriculum Corporation of Australia should assimilate the program as an integral and accepted component within the ongoing system.

References

Bennett, W. J. (1988). American Education: Making It Work. Washington: United States Department of Education.

Eraut, M., Goad, L. and Smith, G. (1975). The Analysis of Curriculum Materials (University of Sussex. Education Occasional Paper 2). Brighton: University of Sussex.

Komoski, P. K. (1987). Educational Technology: The Closing-in or the Opening-out of Curriculum and Instruction. Syracuse: ERIC Clearinghouse on Information Resources

Madey, D. L. and Stenner, A. J. (1981). Policy Implications Analysis: a method for improving policy research and evaluation. In Aslanian, C. B. (ed), Improving Educational Evaluation Methods: Impact on Policy (Sage Research Series in Evaluation, Vol.11). Beverly Hills: Sage Publications, 23-39.

Marsh, C. J., Willis, J., Newby, J. H., Deschamp, P. and Davis, B. P. (1985). Teachers' perceptions about the selection, distribution and use of social studies and mathematics curriculum materials within a state education system. Journal of Curriculum Studies, 17(1), 49-61.

National Commission on Excellence in Education. (1983). A Nation at Risk: The Imperative for Educational Reform. Washington: United States Government Printing Office.

Sigurgeirsson, I. (1986). Improving Curriculum Materials Development in Iceland through Curriculum Analysis, M.A. in Education thesis, University of Sussex.

Stufflebeam, D. L., Foley, W. J., Gephart, W. J., Guba, E. G., Hammond, R. L., Merriman, H. O. and Provus, M. M. (1971). Educational Evaluation and Decision Making. Itasca: F. E. Peacock Publishers.

Watt, M. G. (1988). A System for the Exchange of Information on Instructional Materials: An Evaluation for Planned Change in Australian Education, M.Ed. Studies thesis, University of Tasmania.

Wright, I. (1983). The polities of curriculum materials selection: the British Columbia case. History and Social Science Teacher, 18(4), 211-218.

Author: Michael G. Watt, BA, Dip. Ed, B. Ed (Tas), is an Education Officer with the Staff Development Section, Department of Education and the Arts, 71 Letitia Street, North Hobart, Tasmania 7000.
Please cite as: Watt, M. G. (1989). The exchange of information on instructional materials: An evaluation for planned change in Australian education. Australian Journal of Educational Technology, 5(2), 77-88. http://www.ascilite.org.au/ajet/ajet5/watt.html

[ AJET 5 ] [ AJET home ]
HTML Editor: Roger Atkinson [rjatkinson@bigpond.com]
This URL: http://www.ascilite.org.au/ajet/ajet5/watt.html Last revision: 23 Sep 2002.
Previous URL 21 Nov 1996 to 23 Sep 2002: http://cleo.murdoch.edu.au/gen/aset/ajet/ajet5/su89p77.html