Australasian Journal of Educational Technology
2008, 24(2), 143-149. 		AJET 24

Promoting staff learning about assessment through digital representations of practice: Evaluating a pilot project

Rosemary Thomson and Gail Wilson
University of Western Sydney

An Outstanding Paper Award recipient, ascilite Singapore 2007 Conference

The Assessment Snapshots digital resource is a current project to support academic learning about assessment by diffusing knowledge and understanding of locally contextualised good practice in assessment at the University of Western Sydney. An initial collection of Snapshots was made available to academic staff on the university web site in early 2006. This paper describes how the resource has been utilised by teachers and explains the uses that academic developers have made of the resource in supporting teachers to extend their learning about assessment. The paper reflects on the extent to which the pilot project has been successful in disseminating effective assessment practice and promoting reflection and discussion about assessment issues. It proposes future directions for more effectively integrating and contextualising resources for professional learning with teachers' everyday teaching practice.

Introduction

This paper describes the evaluation of the Assessment Snapshots Project, a pilot professional development project at the University of Western Sydney which aimed to diffuse knowledge and understanding about effective assessment practices. The project focused on the development of digital, readily accessible resources for academic staff. The rationale for and processes of developing the initial collection of digital resources about assessment practice was described in an earlier paper presented at the ascilite Sydney 2006 Conference (Wilson, Thomson, Malfroy, 2006). In that paper we foreshadowed further work to evaluate the effectiveness of these resources for professional development of academic staff. This paper reports on our approach to evaluating the project and plans for extending the resource in the future.

Background to the project

The pilot phase of the project produced a small collection of seven Assessment Snapshots which was made available to university staff in early 2006. The intention of the project was to diffuse knowledge and understanding about assessment practices and to promote reflection on and discussion about assessment issues within the University. The Snapshots collection profiles diverse assessment practices in large undergraduate classes, across the disciplines: The resource was designed to make public some of the wealth of effective assessment practices at the University - practices which are known within the program group or school where they originate, but which may not be disseminated more widely throughout the University. The snapshots themselves were designed as self access learning resources for academic staff which could also be used as a professional development tool by academic developers in workshops on assessment. Each assessment snapshot is told from the perspective of the teacher and is made available in read on screen and downloadable PDF format. Snapshots are constructed as narratives or stories describing: contributors' motivation for developing the assessment activity; their approach to resolving the issues they faced in the particular assessment environment; the assessment activity or resource they developed; their evaluation of the efficacy of the assessment activity; their advice for others who may be interested in adapting the practice; and the learning about their own teaching and assessment practice which they gained from the experience.

Figure 1

Figure 1: An example of a Snapshot, from http://tdu.uws.edu.au/qilt/assessing.htm

Representations of practice using case studies

Representations of knowledge in the form of digital resources are recognised in the literature to be potentially useful to teachers to help them change their current practices (Sharpe, Beetham & Ravenscroft, 2005). Included as part of a wide ranging list of types of representations are stories, narratives, profiles, and case studies. For the purposes of this paper, we use these terms synonymously. These authors have also pointed out that often such knowledge has been difficult to access, and that some ways of representing this knowledge may be more effective than others.

Evaluating the project

In evaluating the project, we sought to answer the question "To what extent has the Snapshots project been successful in (a) disseminating effective assessment practice and (b) promoting reflection and discussion about assessment issues amongst academic staff?" In seeking answers to these questions, we drew on two sources of data:
  1. Feedback from academic staff obtained through two focus groups held in June 2007. Focus groups were used as a data gathering method based on their "...ability to produce concentrated amounts of data... directly targeted to the researcher's interests" (Morgan, 1997, p.13), within a relatively short time frame. Focus groups afford an opportunity for group interaction thereby allowing participants to make comparisons amongst their experiences and opinions. This interaction can provide researchers with "a valuable source of insight into complex behaviors and motivations" (Morgan & Krueger, 1993, cited in Morgan, 1997, p.15).

  2. Our own reflections on our experiences as academic developers who created the Snapshots project and have used the resource in various professional development contexts throughout 2006 and 2007.
Two other methods of gaining data about Snapshots usage had been envisaged at the start of the project but were less productive of feedback. The Snapshots web site provided an avenue for online feedback to the team and/or to the contributor of individual Snapshots via an email link, however little unsolicited feedback was received through either of these channels. A 'hit counter' had originally been planned for the web site but was not installed, partly due to the location of the Assessment Snapshots as one element in a larger site on teaching quality. A hit counter would have produced data on visits to the larger site hence was not considered to be valuable for the team's purposes.

Focus group feedback

A total of thirteen academic staff representing eight schools from across three colleges at the University participated in the focus groups. All teachers had been asked to prepare for these discussions by selecting and reading at least two of the Snapshots. For some staff, this was their first encounter with this resource, while others had previously made use of them in workshops or curriculum project contexts. Both focus group discussions were recorded and transcribed for analysis by the authors. Overall, staff in both groups found the resource useful in exposing them to assessment methods which were new to them and to approaches which could be adapted to their own context.

Focus group feedback was analysed against Rogers's (2003) attributes of an innovation - relative advantage, compatibility, complexity, trialability, and observability - that can influence an adopter's decision to take up an innovation. These attributes are crucial to the persuasion stage of Rogers's five-stage innovation-decision process that he argues characterises the adoption of all innovations. Quotations are used to illustrate staff responses in relation to each of these attributes.

  1. Cases can show the relative advantage of a new practice or approach to teaching and learning.
    Teachers in the focus groups saw clear advantages in the Snapshots for their own teaching practices. In commenting on a Snapshot describing an authentic assessment task in a capstone unit, one teacher identified multiple ways in which this Snapshot improved his understanding of what can be achieved through such an assessment: the development of strong group ownership by students through allowing a series of ad hoc groups in the initial weeks of semester before students self select their group for the remainder of the semester; improved business outcomes for the external client partly due to the lecturer's process of assembling student questions prior to their submission; and the respondent noted how the task forged better relationships with companies which "... might turn into opportunities for students for employment". A comment made by another teacher and supported by other respondents was that teachers tend to become 'socialised' into their school assessment practices: "... my thinking starts to narrow in terms of how we do assessment in the school. So having a look at [Assessment Snapshots] I thought was really useful."

  2. Cases can emphasise the compatibility of an innovation with academic staff values and current approaches to teaching.
    The focus group responses indicated that teachers were able to find something in the Snapshots they viewed that resonated with their own values and current teaching practices. For example, several respondents indicated strong interest in using strategies described in a Snapshot detailing how teacher education students, many of whom have a negative self concept as mathematics learners, had been motivated to change their attitudes to learning mathematics: "I like the strategy they used to motivate the students... developing a resource for use by others is something that tends to engage students, particularly when they're also being led to believe that perhaps maths is not as threatening as they first supposed it was...". Another teacher commented "... so I was interested in looking at how they engage the students in... an area that is not too popular with them. And I found their techniques really useful to reflect on and I also like the kind of outcome in terms of developing a resource".

  3. Cases can be used to acknowledge the complexity or level of difficulty potentially faced by staff taking up new teaching approaches, including the amount of time and effort required by staff to learn how to incorporate them successfully into their everyday practice.
    The Snapshots focused on the difficulties their authors had faced and resolved in their assessment practices. Several teachers in the focus groups valued this aspect of the resource. One comment singled out the idea of continuous improvement with regards to assessment task design, noting " the way [the teacher] had improved or changed it as he'd gone along, running this unit for some time I think, it was impressive". Another comment highlighted the complexity of assessment innovation in the light of large class sizes in this way: "In the case of X, I think she had very large numbers for that, didn't she? ... and it's possible that she can do that without being intimidated by the large numbers and has had success and been innovative...". A third focus group participant appreciated that the Snapshots provided her with sufficient detail to get "... a good idea of their motivation, how they planned it... a good idea of how it would actually work in practice and their evaluation of it - what the pitfalls are, what you need to look for, what resources you might need. I found that quite useful."

  4. Cases can also illustrate how an innovation or change to teaching practice can be trialled or experimented with on a limited scale.
    In designing case studies for professional development purposes, it is important to provide sufficient detail in the narrative to support teachers' being able to try out the innovation being described. To illustrate, from reading one of the Snapshots about the use of drama in Marketing education, one focus group teacher indicated that she had "picked up the use of drama and I actually do make my students present their story ideas before they're starting and engaging with the investigative process. And it doesn't end there. I realise that drama is also applicable in life generally." Another respondent found the community engagement model that featured in another Snapshot "quite handy for me because I could see that would work in some of the things that I do."

  5. Cases can provide opportunities for teachers to observe first hand the technical and pedagogical aspects of an innovation.
    This attribute of an innovation is related to what Wilson, Shulman and Richert (1987) have called the pedagogical reasoning of others. The Snapshots were designed to provide teachers' explanations of their reasoning behind their design and implementation of assessment tasks, and from feedback in the focus groups it seemed that they were successful in this aspect. Teachers in the focus groups appreciated the fact that the resource provided a window into individual teachers' rationales and approaches to assessment, providing, as one teacher commented "... a bit more of a context for how and whether I could actually apply some similar kind of approach in my own unit."

Using Snapshots in professional development contexts

As a team of academic developers we have used Snapshots as resources in workshops on assessment and blended learning and as triggers for discussion in working with staff on curriculum renewal projects. In a workshop on designing blended learning, the online negotiation role play Snapshot proved successful in guiding staff to examine in some detail how one 'local' example of a blended learning activity functions. We have taken this particular Snapshot and, using Littlejohn's LD_Lite planning tool (Littlejohn & Pegler, 2007), constructed a more micro-focused learning design which describes the role of the teacher, role of the student, resources, assessment, feedback, location and sequencing of the activity. The original Snapshot has thus been used to produce a further representation of practice. In other workshops, we have referred staff to the Snapshots collection, providing an explanation of why particular cases may be useful to them. In the context of curriculum renewal projects, several Snapshots have been used effectively as triggers for discussion about assessment issues as well as examples of what has been achieved locally within the constraints faced by all teachers at the University.

Conclusions

Overall, staff feedback supported our view that Snapshots are a potentially valuable resource for extending knowledge about options for assessing learning and understanding how particular assessment innovations work in practice. Looking back over the data gathered from the focus groups, there was a strong need expressed by the staff in these discussions for further support in relation to assessment, particularly in the design and marking phases of the assessment process. Staff requested that the Snapshots collection be expanded with a particular focus on further examples of creative uses of blended learning. The cases in this resource appeared to work well in convincing teaching staff who encounter them that innovation is feasible and achievable. The expanding Snapshots collection will continue to be used in workshops and curriculum projects and each of the Snapshots will be mapped onto the LD_Lite planning tool (Littlejohn & Pegler, 2007) mentioned earlier.

However, our experience throughout 2006 and 2007 has shown us that case studies such as these are not sufficient on their own, nor are teachers likely to use these resources unless they are mediated by someone else. Teaching staff used the Snapshots when academic developers actively facilitated this within particular contexts, for specific purposes. In line with current thinking about the relationship of resources to the professional learning experiences of staff and to extend the useability of Snapshots as self learning tools, future Snapshots need to be designed that will better facilitate the mediating of practice, or as Sharpe and Oliver (2007, p. 120) put it "moving from representations to interventions." Here, careful attention needs to be paid to the principles of how professionals learn and how academic developers' use of resources such as these Snapshots can be more effectively integrated and contextualised with teachers' everyday professional teaching practice. Possible solutions to this challenge lie in designing the resources to be more interactive, allowing staff to use and reuse them in different ways; or providing opportunities for staff to develop their own resources which they can share with their peers, debate, discuss, and interact with in social contexts.

Acknowledgements

The authors would like to thank staff at the university who contributed to the Assessment Snapshots project. We also acknowledge the generosity of staff who contributed to evaluating the project. The Assessment Snapshots can be viewed at http://tdu.uws.edu.au/qilt/assessing.htm

References

Littlejohn, A. & Pegler, C. (2007). Preparing for blended e-learning. London: Routledge.

Morgan, D. (1997). Focus groups as qualitative research (2nd ed). Thousand Oaks, CA: Sage.

Morgan, D. & Krueger, R. (1993). Successful focus groups: Advancing the state of the art. Thousand Oaks, CA: Sage.

Rogers, E. (2003). Diffusion of innovations (5th ed.). New York: The Free Press.

Sharpe, R., Beetham, H. & Ravenscroft, A. (2005). Active artefacts: Representing our knowledge of learning and teaching. HERDSA News, 27, 8-11.

Sharpe, R. & Oliver, M. (2007). Supporting practitioners' design for learning: Principles of effective resources and interventions. In H. Beetham & R. Sharpe (Eds.), Rethinking pedagogy for a digital age: Designing and delivering e-learning (pp. 117-128). London: Routledge.

Wilson, G., Thomson, R. & Malfroy, J. (2006). Gathering online representations of practice in assessment as a professional development tool: A case in progress. In Who's Learning? Whose Technology? Proceedings ascilite 2006 (pp. 893-897), 3-6 December, Sydney. http://www.ascilite.org.au/conferences/sydney06/proceeding/pdf_papers/p47.pdf

Wilson, S., Shulman, L. & Richert, A. (1987). 150 different ways of knowing: Representations of knowledge in teaching. In J. Calderhead (Ed.), Exploring teachers' thinking (pp. 104-124). London: Cassells.

Authors: This article received an Outstanding Paper Award at ascilite Singapore 2007 Conference, gaining the additional recognition of publication of a revised version with minor changes in AJET. The reference for the Conference version is:
Thomson, R. & Wilson. G. (2007). Promoting staff learning about assessment through digital representations of practice: Evaluating a pilot project. In ICT: Providing choices for learners and learning. Proceedings ascilite Singapore 2007. http://www.ascilite.org.au/conferences/singapore07/procs/elliott-k.pdf
Rosemary Thomson, Lecturer in Higher Education, Teaching Development Unit, University of Western Sydney, Locked Bag 1797, Penrith South DC 1797, NSW, Australia. Email: r.thomson@uws.edu.au

Gail Wilson, Manager, Teaching & Learning Services, Bond University, Gold Coast, Queensland 4229, Australia. Email: gwilson@bond.edu.au

Please cite as: Thomson, R. & Wilson. G. (2008). Promoting staff learning about assessment through digital representations of practice: Evaluating a pilot project. Australasian Journal of Educational Technology, 24(2), 143-149. http://www.ascilite.org.au/ajet/ajet24/thomson.html

[ PDF version of this article ] [ AJET 24 ] [ AJET home ]
HTML Editor: Roger Atkinson [rjatkinson@bigpond.com]
This URL: http://www.ascilite.org.au/ajet/ajet24/thomson.html
Created 22 Feb 2008. Last revised 22 Feb 2008.
Page access count