Australian Journal of Educational Technology
2002, 18(2), 147-168. 		AJET 18

Understanding common interviewing pitfalls: An evaluation of Legal Interviewing Skills

Kathy Lynch
Monash University
Gregor Kennedy
The University of Melbourne
This paper reports on the evaluation of a computer facilitated learning (CFL) resource which allows law students to identify common pitfalls in client-lawyer interviews. Using an evaluation framework developed by Alexander and Hedberg (1994) and Bain (1999), the CFL resource, Legal Interviewing Skills, was evaluated in three areas: interface design, learning processes and interview skills. Twenty-three postgraduate law students participated in the evaluation which involved the observation of students as they interacted with the resource, together with think aloud protocols, questionnaires and focus groups. Students were generally positive about the program and showed greater confidence in interviewing after using it. However, there are a number of areas where the program could be improved; namely, ease of navigation, the structure of learning tasks and the use of feedback to promote meaningful engagement with the learning.

Background

Like many other undergraduate law degrees, the law curriculum at Monash University has a subject that deals with the law of torts. A 'tort' in law describes a breach of duty between parties that have no explicit contract. The torts subject in the undergraduate course at Monash University covers a number of areas of law including torts concerning goods and land, negligence, vicarious liability and contribution between tortfeasors; and skills of a torts lawyer. More generally there are two aspects to the torts curriculum which students are required to become skilled in: content and practice. Students must be able to analyse and critically examine the current state of the law of torts from different perspectives and be able to explain how that law can protect the personal, proprietary and economic interests of people in Australia. In addition to understanding the content of the law, students must be able to examine a situation that has caused a person loss or damage and determine whether that person has a course of action which falls within the description of any of the torts covered in the course. Students must be able to interview and advise potential clients and, therefore must have refined legal interviewing and communication skills.

In recent years the tort law lecturers introduced optional, problem based, interviewing and negotiation exercises for small groups of torts students. A major difficulty for the course coordinators was finding effective methods for carrying out this form of skills training with large groups of students (300-400 students each year). The existing methods (such as supervision, role playing, answering queries, and participating as interviewees) for teaching interviewing and negotiation skills have proven to be very resource intensive. A shortage of suitable rooms, the demand on academics' time to be present for the skill acquisition, the absence of 'credit' allocated to the interviewing skill development, and the lack of available 'spare' hours in a student's day, are all issues that adversely affected the current teaching and learning of interviewing skills. Even though the law school and the legal profession recognise the need to teach these skills, most law students miss out on adequate training because of the lack of these resources.

Redesigning the learning environment

The traditional learning environment for the acquisition of interviewing skills in the legal arena has been, as stated previously, through the use of role play and negotiation exercises. Though these techniques are beneficial, large student numbers have hindered the skill acquisition and placed pressure on resources - both human and material. With this in mind, it was envisaged that taking the learning outside the traditional environment and placing the student in a simulated environment through the use of a multimedia interview scenario would enable the student to acquire the required interviewing skills and strategies, without taxing the already limited resources.

The aim was to produce a CD-ROM resource that would encourage the user to navigate through a situation in a manner with which they feel most comfortable, or most need, giving the user an opportunity to construct their own knowledge (Kennedy and McNaught, 1997). This learning approach is unlike traditional university learning, where content is structured, regimented and often linear. Laurillard (1993), and Kennedy and McNaught (1997) argue that giving students control over navigation complements differences in students' approaches to learning; some learners work and learn in a linear manner, others jump to the end and work backwards.

The philosophy underpinning the basic idea of the CD-ROM is that once students have navigated their way through the product they will be better prepared and more confident to participate in a role play exercise or a real world interview. Added to this, resources are not wasted or sitting idle whilst students achieve a basic understanding of interviewing skills and strategies.

CD-ROM design and rationale

The product Legal Interviewing Skills consists of a suite of three CD-ROMs. The first in the series covers material on generic legal interviewing skills, the second CD-ROM looks at an accident scenario from the complainant's perspective, and the third CD-ROM looks at the same scenario from the defendant's perspective. Like the development of many computer facilitated learning resources, the development of Legal Interviewing Skills has adopted a modular approach with the content of each module determined by academic staff. As such the development and implementation phases of the program's development cycle overlap. The first module of the program (CD-ROM-1) is stable, and is the focus of the investigation reported here. Evaluation conducted on this module will inform the development of the remaining modules in Legal Interviewing Skills.

The first module of the program is divided into four inter-related sections, with the user able to navigate between each section. The first section is an optional introductory tutorial detailing the basic skills required for conducting legal interviews. The second section of the program presents students with a nine minute video that shows an interview between a lawyer and a client. The video exemplifies 'poor' interviewing techniques. The third section presents a similar interview situation, but, in this case it exemplifies 'good' interviewing skills. The final section of the module is the 'interview evaluation' section in which students are asked to critically evaluate the poor interview. It is this final section of the CD-ROM that is at the crux of the student's learning.

The interview evaluation section is conducted by reviewing segments of the poor interview video, with each segment exemplifying a subset of the communication and micro-skills identified in the tutorial section of the product. The students' task is to identify which of the listed skills are evident in the segment of the poor interview video. The user can seek revision of a skill by clicking on the skill in the skill list, by returning to the tutorial section of the product using the menu option, or by toggling to the matching segment in the good interview video. Each segment of the poor interview video has named 'errors' which need to be identified. The user can move to the next video segment without identifying any or all of the skills. At any time the user can toggle to the same section of the good interview to make a comparison, clarify their judgement, or to review the section in a more favourable light.

After the user has identified and selected the corresponding skill in the skill list they can request feedback. The feedback is simple in that a tick is given next to the skill name if they selected correctly. The student can de-select or select skills from the skill list, and request feedback any number of times. The task is cumulative in that students are exposed to a greater variety of skills as they progress through the evaluation exercise.

The design of the Legal Interviewing Skills CD-ROM was influenced by a number of educational concepts: situated learning; constructivism; and learning by negative example.

Situated learning

The teaching practice of 'situated learning' is one in which students are placed in a realistic situation, in this case, the initial client-lawyer interview. The environment created as the students worked through the Legal Interviewing Skills CD-ROM was to give the students an insight into legal interviewing and feel as if they actually were part of the interview; a feel of presence in the interview. Situated learning is commonplace in educational resource material, and is closely associated with constructivism in which students construct their own knowledge from multiple perspectives. Navigating and working through a situated learning environment, '...creates a zone of proximal development' (Vygotsky, 1978:102). The Legal Interviewing Skills CD ROM, like many others before it, "is made as rich as possible to enhance students' ability to construct knowledge and resolve conceptual difficulties." (Kennedy & McNaught, 1997).

It was deemed important by the developers to make every effort to 'immerse' users in the interview. Adding to the realism were professional actors, actual torts scenarios, and realistic legal contexts. The use of the high quality audio and video was deemed necessary to depict individual communication and micro-skills. These skills cover both verbal and non-verbal behaviour, and are often quite subtle. The developers thought that it would be difficult to capture these subtleties using other media (eg., an interviewer's condescending attitude). Although perhaps inconsistent with the realistic feel that was being developed, users could toggle between full screen and reduced screen video (one third of screen size) to suit their own viewing preference.

Constructivism

Legal interviewing requires much investigation and filtering of data to discover the facts of a case. There is never a direct line or path to data or an answer; the lawyer is required to actively traverse the many variables in a given situation. The CD-ROM was designed to simulate this, empowering the students to be active rather than passive learners through the use of alternative paths through sections of the application. This gives the student the freedom to reflect, go back, rediscover and construct their own learning. Ramsden (1991) argues that multiple paths through an application provide students with the ability to reach higher levels of cognition. The design of the CD-ROM also provides students with a suggested path to follow.

Negative example

The use of a poor and a good example of interviewing practices was to emphasise, without confusion, what not to do in an interview. There were two very different actors used in the videos. The poor interview skill actor was disorganised, discourteous and distracted. The good interview actor was well organised, polite and on task. The client and the base script were the same for both videos. The poor interview was the first one the learner encountered, and as such set the scene for learning by the negative example situation. Learning by negative example is centred on a situation (or example or hypothesis) that is incorrect (Winston 1975), and allows the learner to identify the differences between the example and the concept. Winston (1975) concludes that positive examples depict a generalist concept, whereas negative examples depict a specialised example of a concept. The functionality of the product allowed the user to toggle between the two videos, encouraging a 'learning by comparison' process. The negative or poor video over-emphasised the inappropriate manner in addressing an issue, and the good or positive interview video exemplified how it should be done. This allowed the user to clearly identify opposite ends of the continuum, and clearly identify their differences. Humour was paramount in the poor video but not in the good interview video, because it would have been inappropriate and would have diminished the realistic situation that was being depicted.

Evaluation model and questions

The focus of this evaluation study was guided by the evaluation framework proposed by Alexander and Hedberg (1994) and extended by Bain (1999), which was developed with specific reference to computer facilitated learning programs. This framework has four primary phases: analysis and design, development, implementation and institutionalisation and its core features are similar to other evaluation frameworks (see Draper, Brown, Henderson & McAteer, 1996; Reeves, 1989, 1993). The focus of the analysis and design phase is the current curriculum and teaching and learning practices within it. In what has typically been called a 'front end evaluation' or 'needs assessment' (Flagg, 1990; Reeves, 1993), an evaluation is carried out on how a computer based innovation can be aligned with the needs of students and the objectives of the curriculum. In the development phase of Bain's (1999) framework the evaluation is formative and as such focuses on refining and improving programs. In this phase Bain (1999) highlights the need to investigate the learning environment and processes. The implementation phase is student centred and calls for summative evaluation of both students' learning processes and outcomes. Finally, the institutionalism phase considers long term evaluation of impact and whether there have been lasting benefits (or disadvantages) as a result of the program's implementation. The evaluation reported here falls primarily into the development phase of Bain's (1999) framework.

Within the development phase the evaluation was formative, concentrating on students' learning processes as they navigated their way through the program. The evaluation for this initial development was focused on students' perceptions and use of the product, with particular emphasis on the video and navigational structure, and did not specifically include learning outcomes because the CD-ROM is not currently embedded in the course it was designed for. Nevertheless, although outcomes were not assessed directly, we planned to see whether students' appreciation of these skills changed as a result of working through the Legal Interviewing Skills program. It was envisaged that the evaluation would highlight areas in the program's initial development which could be refined and improved in future developments. We also assessed students' confidence about the nature and use of these skills and asked students to reflect on whether the program assisted them in recognising common pitfalls and problems in legal interview situations. Future evaluations will examine students' learning processes in terms of the cognitive strategies they employ while using the program (such as general learning strategies, critical thinking and reflection).

In summary the specific focus areas of this evaluation were:

Evaluation Method

Sample

Twenty-three students undertaking the Postgraduate Diploma in Legal Practice, Skills and Ethics at Monash University participated in the evaluation. Although the program was developed with students in the early years of an undergraduate law degree in mind, the lecturer in charge of the postgraduate program thought that it would be a useful addition to the postgraduate diploma. The sample was considered suitable for this reason, and it also was expected that students with more experience in law and interviewing would make a valuable contribution to the evaluation. Ten males (43%) and thirteen (57%) females participated in the evaluation and their ages ranged from twenty-three to sixty-one, with the average age 35.52. Of the sample, almost half (43%) had had previous interviewing experience, and two (9%) had experience as a lawyer. The fact that many of the participants had previous interviewing experience was not known at the time of selecting the sample, and was initially thought of as a disadvantage as far as the evaluation was concerned. It was discovered during analysis of the findings that much of this experience would be better named as 'inexperienced interviewing', with only two of the participants having any experience in conducting legal interviews.

Data collection

The class was divided into two groups for the evaluation which took place in week eight of second semester, 2000. Students worked individually with the program over a one-hour period in a computer lab. The data collected in this investigation derive from both quantitative and qualitative techniques and thus represent a mixed method approach (Reeves & Hedberg, 2000). Various data collection techniques were used including observation, think-aloud protocol, think-write protocol, questionnaires and a focus group. Table 1 presents a summary of the methods employed in this evaluation and details of the specific data collection techniques used in each focus area of the evaluation.

Table 1: A summary of the data collection techniques employed in each focus area of the evaluation


Interface
design		 Learning
processes		Interview
skills
Observation XX
Think aloud protocols XX
Think write Protocols XX
Pre-task questionnaire

X
Post-task questionnaireX
X
Focus groupXXX

Data collection was carried out in four stages. Before students began working with the program a pre-task questionnaire was administered. While using the program participants were observed and were asked to complete a think-write protocol. A subset of participants was asked to undertake a think-aloud protocol. After spending approximately an hour on the program students were asked to complete a post-task questionnaire. Finally, a focus group session was conducted with a small subset of students at the conclusion of the session. Details of the data collection techniques employed in this investigation are outlined below.

Observation

Two methods of observation were employed simultaneously in the investigation: real time and deferred. Four trained observers carried out real time observation of eight students completing the program. In addition, deferred observation was conducted with four students using a video while they worked through the program. Guidelines were used to structure the observations so that there was some consistency across the observers. These guidelines fell into four categories of technical, interface, useability problems and learning processes.

Think-aloud and think-write protocols

In each group, two students used tape recorders to construct an oral account of their progression through the program. It was expected that the use of the think-aloud protocol would be useful in this aspect of the evaluation as this think-aloud protocol encourages participants to articulate some of the cognitive processes they are employing. All other students were encouraged to write down their thoughts of the program as they worked through it. All student wore head sets during their interaction with the product.

Questionnaires

The pre- and post-task questionnaires contained both scaled response items and open ended questions. Quantitative items employed five-point Likert scales and were labelled from "strongly agree" to "strongly disagree". The pre-task questionnaire administered before students began using the program focused on students' confidence in conducting client interviews. The post-task questionnaire was comprised of three sections. The first section was based on the questionnaires of Reeves and Harmon (1993) and Kennedy (1998) and focused on the interface design of the program and the design approach adopted by the developers. The second section asked students about the interview skills that were central to the program. These items asked students to indicate the degree to which the program helped them identify specific communication and micro-skills (such as body language, closed questions and empathy) and common problems or pitfalls in the client interviewing process. The final section replicated the pre-task questionnaire and asked students to reflect on their confidence in conducting an interview now that they had worked through the program.

Focus group

Seven students from the first evaluation group were asked to participate in an informal, but semi-structured focus group at the conclusion of the evaluation session. The first author led the focus group in which three other researchers participated. The focus group was audiotaped and guided by a number of prepared questions. The questions were based on usability and interface design including the use of audio and video, students' perceptions of how the program affected their understanding of interview skills, what strategies students used while using the program, and areas where the program could generally be improved.

Results

The results presented below are organised into three main sections (interface design, learning processes and interview skills) reflecting the three evaluation foci of this investigation. Data collected using the methods outlined in Table 1 are reported in each of these sections where appropriate. The scaled responses to questionnaire items have been collapsed into a three-category classification ("agree", "neutral" and "disagree") to assist with interpretation.

Interface design

Results regarding the evaluation of interface design were divided into two areas: the use of audio and video, and navigation and usability.

Use of audio and video

The use of audio and video multimedia elements in the program was singled out because the developers were very keen to see if the use of high quality audio and video, and full screen video, was seen as appropriate and valuable by students. In the questionnaire, three items directly related to the use of video and audio within the program. All students indicated that they thought that the use of video and audio was appropriate and that the aesthetic quality of the program including audio and video was high. In the focus group students were asked more generally about their impressions of the use of video in the program. The response was generally positive with students commenting on both the technical quality of the video and how the video was particularly useful given the content focus of the program.

Students' positive perceptions of the video were also recorded in their general comments about the program. Approximately half of the students mentioned it as one of the program's 'best' aspects (see Table 2).

In response to the question "Did you use the 'full screen' for viewing the video?" the majority of students (78.3%) indicated they used the full screen video while 17.4% said they did not (one student did not respond to this question). A number of students (4) indicated they used the video at full screen only after being told that it was available. Written responses indicated that students liked the full screen video because it was less distracting and made the scenes feel realistic.

Table 2: Students coded responses to the question
"What were the best aspects of the program?"

Video
(n = 11)		Generally good5
Better than a text-based explanation2
The ability to review2
Comedic2
Interview
skills
(n = 11)		Ability to compare good and bad interviews6
Seeing the interviewers skills (questioning and legal advice)2
Better for students with no interviewing experience2
The good explanations of interview skills1
Able to see all 20 micro-skills in the evaluation section1
Other
(n = 9)		Interactivity3
Able to go at own pace2
Good actors2
General presentation of the program2

Open responses revealed that six students who said they had used the full screen video indicated they actually preferred to toggle between the full screen and the small screen video depending on their needs. These needs seemed to hinge on the student's location (home or university) when working through the product, which section of the program they were in, and the availability of controls while viewing the video. This indicated that users liked being in control of how the video was displayed.

In the focus group, there was a great deal of discussion about how students used, and would like to use, the video to investigate micro-skills and legal interviewing techniques. The general consensus was that the video quality was excellent and gave a very realistic feel to the resource. However, while students were very positive about the quality of the video and saw it as a useful tool in the teaching and learning of interview skills, they did express a number of reservations about how they interacted with the video within the structure of the program. Although related to the use of video, these results will be reported below in reference to students' learning processes.

Finally, three of the seven students in the focus group expressed feeling frustrated with the audio, or more precisely with the lack of audio, as they navigated through the program. This was also evident in the data collected from the think-write protocol and the observations. Students expected the audio to be present from the beginning of the resource and commented that they thought their headsets were faulty because no sound was present. Being a multimedia product, sound is expected, but for this product it was only present in the video.

Navigation and usability

The questionnaire contained three items directly related to navigation together with an open response item on the navigational structure of the program and one item on usability. The data that emerged from these questions were somewhat contradictory. It can be seen from Table 3 that the majority of students (57%) found it easy to navigate around the program and approximately 70% of students said they found the program easy to use. However, over half the students (61%) indicated there were times when they were confused about their location in the program and a number of students (36%) indicated they had trouble locating information they had previously visited.

Table 3: Students perceptions of the navigation and usability

Navigation and usability questionsMean
(SD)		Percentage
AgreeNeutralDisagree
I found it easy to navigate my way around the program 2.61
(1.08)		572618
There were times when I was confused about my location in the program 2.52
(1.24)		63927
Sometimes I found it difficult to locate pieces of information I had previously located 3.26
(1.29)		361450
I found the program easy to use 2.26
(0.92)		691713

Only nine students responded to the open response navigation question in the questionnaire and the responses did not shed much light on why some students were confused about their location within the package. This was of concern to developers as they had tried to make the navigation simple and intuitive. Problems with the navigation structure were also obvious during observations which showed that students were sometimes not entirely sure about where they had been, where they were going, or how to access information they had once seen. In the open ended responses describing the 'worst' aspects of the program (see Table 4), a number of students offered recommendations about how to reduce navigation confusion, such as including a back button in some sections and having a system that alerts users to where they have been already in the program. Another recommendation suggested by two of the focus group participants, and agreed to by the others, was to include an overview of the resource, a suggested pathway through it and the intended outcomes. From this feedback it became evident that with minor modifications, such as consistent use of 'back' and 'forward' arrows and menu structures, and by adhering to consistent guidelines, major navigational problems could be overcome. Several students (4) indicated that navigation became easier as they progressed through the program.

Students' learning processes

Our investigation of students' learning processes was carried out primarily using qualitative data collection techniques (observation, focus group and protocols). Data collected using these methods were supplemented with open responses from questionnaires. The most overwhelming response in both focus group and think-aloud data related to the way students negotiated the 'interview evaluation' section of the program. As mentioned above, in this section a student's task is to evaluate a number of interview segments and to identify aspects of the interview that have been poorly conducted by the lawyer. For each segment, students are asked to identify (by checking a box) all 'problem' skills in evidence before moving on to the next segment. After indicating which skills they feel are problematic in the segment, students can obtain feedback as to whether their selections are correct or incorrect. Further information can be retrieved by hyperlinking to an explanation of each of the identified micro-skills. Accessing this information is optional and students may choose to skip this activity and progress to the next video segment.

Results from observation and the focus group session indicated that students found the interview evaluation task frustrating and tedious, especially when they could not easily identify all the interviewing problems for a particular video segment. It is interesting to point out that the data collected using the think-write protocol was minimal. We put this down to the users being so pre-occupied with the program that they didn't take time out to move their hand from the mouse to the pen. The think-aloud students expressed frustration when they repeatedly selected a skill incorrectly. They 'demanded' useful feedback to guide them to the correct response. Although students were not restricted by the application, some students felt they could not move on until they had correctly identified all the answers.

Possibly as a result of this frustration, a number of students indicated that the task soon became one of 'getting the right answer' rather than reflecting on the use of appropriate and inappropriate interview techniques. A number of students in the focus group and those participating in the think-aloud protocol suggested that their completion of the evaluation section became more an exercise in 'trial-and-error' or 'systematic guessing' to get the right answer, rather than reflecting on how particular problems manifest themselves in interviews. As a result, the goal of the learning task became the identification of the full complement of problem skills for each interview segment rather than how these skills affect the interview process.

Despite negative reactions, 65% of students agreed that the program encouraged them to reflect on the content area and only three students (13%) thought this was not the case.

Students also expressed concern about the provision of feedback in the program generally and in relation to their learning and understanding specifically. Although students were told whether they had correctly or incorrectly identified particular micro-skills, they were not provided with extensive feedback about why their choices were correct or incorrect. Although half the respondents thought the feedback was sufficient, a third indicated that they felt the feedback was not meaningful to them and 44% suggested the feedback did not help them understand where they went wrong. It seems that the process of 'systematic guessing' that many of the students reported using, may be linked to the failure to provide adequate feedback. That is, if more meaningful feedback were provided to students detailing why an answer is wrong, they may be encouraged to make a more considered second attempt at the problem rather than adopting a 'hit or miss' approach to simply obtain the 'right' answer.

Another reason why a number of students felt the feedback was inadequate was that they did not necessarily agree with the expert's opinion of what constituted a 'problem' micro-skill in the context of the interview. This was borne out in students' comments on the 'worst' aspects of the program where three students commented about disagreeing with the answers (see Table 4). This finding may be a function of the more experienced sample used in this evaluation.

Table 4: Students' coded responses to "What were the worst aspects of the program?"

Learning
process
(n = 10)		Disagree with the answers, no justification given3
Pre-occupation with faults with no opportunity to identify good features3
Evaluation exercise not complex enough2
Evaluation a bit monotonous2
Interface
design
(n = 7)		Hard to know where to go next4
Hard to know how to get back2
Display right answer sooner1
Other
(n = 3)		No live links1
Assumes a certain level of computer knowledge1
Bad interviewer was too crass, rude and obnoxious1

Although it was interesting to see that students used a number of strategies to complete the interview evaluation, one strategy that students did not use came as a surprise to the developers. Students had the ability to hyperlink from any of the micro-skills within the evaluation exercise to detailed information about that particular skill, but they rarely used it. The developers assumed that students would use the hyperlink function to re-visit the summaries of general interviewing skills to assist them with their analysis of the interview segment. However, most students attempted the evaluation exercise without this information and seemed to rely on their prior knowledge, the knowledge they had previously acquired from the resource, or the ability to toggle between the good and poor interview videos. When participants were questioned about this in the focus group, many said the reason they did not use this function was they did not know it was available. Clearly this has implications for the instructional and interface design of this section of the program.

Another aspect of students' learning processes related to the program's general structure and how students would prefer to interact with it. It emerged in the focus group that the majority of participants would have preferred to have completed the evaluation section of the program first, before accessing general information on how to conduct a legal interview and micro-skills. One student in the focus group also suggested that after completing the evaluation section, they would have liked to label segments of the complete video interviews as good and poor, rather than having the videos labelled for them.

Table 5: Degree to which students indicated the program helped
them identify specific interviewing problems/skills.

Specific problems or skillsPercentage
AgreeNeutralDisagree
Poor introduction/rapport 83180
Discourtesy 78220
Empathy lacking/misplaced 65304
Condescending attitude 78144
Judgmental attitude 83170
Gender/cultural bias 533017
Poor preparation 78174
Closed or narrow questions 74179
Leading questions 691714
Complex questions 54369
Irrelevant questions 69179
Confusing jargon 74149
Implications overload 572914
Opinion mistaken for fact 522424
Unwarranted assumptions 621919
Practical issues overlooked 82145
Pre-occupation with litigation 9190
Options not canvasses or explained 86140
Financial and personal costs ignored 9190
No clear instructions sought 73234

Finally, a number of participants said they were disappointed that the evaluation section only focused on 'poor' interview skills rather than a balance between 'poor' and 'good' skills. Although six participants appreciated being able to compare poor and good interviews (see 'best' aspects of the program Table 2), three participants mentioned that one of the worst aspects of the program was that the evaluation section focused only on the poor interview (see Table 4). This issue was discussed in the focus group.

Interview skills

The final focus of this evaluation was to investigate whether the program affected students' confidence in conducting a legal interview. Participants were asked whether the program helped them identify specific interviewing problems and micro-skills. The twenty specific skills that were targeted are shown in Table 5. The results show that for each skill the majority of students indicated that the program helped them identify these skills in the course of an interview. While a positive result, it is perhaps not surprising given the focus of the program was in this area and that the sample were postgraduate students, of whom almost half had previous interviewing experience, even if minimal. It remains to be seen whether students' ability to identify these problem areas in a legal interview correlates with their successful management of these problem areas when conducting a real legal interview. Areas which may be of particular concern are gender and cultural bias, the use of leading and complex questions, overloading clients with implications and mistaking opinions for facts, as these are areas some students had difficulty identifying.

Table 6: Students' confidence in conducting a legal interview
before and after using the program (n=23)


Not confidentNeutralExtremely confident
Pre-test3 (14%)10 (43%)10 (43%)
Post-test1 (4%)7 (30%)15 (65%)

In the pre-task questionnaire participants were asked how confident they would be conducting an interview with a client about a fencing dispute with an elderly neighbour. Participants recorded their responses on a Likert scale that was labelled "not confident" to "extremely confident". This same question was asked of participants once they had completed the program. A comparison of participants' confidence before and after using the resource is presented in Table 6. It can be interpreted from these figures that participants generally reported being more confident about their abilities in conducting legal interviews with clients after having completed the program, with only one participant reporting not being confident about their interviewing skills after completing the session.

General discussion and conclusion

The development of the Legal Interviewing Skills CD-ROM was initiated through the need for undergraduate law students to have the skills to enable them to interview and advise potential clients effectively. Together with communication skills, refined skills in conducting legal interviews are paramount. Though the law school at Monash University and the legal profession recognise the need to teach these skills, most law students miss out on adequate training because of the lack of resources. To this end, the Legal Interviewing Skills product precedes any role play or negotiation exercise placed before the students. This reduces the strain on resources and gives the students an opportunity to gain confidence in interviewing prior to any mock exercises, or real world experiences.

In the development of educational multimedia, users (students) need to be involved in the iterative design process to ensure that the product is usable and effective (Kennedy and McNaught, 1997). This iteration is often the result of feedback from an evaluation such as the one conducted for the Legal Interviewing Skills. The general findings of the evaluation of the Legal Interviewing Skills are similar to formative evaluations conducted by other multimedia developers (such as quoted in Kennedy and McNaught, 1997 - section 3.17), and have and will continue to influence the development of this product, as multimedia development is an iterative design process involving students to produce "more usable and effective IMM." (Kennedy and McNaught, 1997).

At the nucleus of the design of the CD ROM are three key concepts: situated learning; constructivism; and learning by negative example. These concepts were juxtaposed to three areas of importance to the developers: interface design; students' learning processes; and students' learning outcomes. To assess whether the multimedia product portrayed or delivered any of these, this evaluation was conducted.

The specific focus of the evaluation was concerned with the development phase of Bain's (1999) evaluation framework. As a result, this evaluation sought to gather information which would assist with the refinement and improvement of the CFL program Legal Interviewing Skills. Consistent with this goal, the evaluation produced both positive and negative findings in the three focus areas of interface design, students' learning processes and interview skills.

The research sample for the evaluation was a group of graduate students undertaking a postgraduate program in law, with 9% having previous experience as a lawyer. A questionnaire was used before and after the participants used the product. During the participants' use of the product, they were observed in 'real time', others were video taped for 'deferred observation', and all participants either used a pen or microphone to record an annotated dialogue of their experiences as they negated the product. In addition, several participants were involved in a focus group session.

In the area of interface design participants were overwhelmingly positive about the use of video: both its technical quality and its appropriate use, given the content of the program. A number of participants commented that video was an easy way to see common pitfalls in conducting interviews. This validates the design focus of the development team, which employed video because it was seen as not only a way to make the program engaging and situated but also as the most appropriate medium to display interview and micro-skills. Most participants made use of the full screen video option and many participants toggled between full and reduced screen video. However, it is clear that further attention should be given to making video controls available in the full screen mode. Likewise, attention should be given to the use of audio controls throughout the program. Confusion could have been reduced by allowing participants to control the level of audio and by clearly indicating when an audio track was playing.

An area of the interface, and the program generally, that participants did report having trouble with was navigation. Despite the developers' best efforts to make the interface intuitive, a number of participants reported being confused about their location in the program and about how to get to where they wanted to go. Although a number of participants reported having less trouble as they became more familiar with the interface, it is clear that some redevelopment is needed in this area. Participants offered a number of suggestions which will be considered by the development team. These recommendations involve the consistent use of buttons and menus and a gauge to alert students whether they have been in a particular area of the program previously. With relatively minor modifications a substantial improvement in the area of "ease of navigation" is expected.

The evaluation of students' learning processes was found mainly in the "interview evaluation" section of the program. We discovered that participants found this section somewhat tedious. There are a number of possible explanations for this. The developers underestimated how persistent law students would be in this section. That is, participants refused to move on to the next video segment until they had satisfactorily completed the section they were in, which meant finding all the correct answers. This desire for success led many participants into a learning strategy of "systematic guessing". We hope this problem will be alleviated by providing more adequate feedback to students (an aspect of the program they were critical of) in order to encourage greater reflection and more meaningful interaction with the evaluation section. It may be possible to achieve a similar outcome by restructuring the task itself so that students are given a limited number of chances to select the correct answer.

Participants, on the whole, did not use the hyperlinked explanations of micro-skills in the interview evaluation section. When asked why this was the case, many participants said they did not know this option was available. Clearly this is an area for redevelopment and would become more important if the general structure of the program was changed as many participants suggested. For example, if students were encouraged to evaluate interviews as their first task, rather than after completing a general overview of micro-skills, students may have a greater need to link to unfamiliar terms and micro-skills. The restructuring of the program is an issue which will be considered by developers. The rationale for placing a general overview before the interview evaluation section was that students may not be familiar with many of the terms associated with communication and micro-skills. However it may be more worthwhile to introduce students to these terms in the context of an interview, thereby promoting an intrinsic motivation to understand their relevance.

Finally, the evaluation revealed preliminary evidence that the program was on the right track in terms of achieving its learning objectives. These were associated with promoting students' awareness of communication and micro-skills associated with conducting a legal interview. The content developers were particularly interested in getting students to appreciate common problems or pitfalls in interviews. The results suggest that students are able to identify these pitfalls and are generally more confident about conducting a legal interview after completing the program. However, as noted above, these are preliminary results only, focussing on students' perceptions of their own skills. Further investigation is required to determine whether the program has an impact on how students actually conduct interviews.

The development team will use the findings of this evaluation, not only to edit and improve the first module in the Legal Interviewing Skills suite, but also to inform the development of the remaining two modules. The evaluation has focussed developers' attention on critical areas of development: navigation; the structure of the program generally; individual learning tasks; and the provision of feedback. The development team plans to implement the resource, module by module, then in its entirety after further development, evaluation and testing with undergraduate law students at Monash University. It is hoped that Legal Interviewing Skills will fill a gap in the law curriculum - caused by overcrowding and large class sizes - and become an invaluable teaching and learning resource.

References

Alexander, S. & Hedberg, J. (1994). Evaluation of technology-based learning: Which model? In Beattie, K., McNaught, C. & Wills, S. (Eds), Multimedia in Higher Education: Designing for Change in Teaching and Learning. Elsevier: Amsterdam.

Bain, J. (1999). Introduction: Learning centred evaluation of innovation in higher education. Higher Education Research and Development, 18(2), 165-172.

Draper, S.W., Brown, M.I, Henderson, F.P. & McAteer, E. (1996). Integrative evaluation: An emerging role for classroom studies of CAL. Computers in Education, 26(1-3), 17-32.

Flagg, B.N. (1990). Formative Evaluation for Educational Technologies. Lawrence Erlbaum.

Kennedy, D. M. & McNaught, C. (1997). Design elements for interactive multimedia. Australian Journal of Educational Technology, 13(1), 1-22. http://www.ascilite.org.au/ajet/ajet13/kennedy.html

Kennedy, G (1998). CAL Formative Evaluation Questionnaires. Biomedical Multimedia Unit, University of Melbourne.

Phillips, R., Bain, J., McNaught, C., Rice, M. & Tripp, D. (1999). Handbook for Learning Centred Evaluation of Computer Facilitated Learning Projects in Higher Education. Perth, Murdoch University and ASCILITE. [verified 21 Jul 2002] http://cleo.murdoch.edu.au/projects/cutsd99/

Laurillard, D. (1993). Rethinking University Teaching: A Framework for the Effective Use of Educational Technology. London: Routledge.

Parkes, R. J. (2000). On the subject of pedagogies. Australian Association for Research in Education Paper 272. [viewed Oct 2001, verifed 21 Jul 2002] http://www.aare.edu.au/00pap/par00272.htm

Ramsden, P. (1991). A performance indicator of teaching quality in higher education: The Course Experience Questionnaire. Studies in Higher Education, 16(2), 129-150.

Reeves, T. C. (1989). The role, methods, and worth of evaluation in instructional design. In K.A. Johnson & L.J. Foa (Eds), Instructional design: New alternative for effective education and training. pp. 157-181. Macmillan.

Reeves, T. C. (1993). Evaluating technology-based learning. In G.M. Piskurich (Ed), The ASTD Handbook of Instructional Technology. pp. 15.1-15.32. McGraw-Hill.

Reeves, T. C. & Harmon, S. (1993). User Interface Rating Tool for Interactive Multimedia. Georgia Tech. [viewed Aug 2000, verified 21 Jul 2002] http://mime1.marc.gatech.edu/MM_Tools/UIRF.html

Reeves, T. C. & Hedberg, J. G. (2000). Evaluating Interactive Learning Systems. Athens GA: University of Georgia, College of Education.

Winston, P. (1975). Learning structural descriptions from examples. In P. H. Winston (Ed), The Psychology of Computer Vision. New York: McGraw-Hill. pp. 157-209. [viewed Apr 2001, verified 21 Jul 2002] http://www.rci.rutgers.edu/~cfs/472_html/Learn/Winston.html

Authors: Kathy Lynch, Centre for Educational Multimedia, Monash University
Gregor Kennedy, Biomedical Multimedia Unit, The University of Melbourne

Please cite as: Lynch, K. and Kennedy, G. (2002). Understanding common interviewing pitfalls: An evaluation of Legal Interviewing Skills. Australian Journal of Educational Technology, 18(2), 147-168. http://www.ascilite.org.au/ajet/ajet18/lynch.html

[ AJET 18 ] [ AJET home ]
HTML Editor: Roger Atkinson [rjatkinson@bigpond.com]
This URL: http://www.ascilite.org.au/ajet/ajet18/lynch.html
Created 16 Aug 2002. Last revised 27 Aug 2002.
Page access count