Creating web-integrated learning environments: An analysis of WebCT authoring tools in respect to usability

This study focuses on the creation of web-integrated learning environments, using WebCT [http://www.webct.com] More specifically, it targets the user, as a course designer, who interacts with several authoring tools in order to produce educational scenarios. Two factors were measured in terms of their significance within the user-interface interaction: user satisfaction and control. A positive linear dependence between users' perception of satisfaction and control was observed for a variety of WebCT tools. In conclusion, this positive correlation could be explained both quantitatively, and qualitatively in terms of a tool's perceived aspect of usability.

Introduction

This study highlights course designers using WebCT and their interaction with the warranted authoring tools, fundamental for creating web-integrated learning environments. Therefore, a closer look is given to the utilisation of various WebCT tools that positively influence these course designers by evoking high perceptions of control and overall satisfaction. More specifically, this study combines these perceptions of user control vs. overall satisfaction and applies them to further analyse the aspect of usability according to specific WebCT authoring tools. These two variables, control and satisfaction were measured both quantitatively and qualitatively, providing the basis for the experimentation, analysis, and discussion included in this research study. Additionally, the attribution of usability is explained by considering these two elements in the user-interface interaction.

Fundamentals of a web-integrated learning environment

For the past two years, we have explored different authoring tools, both commercial and non-commercial that facilitate the creation of web-integrated learning environments. WebCT, was a particular tool that attracted our interest for two reasons. First, was its ability to combine single function Internet based tools to create varies multifunctional tools that could suit our particular teaching and learning purposes at TECFA. Secondly, WebCT provides an authoring environment that portrays a "desktop model," where tools are available and unified from within a common restricted web based interface. As a result, we have implemented WebCT in one of the teaching and learning modules at TECFA. In the course, the students take the seat as the course designers and have the opportunity to author their own unique learning environments. Therefore, studying these two aspects, interaction between course designer and WebCT authoring tools, was of great interest to us in our research domain.

Measuring usability in the human-computer interaction

Applying these definitions, there has been a continuing effort to further define all of the multiple dimensions of usability. More concretely, a set of usability principles has been explained and agreed upon by several HCI researchers, including Shneiderman (1986), Norman (1988), Nielsen (1993), and Shackel (1991). Each of their approaches defines the concept of usability by naming examples of system properties or qualities that influence usability (ie. consistency, presentation, error handling and recovery, memory load reduction, task match, flexibility, ...). These functional operations of usability are measured only in terms of the system's operation.

Besides from the usability of the system's operation, it also equally important to consider the point of view of the user in regards to usability. Therefore an attempt to study the experience of a user by introducing approaches that link usability and human emotion can also be introduced. These include, both the attitudes and perceptions of the user in terms of satisfaction and control.

Further, user satisfaction is explained as the affective attitude towards a particular computer application by a user who interacts with the application directly. Along with this, there is also a practical assumption that user satisfaction with an information system results in some positive change in user behaviour, resulting in increased effectiveness. Additionally, user participation in design was found to be positively correlated with user satisfaction (Doll and Torkzadeh 1988).

On the other hand, user control in a workspace concerns those aspects of interface and instructional design that provide users with functions in order to choose the nature of the feedback they receive the navigation paths through the content, the content to be engaged in and the pace of the engagement (Reeves, 1993). It has been suggested that giving the user control over system, process, and content generally increases intrinsic motivation to learn (Becker & Dwyer 1994).

Perception of usability

Fishbein and Ajzen's (1975) theory of reasoned action shows that beliefs influence attitudes which lead to intentions and therefore generates behaviours. Davis (1989) explains this theory of Technology Acceptance Model, or TAM, in relation to both a user's attitude toward use and his/her behavioural intention to use. Attitude towards use is the user's evaluation of the desirability of employing a particular information systems application. Behavioural intention to use is a measure of the likelihood a user will employ the application (Ajzen and Fishbein, 1980).

An important topic of concern for these theories is the issue of "perception," experienced by users when using authoring tools in a system. This study will focus on the users' "perception" of control and "perception" of satisfaction when using different tools. It will be considered that course designers construe and create their own unique way of interacting with tools and construct their own interpretation of usability based on their own perception of satisfaction and control.

The present study

For this study, six specific course design tools were chosen and tested in the experiment. These tools reflected important pedagogical components for effective course design and delivery for web-based online courses conducted at a distance. These six selected tools are detailed in Table 1.

Continuing, the purpose of this study was to investigate the following questions regarding the use of WebCT authoring tools:

1. What is the variation of a course designer's perception of control and satisfaction between different types of WebCT authoring tools?

2. Do some WebCT authoring tools have a higher degree of usability than other types of tools?

3. What are the implications of these results for course designers using WebCT?

Method

Subjects

The targeted population of the experiment included a sample of individuals, representing higher educational institutes, who had developed web-integrated learning environments using WebCT. More specifically, the sample profile included 35 participants who took part in the experiment; 13 professors, 3 teaching assistants, 7 graduate students, and 12 computer administrators. 46% of the participants had previous experience using other environments similar to WebCT, and 54% had never worked with similar environments. The participants differed widely in their level of computer experience using WebCT; 14% had only been working on WebCT for less than 1 month, 15% had worked with WebCT for one to six months, 31% worked with WebCT for six months to one year, and 40% had worked with WebCT for one year or more. Additionally, the participants spent varied amounts of time each week working on WebCT: 17% spent less than one hour on WebCT, 23% spent one to four hours, 26% spent four to ten hours, and 34% spent over ten hours.

Materials

In total, a seven point, seventy two item Likert scale was implied in the questionnaire. The questionnaire contained 6 different sections, each section corresponding to one of the chosen WebCT tools. Within each section, a series of check box type format and text areas for user comments were provided. The Likert type scale consisted of a series of declarative statements, asking the participant to indicate whether he/she agreed or disagreed with each statement. These statements represented a set of user attitude statements that reflected both overall usability and satisfaction plus a component of user perception of control. Some of the questions included: a) Whenever I make a mistake using the tool, I recover easily and quickly. b) It was easy to learn to use this tool. c) I am able to complete my work quickly using this tool.

Each question item was rated on a scale from 1 to 7, ranging from 1 (strongly agree) to 7 (strongly disagree). In addition, "not applicable" was listed as an option. At the end of each set of questions for each tool, additional space was given that prompted the user to make qualitative comments regarding their personal experience using each tool.

Procedure

Results

Reliability

Additionally in this study, the questions were based on standardised 7-point scale containing a mid-point. This choice of using a 7-point scale is justified by Matell and Jacoby's (1972) study that demonstrated that as the number of scale steps increases, the respondents' use of the mid-point category decreases as well. However, studies have shown that scales without a mid-point would be preferable due to the phenomenon that is pushes more respondents towards the positive end of the scale (Worcester and Burns, 1975).

Quantitative analysis

Table 2: The values for each of the scatter plot graphs for the six tools demonstrated that there was a positive and dependent relationship between satisfaction and control. More interestingly, for each of the tools, the value of slope b illustrated that a change in control directly influenced the change in satisfaction.

Validity

The results from the t-test proposed that the slopes between the different tools varied in a statistically, significant manner. From here, the slope values were used to determine the scalable usability of a tool. It was suggested that a tool with a higher the slope value would equate to a higher rank of usability for the tool. Therefore, in accordance with the quantitative results, the slope values for each tool were applied to this scaleable attribution of usability. However, it was imperative that the qualitative results be examined in order to confirm our quantitative findings.

Qualitative analysis

Measure

Figure 1: The pyramid of tools combined both the quantitative
and qualitative results to show the degree of usability of each of the tools.

Conclusions

Secondly, there is a relative scalable measure of usability that can be applied to each WebCT tool. These findings were flavoured by our qualitative results, demonstrating that user comments can be categorised in two methods: 1) by usability principles defined by HCI specialists and 2) by positive and negative comments regarding these principles. These findings also suggests that each individual WebCT tool can collectively influence a course designer's overall authoring experience, either negatively or positively.

Finally, from our combined quantitative and qualitative results, our findings suggest that perception of control and perception of satisfaction are correlated with the perception of usability for six specific WebCT authoring tools. This phenomenon was demonstrated in our pyramid of tools.

These results offer the idea that each individual authoring tool is important for determining the outcome of the course designer's overall experience using WebCT. With this in mind, a course designer who uses several authoring tools with high usability will most likely have a positive experience when creating his/her web-integrated learning environments, and therefore continue to use WebCT. On the other hand, the use of several authoring tools with low usability could inflict a negative experience for the course designer, possibly leading to the discontinuation of WebCT. Therefore, it is important for authoring tools to provide the highest level of usability for its users.

Overall, these results answered our previous questions regarding the six WebCT tools and were important to us for two reasons: 1) to have a better perspective of the usability of the WebCT authoring environment, and 2) to provide us with a further research basis in order to suggest usability improvements for specific WebCT authoring tools.

Discussion

The usability measures proposed in this study relate to only two main concepts of HCI, satisfaction and control, while targeting only six tools. However, it can not be excluded that there are multidimensional and diverse angles to approach usability. It can be considered that different measures of usability can be constructed in different ways, each influencing a different result. With this in mind, this study tends to lead us in the direction that when satisfaction and control are measured together, a decent scale of usability can be applied to a WebCT authoring tool.

Taking this in to account, it would be valuable to extend the development of this study to include examination based on the aspect of user experience when using authoring tools. More precisely, the investigation could include users' perceptions based on novice or expert experience using these tools. Additionally, it would be equally interesting to test different authoring environments, similar to WebCT, and conduct the same experiment using similar authoring tools. These findings would perhaps provide a more lucent explanation on the influences of "perception."

References

Becker, D. A. A. and Dwyer, M. M. (1994). Using hypermedia to provide learner control. Journal of Educational Multimedia and Hypermedia, 3(2), 155-172.

Chau, P. Y. K. (1996). An empirical assessment of a modified technology acceptance model. Journal of MIS, 13(2, Fall), 185-204.

Davis, F. D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, 13(3, Fall), 319-339.

Doll, W.J. & Torkzadeh, G. (1988). The measurement of end-user computing satisfaction. MIS Quarterly, 12(2), 259-274.

Fishbein, M. and Ajzen, I. (1975). Belief, Attitude, Intention and Behavior: An Introduction to Theory and Research. Addison-Wesley, MA.

Garrison, D. R. & Baynton, M. (1987). Beyond independence in distance education: The concept of control. The American Journal of Distance Education, 1(3), 3-15.

Lewis, J. (1995). IBM Computer usability satisfaction questionnaire: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1), 57-78.

Matell, M. S. & Jacoby, J. (1992). Is there an optimal number of alternatives for Likert scale items? Study 1: Reliability and validity. Educational and Psychological Measurements, 31, 657-674.

Nielsen, J. (1993). Usability Engineering. Academic Press Inc., Boston.

Norman, D. A. (1988). The Psychology of Everyday Things. Basic Books, New York.

Nunnally, J. C. (1978). Psychometric Theory. New York: McGraw-Hill.

Preece, Jenny, et al (1994). Human-Computer Interaction. Addison-Wesley Publishing Company, 1994.

Reeves, T. (1993). Pseudoscience in computer-based instruction: The case of learner control research. Journal of Computer Based Instruction, 20(2), 39-46.

Shackel, B. (1991). Usability - context, framework, design and evaluation. In Shackel, B. and Richardson, S. (Eds), Human Factors for Informatics Usability. Cambridge University Press, Cambridge, 21-38.

Shneiderman, B. (1986). Designing the user interface: Strategies for effective human-computer interaction. Addison-Wesley, Reading MA.

Wilson, B. G. (1995). Metaphors for instruction: Why we talk about learning environments. Educational Technology, 35 (5), 25-30.

Worcester, R.M. & Burns, T.R. (1975). A statistical examination of the relative precision of verbal scales. Journal of Market Research Society, 17(3), 181-197.

Authors: Allison Piguet is a PhD student in the Department of Educational Technology at the University of Geneva, Switzerland. She received her BA and teaching degree from San Diego State University and a Master's Degree in Educational Technology from the University of Geneva. Her research focuses on virtual campuses, authoring tools, and the development of pedagogical activities for virtual learning environments . Address: TEFCA-FPSE - University of Geneva, 40 bd du Pont d'Arve, 1205 Geneva, Switzerland. Dr Daniel Peraya is an Assistant Professor in the Department of Educational Technology at the University of Geneva, Switzerland. His research examines computer mediated communication and web-based integrated learning environments. Address: TEFCA-FPSE - University of Geneva, 40 bd du Pont d'Arve, 1205 Geneva, Switzerland. TECFA [ http://tecfa.unige.ch/ ] is an Educational Technology Department within the Faculty of Psychology and Educational Sciences, University of Geneva. Please cite as: Piguet, A. and Peraya, D. (2000). Creating web-integrated learning environments: An analysis of WebCT authoring tools in respect to usability. Australian Journal of Educational Technology, 16(3), 302-314. http://www.ascilite.org.au/ajet/ajet16/piguet.html