Figure 1: Path of technological and vocational school teachers' evaluation effectiveness materials
| Australasian Journal of Educational Technology 2012, 28(4), 719-739. |
AJET 28 |
Influence of teachers' perceived e-portfolio acceptance on teacher evaluation effectiveness in Taiwan
Chun-Mei Chou
National Yunlin University of Science and Technology
This study examines technological and vocational school teachers' perceived e-portfolio acceptance, computer self-efficacy, and evaluation effectiveness in Taiwan. Teachers' perceived e-portfolios acceptance includes four factors, namely, staff commitment, performance expectancy, performance expectancy, and technology training. Computer self-efficacy includes four factors, namely, playfulness, ease of use, effectiveness, and usefulness. Evaluation effectiveness includes four factors, namely, classroom management, instructional design, dedicated attitude, professional development. Participants (N=318) responded to a 5-point Likert-type scale for each factor. Analysis was conducted using the structural equation modeling (SEM), and a good model fit was found for both the measurement and structural models. Findings demonstrate that technological and vocational schools teachers' e-portfolios acceptance significantly and directly influences evaluation effectiveness, and e-portfolio acceptance influences evaluation effectiveness by computer self-efficacy. The teachers' e-portfolio acceptance and computer self-efficacy fit the influence model and empirical data of evaluation effectiveness.
One future trend in education is to cope with changes in the external environment through making appropriate adjustments, with the assistance of teacher professional development evaluations, to better develop education quality and the professional know-how of teachers (Tsai, Tsai & Hwang, 2011; Yao, Aldrich, Foster & Pecina, 2009). The purpose of teacher professional development evaluations is to ensure instructional quality, and establish standards and performance levels for teacher professional development (Meyer, Abrami, Wade, Aslan & Deault, 2010). Ifinedo (2006) suggested that teaching portfolios should meet the three requirements of performance evaluation, the promotion of teacher professional development, and giving teachers better chances to obtain employment. Through the systematic collection of data on teacher instruction, student counseling, administration and other services, teaching portfolios can be used to verify teacher professional development. Teacher e-portfolios can also be used to evaluate the results of student learning, can be used to evaluate teacher professional development and instructional performance, and can also be used as criteria for teacher license renewal (Joyes, Gray & Hartnell-Young, 2010; Pynoo, Devolder, Tondeur, van Braak, Duyck & Duyck, 2011; Swan, 2009).
Some studies have suggested that teachers' perceived e-portfolio acceptance will affect teacher evaluation effectiveness (Garrett, 2011; Nicholson, 2004; Seldin, Miller, Seldin, McKeachie, 2010). From the social cognitive perspective, teacher acceptance of e-portfolios is affected by factors such as social pressure and the school environment, which in turn affect teacher motivation, attitude, and the adoption of e-portfolios (Bandura, 2007; 2008; Saade & Kira, 2009; Shepherd & Bolliger, 2011). A number of schools have established e-portfolio and web-portfolio symposium websites in the hopes of effectively promoting teacher professional evaluation systems to help enhance school effectiveness (Pynoo, Devolder, Tondeur, van Braak, Duyck & Duyck, 2011; Wong & Li, 2011). Teacher perceptions of the usage of e-portfolios not only improve actual teaching performance, but also increase the information and feedback from student learning accomplishments. In addition, the perceived usage of teacher evaluation systems can be used to remove unsuitable teachers (Chang & Tseng, 2009).
In terms of the theory of planned behaviour, it is possible to explore teachers' personal voluntary behavior, external influences, and the reasons for obstacles when they adopt new technology or accept innovative management mechanisms. The theory of planned behaviour stresses teachers' perceived behavioural control. When observing behavioural intentions, attitude, subjective norms, and perceived behavioural control will all affect personal behavioural intentions and in turn affect behaviour (Chai & Lim, 2011; Pynoo, Devolder, Tondeur, van Braak, Duyck & Duyck, 2011). Teachers' perceived e-portfolio acceptance will in turn affect their perceived behavioural control. This reflects their past experiences and expected obstacles, which are adjusted through internal and external personal behavioural controls and then shown by their teacher evaluation effectiveness. Most studies relating to teacher e-portfolios have focused on student learning accomplishments (Chang, 2008; Chang & Tseng, 2009; Chau & Cheng, 2010), teacher training (Chuang, 2008), instructional accomplishments (Avraamidou & Zembal-Saul, 2002), the acceptance of information (Pynoo, Devolder, Tondeur, van Braak, Duyck & Duyck, 2011), technological leadership (Chang, Chin & Hsu, 2008), and instructional reflections (Sung, Chang, Yu & Chang, 2009; Strudler & Wetzel, 2005), but there have been few studies that have explored the extent to which working teachers accept professional evaluations and the testing of their evaluation effects. In addition, it is difficult to understand the professional development of teachers, and further evaluation of its effects is even more difficult. Evaluation is a formal procedure, but in professional development evaluation there are only documentary records without actual evaluation; the evaluation content is too superficial and tends to be focused on surface effects. Thus, the influence of teachers' perceived e-portfolio acceptance on teacher evaluation effectiveness is an important issue in the exploration of teacher evaluation effectiveness.
Schools promote teaching portfolio measures in order to achieve the needs of teacher professional evaluations, allowing teachers to have effective management and production. Additionally, teachers' computer self-effectiveness abilities have become increasingly important (Nicholson, 2004; Pynoo, Devolder, Tondeur, van Braak, Duyck & Duyck, 2011). The internal factors that affect teachers' computer self-effectiveness include personal differences, information techniques and abilities, personal will, attitude cultivation, and memory. Among these, some can be improved through educational training and experience (Guriting, Chunwen & Ndu, 2007; Vajargah & Jahani, 2010; Koc & Bakir, 2010; Saade & Kira, 2009). The external factors that affect teachers' computer self-effectiveness include time, opportunities, and assistance from others. Changes in time, context, or strategic transformations cause new behavioural intentions (Anderson & Maninger, 2007; Guriting, Chunwen & Ndu, 2007; Hoffer & Alexander, 1992; Koc & Bakir, 2010).
The technology acceptance model aims to explain the decisive factors of users' acceptance of information technology, and analyses the influences of external factors on users' beliefs, attitudes, and intentions concerning technology use (Saade & Kira, 2009; Teo, 2009a, 2009b). Analysis of the factors of teachers' evaluation effectiveness, through e-portfolio acceptance and computer self-effectiveness, can help to probe into their perceived usefulness and ease of use. These two factors will influence users' attitudes, behavioural intentions, and usage behaviours toward technology. Through the above, the perceptions of teachers when they participate in e-portfolios, the influence of teacher perceptions of e-portfolios on teacher evaluation effectiveness, and whether teachers' computer self-effectiveness is beneficial for teachers, can all be shown. Analysis of the factors that influence teacher evaluation effectiveness could help schools to better understand vocational and technical school teacher needs in e-portfolio and professional development evaluations, in order to engage in improvements of digital instruction and instructional guidance and cope with the urgent needs of vocational school teacher professional development and the sustainable management of schools.
In the context for this research, effective e-portfolios should contain: (1) a description of teaching responsibilities; (2) reflections on the teaching objectives; (3) a collection of instructional data; (4) student, peer and administrative evaluations and the subsequent reflection; and (5) descriptions of instructional activity improvements. The data collected for teacher e-portfolios may come from the process of instruction, including instructional design, class management, research development and advancement, and from the accumulated results of a diligent spirit, as well as from evidence of teachers' personal reflections. Teacher instructional portfolios may be divided into seven types: showcase portfolios, progress portfolios, process/product portfolios, reflective portfolios, rhetorical portfolios, teaching portfolios, and accreditation portfolios. Among these, accreditation portfolios may take the form of university or department evaluations; the angle and needs of the evaluation encompass the perspectives of universities or departments (Lee, 2009; Shepherd & Bolliger, 2011; Shroff, Deneen & Ng, 2011; von Konsky & Oliver, 2012).
Teachers' e-portfolio acceptance indicates that according to instructional content and strategies, they have abilities in applying computer media, the Internet, and information technology as instructional tools (Garrett, 2011; Hsu, Ju, Yen, & Chang, 2007; Pynoo, Devolder, Tondeur, van Braak, Duyck & Duyck, 2011). According to the Technology Acceptance Model (TAM), the determining factors for teacher acceptance of information technology include the effect of external factors on internal beliefs, attitudes, and intentions, which affect the conditions of technological use.
Scholars believe that the factors affecting teacher acceptance of teacher evaluation policies can be explored from the three aspects of teachers, policies, and environmental variables. Among these, teacher factors can be divided into personal background factors and personal work factors, such as staff commitment and expected level of effort. Policy factors include performance expectancies, and school environment variables include school size or location, and technology training.
Davis (1989) referred to the Technology Acceptance Model (TAM) according to Fishbein and Ajzen's Theory of Reasoned Action (TRA). TRA theory is that human use of appropriate information is often quite rational and systematic (Fishbein & Ajzen, 1975; Ajzen, 1988). The technology acceptance model theory indicates that perceived usefulness of information technology (perceived usefulness), and ease of use (perceived ease of use) are the two main determinants of intent (usage) behavior. This includes the four variables of perceived usefulness, confirmation, satisfaction, and willingness for continued use. It believes that perceived usefulness and satisfaction will affect willingness for the continued use of information systems. Perceived usefulness and confirmation will also affect user satisfaction (Ifinedo, 2006; Larsen, Sørebø & Sørebø, 2009; Ryu, Kim & Lee, 2009). Therefore, teacher willingness to use e-portfolios will affect satisfaction and perceived usefulness, and will indirectly influence the confirmation factor of willingness of use.
Some studies have pointed out that computer self-effectiveness has a decisive influence on teacher instructional behaviour. The sense of effectiveness produced by computer self-effectiveness is not only related to the future instructional context of computer use by teachers, but will also affect whether they are willing to understand computers and apply them to instructional activities. If there is low teacher computer self-effectiveness, they may resist using computers for e-portfolios. Teachers who use technology in schools and have a positive attitude and a sense of effectiveness toward computers can help students have positive attitudes toward computer technology, which will help the school promote professional development evaluations with e-portfolios.
When technology assists teachers in recording student learning processes and in complex preparation work, it can help teachers in completing e-portfolio databases. Teachers can then have more time to analyse student needs and engage in individualised instruction. When teachers are more familiar with the characteristics of technology and resources, they can easily use technology to improve personal instruction professionalism and teacher professional evaluation effectiveness. This shows that the level of teacher computer effectiveness not only influences personal computer behavior and learning intentions, but can also help with e-portfolio acceptance and teacher evaluation effectiveness.
Teacher evaluations can be assessed from the three aspects of teacher competence evaluations and teacher quality evaluations, teacher performance evaluations, and teacher effectiveness evaluations (Cai & Lin, 2006; Chai & Lim, 2011). Teacher evaluation indicators include: (1) a focus on the planning and implementation of instruction; (2) evaluation, including instructional evaluation, feedback, and improvement; (3) planning a learning environment that is safe and conducive to learning; (4) enhancing communication and connections with parents and communities; and (5) an emphasis on professional ethics and development. In Taiwan, the teacher professional development evaluation indicators are divided into course design and instruction, class management and counseling, research development and advancement, and a diligent spirit and attitude.
The perspective of Task-Technology Fit (TTF) states that for information technology to effectively improve personal work performance, such technology must be accepted by the users, and it should accommodate the supported tasks. In other words, if the technological functions to be used by the teacher can make the task smoother, the user's costs in carrying out the task will be lowered. Only when it can support the task to be completed will there be good work performance (Wong & Li, 2011).
Teachers' personal evaluations do not consider the effect of the organisation on teacher evaluation and professional development, which can contribute to adverse teacher evaluation effects. The procedures and content of teaching portfolio evaluations should consider the content items and performance standards to be evaluated, the scope and size of the files, a concrete explanation of the requirements of the constructed files, and a definitive evaluation of the file content. Anderson & Maninger (2007) studied the effects of teacher professional development evaluations, exploring the five aspects of participant responses, participant learning, organisational support and change, new know-how from participant usage, and student learning results.
In summary, teacher evaluation refers to the value determination of teacher performance and the decision-making process, which assists with teacher professional development, in order to understand instructional performance and promote school reform. Through professional development evaluation, teachers are inspired to engage in teaching reflection and the pursuit of accomplishments, giving them a willingness and motivation for development that can effectively improve teacher instructional quality and student learning accomplishments. Teacher professional development evaluations utilise diverse evaluation methods, collecting performance data on course design and instruction, classroom management and counseling, research development and advancement, and a diligent spirit and attitude. Through the establishment of teacher professional development evaluations mechanisms, teacher instruction professionalism and its effects can be measured with objective evaluation standards.
Through the criteria of establishing e-portfolios and the digitisation process, there can be participation in teacher evaluations, and systematic, conscious, and responsible efforts can be used to promote teacher instructional performance (Ahmad, Basha, Marzuki, Hisham & Sahari, 2010). Chang (2008) and Churchill (2009) pointed out that e-portfolios as formative evaluations for teachers to improve their instruction can promote teacher instructional effectiveness and achieve teaching objectives. Instructional effectiveness is the evaluation of personal instructional abilities and the extent of influence for student learning in the course of teaching (Churchill, 2009; Davis, 1989). This type of ability evaluation of computer self-efficacy is a subjectively conscious perception or belief, which will influence personal behaviour (Ahmad, Basha, Marzuki, Hisham, & Sahari, 2010; Pynoo, Devolder, Tondeur, van Braak, Duyck & Duyck, 2011; Tsai, Tsai & Hwang, 2011).
Teachers' perceived e-portfolios acceptance or beliefs in their own e-teaching ability evaluation will affect their instructional behaviour. Furthermore, teacher computer self-efficacy can also explain their instructional behaviour and teacher evaluation effectiveness (von Konsky & Oliver, 2012; Swan, 2009). Highly effective teachers use more difficult and challenging instructional techniques and execute innovative plans. In other words, there is a positive correlation between instructional effectiveness and instructional behaviour (Garrett, 2011; von Konsky & Oliver, 2012). The development of these records and the traces of their gradual deployment can be recorded with teaching portfolios; teachers can continue to elevate their sense of instructional effectiveness from reflecting on the files, and can promote the realisation of teaching objectives.
Teacher performance responsibility and professional development conversion is the trend in school education development. In school education and in teacher professional development, teaching portfolios can play a proactive and intermediary role (Ahmad, Basha, Marzuki, Hisham & Sahari, 2010). Through the establishment of teaching portfolios, teachers can provide records of professional development and proof of their accomplishments. In this way, teacher professional development is guaranteed, student learning accomplishments are assured, and instruction can be affirmed by the public at large (Wong & Li, 2011).
In sum, the establishment of teaching portfolios is being guided by teacher evaluation and diverse evaluation considerations in education, and they have gradually become a part of teacher professional development. Teachers should proactively consider and reflect upon the meaning of teaching portfolios for class instruction and school education. At the same time, the establishment of teaching portfolios can lead to personal professional development, and they can cope with future trends in school education applications.
| Variables | Mean | SD | Skewness | Kurtosis | X2 | p-value |
| Staff commitment | 4.12 | .61 | -.14 | .11 | 3713.07 | .000 |
| Effort expectancy | 4.01 | .55 | -.21 | -.25 | 3924.11 | .000 |
| Performance expectancy | 4.11 | .52 | -.37 | .34 | 4823.59 | .000 |
| Technology training | 3.76 | .62 | -.36 | .19 | 3418.26 | .000 |
| Classroom management | 4.14 | .61 | .23 | .31 | 3361.76 | .000 |
| Instructional design | 4.26 | .56 | -.27 | -.54 | 5235.35 | .000 |
| Dedicated attitude | 4.23 | .61 | -.36 | -.61 | 4128.56 | .000 |
| Professional development | 4.12 | .47 | -.32 | -.24 | 3879.51 | .000 |
| Playfulness | 4.24 | .59 | -.24 | -.34 | 3648.22 | .000 |
| Ease of use | 4.18 | .53 | .13 | .28 | 4279.29 | .000 |
| Effectiveness | 3.65 | .65 | -.12 | .11 | 3721.54 | .000 |
| Usefulness | 3.88 | .59 | -.19 | -.13 | 4013.62 | .000 |
| Total | 4.03 | .58 | -.27 | -.12 | 230.21 | 1.000 |
| Basic information | Group | No of people | % |
| Gender | Male | 184 | 58% |
| Female | 134 | 42% | |
| Educational background | University (or below) | 38 | 12% |
| Master | 76 | 24% | |
| Doctor | 204 | 64% | |
| Seniority | 5 years (and below) | 58 | 18.3% |
| More than 5 years and less than 10 years | 88 | 27.6% | |
| More than 10 years and less than 15 years | 73 | 23.0% | |
| Over 15 years | 99 | 31.2% | |
| Current post | Full time teachers and administration staff | 137 | 43.2% |
| Full time teachers | 181 | 55.8% | |
| Current level | Professor | 29 | 9.2% |
| Associate professor | 124 | 39.1% | |
| Assistant professor | 73 | 23.1% | |
| Lecturer | 94 | 29.6% | |
| Teaching background | Science, engineering, agriculture and design | 115 | 36.3% |
| Business, management, tourism and recreation | 130 | 41.0% | |
| Livelihood, health and medical care | 32 | 10.1% | |
| Liberal arts, law, education and general knowledge | 41 | 12.6% | |
| School attributes | Public | 169 | 53.3% |
| Private | 149 | 46.7% | |
| School category | Uni. of technology (inc. Coll. of technology) | 170 | 46.8% |
| Vocational school | 148 | 53.2% | |
| Number of students | 5,000 (and below) | 136 | 42.7% |
| More than 5,001 and less than 10,000 | 115 | 36.1% | |
| Above 10,000 | 67 | 21.2% |
In this study, four experts evaluated the fitness of the questions in order to verify the expert fitness of the scale. Ten teachers from technological and technological and vocational schools were invited to answer the questionnaire in order to enhance face validity. Five technological and technological and vocational schools were selected for a pre-test, with 115 teachers as the subjects. A total of 108 valid samples were collected; with a valid return rate of 93.9%. The scale in this study was a self-reported inventory, based on a Likert 5-point scale, where the range of "agree" to "disagree" is denoted by 5 to 1, respectively. The factor names, number of items, validity, and reliability levels of each aspect in this scale are as shown in Table 3.
| Factor | Composition of scales | No of items | Factor loading | Cronbach alpha | Accumulated explained variance | KMO | Total reliability Cronbach alpha |
| Scale of e-portfolio acceptance | Staff commitment | 3 | 27.61 | .91 | 65.59 | .901 | .93 |
| Effort expectancy | 3 | 12.30 | .84 | ||||
| Performance expectancy | 3 | 14.47 | .83 | ||||
| Technology training | 4 | 11.21 | .92 | ||||
| Computer self-efficacy | Playfulness | 3 | 29.13 | .89 | 68.94 | .874 | .93 |
| Ease of use | 3 | 21.54 | .90 | ||||
| Effectiveness | 3 | 10.49 | .92 | ||||
| Usefulness | 3 | 7.78 | .91 | ||||
| Teacher evaluation effectiveness | Classroom management | 4 | 24.58 | .92 | 66.37 | .881 | .92 |
| Instructional design | 4 | 23.14 | .89 | ||||
| Dedicated attitude | 5 | 10.04 | .89 | ||||
| Professional development | 4 | 8.61 | .87 |
| Parameter | Standard deviation | t value | Normalised coefficient | Parameter | Standard deviation | t value |
| lambda 1 lambda 2 lambda 3 lambda 4 lambda 5 lambda 6 lambda 7 lambda 8 lambda 9 lambda 10 lambda 11 lambda 12 |
0.65 0.66 0.73 0.71 0.59 0.68 0.69 0.63 0.48 0.63 0.70 |
19.39* 18.14* 19.25* 18.09* 11.46* 19.20* 17.89* 15.59* 12.38* 15.49* 18.23* 12.56* |
0.25 0.26 0.32 0.26 0.48 0.23 0.36 0.25 0.29 0.27 0.29 0.28 |
eta 1 eta 2 eta 3 eta 4 eta 5 eta 6 eta 7 eta 8 eta 9 eta 10 eta 11 eta 12 |
0.34 0.29 0.44 0.33 0.22 0.31 0.39 0.28 0.37 0.29 0.38 0.29 |
16.20* 14.14* 12.58* 13.01* 13.6* 12.67* 14.10* 11.67* 14.45* 13.55* 14.03* 12.49* |
| Note: Those without standard deviations are criterion indicators of *p<.05 | ||||||
According to the definitions of Hair, Anderson, Tatham and Black (1998), three items can be used to examine estimation violations: 1. negative error variable exists; 2. normalised coefficient is above or approximate to 1 (0.95 is the usual threshold), and 3. significant standard deviation. According to Table 4, parameter estimations reveal positive error variables, and there are no negative variables. Normalised coefficient is 0.25~0.95, and remaining within the scope. Therefore, parameters estimations are not violated, and a fit could be conducted.
This study conducted model fit testing by general criteria, according to fit measures estimated by the statistical method, in order to find out the fitness between the research data and model. Measures of this study are based on absolute fit, incremental fit, and parsimonious fit, as classified by Hair, Anderson, Tatham & Black (1998). In addition, construct reliability and validity tests of variables were used to determine the internal structural fit. Analytical results are shown below.
This study first examines theoretical validation of the model regarding unsatisfying evaluation standards. After reviewing ML, the model is modified according to the rationality of the theory. Although model fit after modification is enhanced, estimates of usefulness and computer self-efficacy do not reach a level of significance, and re-estimation is required. Although modified chi-squared testing fails to reach a statistical significance level, the model fit is improved and mostly satisfies the standards. In addition, tests of overall fit are generally positive.
| Evaluation items and outcome (N=318) | Evaluation standard | Researchers | Fit | |
| Absolute fit | chi-squared = 213.91 d.f. = 45 chi-squared/d.f. = 4.75 GFI = 0.91 AGFI = 0.83 SRMR = 0.09 RMSEA = .012 |
<5 >0.9 >0.8 <0.1 <0.08 |
Hair et al. (1998) Hair (2010) Hair (2010) Hu & Bentler (1999) Jarvenpaa et al. (2000) |
Acceptable Acceptable Acceptable Acceptable Acceptable |
| Incremental fit | NFI = 0.92 NNFI = 0.91 IFI = 0.92 CFI = 0.91 |
>0.9 >0.9 >0.9 >0.9 |
Bentler & Bonett (1980) Bentler & Bonett (1980) Bentler & Bonett (1980) Bagozzi &Yi (1988) |
Acceptable Acceptable Acceptable Acceptable |
| Parsimonious fit | PNFI = 0.68 PGFI = 0.57 |
>0.5 >0.5 |
Bentler & Bonett (1980) Bentler & Bonett (1980) |
Acceptable Acceptable |
According to the analytical results of the revised model in Table 5, in absolute fit measures, GFI = 0.91, which is slightly higher than the standard of 0.9. According to Gefen and Straub (2000), GFI should be above 0.90, thus, this model is acceptable. AGFI = 0.83, which is higher than the standard of 0.8. Gefen and & Straub (2000) suggests that AGFI should be above 0.80, thus the model is acceptable. RMSEA = .012 and <0.08. According to Jarvenpaa, Tractinsky and Vitale, (2000), RMSEA should be lower than 0.08, thus, based on the measures above, the absolute fit of this model is good.
As to incremental fit and parsimonious fit measures, according to Gefen and Straub (2000) and Hair et al. (1998), when NFI, IFI, RFI and CFI are above 0.9, PNFI and PGFI should be above 0.5 in order for the model to be accepted. According to tghe data tested, NFI = 0.92, which is above the standard 0.9. It shows that the model is accepted. IFI = 0.92, which is above 0.9 means that the model is accepted. RFI = 0.89, which is lower than the standard 0.9, it means that the model is almost acceptable. CFI = 0.91, thus, the model is relatively acceptable. PNFI = 0.68, which is above the standard 0.5; the model is relatively acceptable. PGFI = 0.57, which is above 0.5; the model is acceptable. According to the incremental and parsimonious fit measures above, the models of this study are acceptable. However, absolute fit is good, which suggests that the models are still acceptable, as the overall model meets empirical data.
Regarding the structural fit test, according to the structural parameters of the influence model of teachers' evaluation effectiveness materials in Table 4, e-portfolios acceptance and computer self-efficacy are significant (t=3.13, p<.05). In addition, as to the evaluation of R2 latent dependent variables, according to Table 40, e-portfolios acceptance, and computer self-efficacy are 0.65 and 0.73, respectively. The R2 of computer self-efficacy is lower than 0.50, which meets the evaluation standards; therefore, the structural model fit of this study is good.
Correlation coefficients of the three latent variables are 0.72~0.91 (Table 6 and Figure 1). The correlation coefficient of teacher evaluation effectiveness on e-portfolios acceptance is higher than 0.90, while the remaining are lower than 0.90. Thus, the three latent variables may affect the structural model fit due to overly high correlations.
Technological and vocational school teachers' e-portfolios acceptance, performance expectancy, and technology training show significant influence on classroom management and instructional design, in teacher's evaluation effectiveness. Effort expectancy allows teachers to recognise objectives for e-portfolios in schools. When schools encounter environmental changes and competition, teachers gradually understand and trust the measures of the schools' innovative leadership. Through performance expectancy and rewards by schools, teachers can fulfill their creativity and further develop courses and e-teaching design. This will allow them to accomplish school objectives and instructional design in order to meet environmental changes (Chau & Cheng, 2010; Pynoo, Devolder, Tondeur, van Braak, Duyck & Duyck, 2011; Damanpour & Wischnevsky, 2006; von Konsky & Oliver, 2012).
| Latent variables | R2 | E-portfolio acceptance | Teacher evaluation effectiveness | Computer self-efficacy |
| E-portfolio acceptance | 0.91 | 1 | ||
| Teacher evaluation effectiveness | 0.72 | 0.72 | 1 | |
| Computer self-efficacy | 0.81 | 0.91 | 0.81 | 1 |
Figure 1: Path of technological and vocational school teachers' evaluation effectiveness materials
Technological and vocational school teachers' e-portfolio acceptance, staff commitment, and technology training show significant influence on ease of use and effectiveness of computer self-efficacy. Instructional design and dedicated attitude in technological and vocational school teachers' evaluation effectiveness significantly influence playfulness and ease of use of computer self-efficacy. Teachers' perceived technology training and staff commitment will enhance teachers' new teaching methods, evaluations, and implementation of e-portfolios. It is one of the key factors on teachers' computer self-efficacy (Chai & Lim, 2011; Karsten & Roth, 1998; Joyes, Gray & Hartnell-Young, 2010; Mohrman, Cohen & Mohrman, 1995; Swan, 2009).
Technological and vocational school teachers' e-portfolio acceptance significantly and directly influences teacher evaluation effectiveness. E-portfolio acceptance significantly and indirectly influences evaluation effectiveness through computer self-efficacy. Technological and technological and vocational schools teachers' e-portfolio acceptance and computer self-efficacy fit the model and empirical data of evaluation effectiveness. Therefore, schools should respect and support instructional technology training. Through recognition of e-portfolios and evaluation contents, teachers can constantly ponder on e-portfolio training and information technology. They will enhance instructional design and adopt a dedicated attitude. Teachers' active promotion of web-based portfolio ideas in e-portfolios and practical use of e-instruction will encourage them in their teaching. They identify with performance expectancy through e-portfolio acceptance in order to develop teachers' e-portfolio plans, and constantly adopt and implement e-portfolios. Thus, they will result in useful e-portfolios and a more dedicated attitude (Pynoo, Devolder, Tondeur, van Braak, Duyck & Duyck, 2011; Angle & Van de Ven, 2000; Hage & Aiken, 1970; Igbahia & Iivari, 1995; Susser & Ariga, 2006).
Although the model of this study reveals goodness of fit, the fit effect remains unsatisfying, which suggest that there are some latent variables that have not been elaborated. Variables in models sometimes fail to reveal the ideal explanatory effect; thus, this study further probes into the model with a more complete overall fit.
This study measures teachers' perceptions during the first, middle and final stages of teacher evaluation effectiveness. Some teachers perceived e-portfolios as a means to evaluate teacher effectiveness. Teachers' perceived computer and evaluations of e-portfolios are insufficient; meaning they may lack knowledge in two scales, namely, "computer self-efficacy" and "evaluation effectiveness", and any effects between the two would be limited.
Scoring according to scales is based on teachers' self-perceived; therefore, this study cannot eliminate samples that were influenced by situations, attitudes, real respondents, emotions, or are seemingly unmatched with reality. Although the use of self-reports to collect data has its benefits, it may lead to a common method variance, namely, a situation that may inflate the true associations between variables.
This study aimed to probe into the factors of technological and vocational school teachers' evaluation effectiveness, which is significantly affected by school feedback regarding evaluation outcomes. Future studies can include measurements of this variable, and modifying the model in order to further probe into the cause-and-effect relations among the variables.
Second, according to results of this study, technological and vocational school teachers' "e-portfolios acceptance" significantly influences "computer self-efficacy". In the post-acceptance model (PAM), the process by which teachers perceive new playfulness or concepts through ease of use, effectiveness, and usefulness, usually requires a long period of use to gain acceptance by teachers. E-portfolios acceptance in schools will influence teachers' use intentions of computer technology. After e-portfolios content and technology training are introduced within an organisation, if it is supported by teachers and supervisors, the users' use attitude and intention would be indirectly influenced by the increased e-portfolios use opportunities, and subsequent evaluation experience (Sorensen, Mathiasen & Dalsgaard, 2009; Wong & Li, 2011).
Third, e-portfolios are more suited to formative evaluations, such as instructional improvement and professional development. When they are applied to summary evaluations such as performance auditing and the hiring and evaluation of teachers, a number of factors must be considered: 1. teacher e-portfolios are open, and it is difficult to compare instructional quality among teachers; 2. teacher e-portfolios must be filed and evaluated in detail; 3. e-portfolios should be focused on the dimensions or criteria by the evaluator; and 4. some instructional qualities cannot be demonstrated through e-portfolios.
Finally, the findings of this study indicate that "ease of use" and "effectiveness" of "computer self-efficacy" can enhance teachers' evaluation effectiveness. Besides computer knowledge and techniques, computer self-efficacy also includes teacher attitudes towards computer technology, such as usefulness, teacher cognition, evaluation value, and motive (Anderson, Klassen & Johnson, 1981). When teachers have a greater knowledge of evaluation effectiveness and performance, they are more likely to have intentions to use the web-based portfolios. There are significant and positive correlations among computer knowledge and skills, and teachers' e-evaluation and performance (Nicholson, 2004; Pynoo, Devolder, Tondeur, van Braak, Duyck, & Duyck, 2011; Seldin, Miller, Seldin & McKeachie, 2010). Measures to enhance technological and vocational school teachers' evaluation effectiveness are as follows: 1. allow teachers to perceive the importance and growing trends in e-portfolio through teacher instructional design and classroom management; 2. assist teachers with e-portfolios in order to integrate the courses and resources of e-evaluation system; 3. instruct teachers to reorganise current teacher evaluation resources into a web-based portfolio system; and 4. plan e-portfolio training with encouragement in order to energise teachers' enthusiasm to adopt e-evaluation information in an e-portfolio system.
Ahmad, T. B. T., Basha, K. M., Marzuki, A. M., Hisham, N. A. & Sahari, M. (2010). Faculty's acceptance of computer based technology: Cross-validation of an extended model. Australasian Journal of Educational Technology, 26(2), 268-279. http://www.ascilite.org.au/ajet/ajet26/ahmad.html
Ajzen, I. (1988). Attitudes, personality and behavior. Milton Keynes (UK): Open University Press. http://books.google.com.au/books?id=dmJ9EGEy0ZYC&printsec=frontcover#v=onepage&q&f=false
Bagozzi, R. P. & Yi, Y. (1988). On the evaluation of structural equation models. Journal of the Academy of Marking Science, 16(1), 74-94. http://dx.doi.org/10.1007/BF02723327
Bandura, A. (2007). Much ado over a faulty conception of perceived self-efficacy grounded in faulty experimentation. Journal of Social and Clinical Psychology, 26(6), 641-658. >http://dx.doi.org/10.1521/jscp.2007.26.6.641http://des.emory.edu/mfp/Bandura2007JSCP.pdf]
Bandura, A. (2008). An agentic perspective on positive psychology. In S. J. Lopez (Ed.), Positive psychology: Exploring the best in people (Vol. 1, pp. 167-196). Westport, CT: Greenwood Publishing Company. http://des.emory.edu/mfp/Bandura2008PP.pdf
Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88(3), 588-606. http://dx.doi.org/doi:10.1037/0033-2909.88.3.588
Boomsma, A. & Hoogland, J. J. (2001). The robustness of LISREL modeling revisited. Psychometrika, 51, 313-325. http://www.ppsw.rug.nl/~boomsma/ssi.pdf
Cai, Y. & Lin, C. (2006). Theory and practice on teacher performance evaluation. Frontiers of Education In China, 1(1), 29-39. http://dx.doi.org/10.1007/s11516-005-0004-x
Chai, C. S. & Lim, C. P. (2011). The Internet and teacher education: Traversing between the digitized world and schools. The Internet and Higher Education, 14(1), 3-9. http://dx.doi.org/10.1016/j.iheduc.2010.04.003 [also at http://repository.ied.edu.hk/dspace/handle/2260.2/11128]
Chang, C. C. (2008). Enhancing self-perceived effects using Web-based portfolio assessment. Computers in Human Behavior, 24(4), 1753-1771. http://dx.doi.org/10.1016/j.chb.2007.07.005
Chang, C. C. & Tseng, K. H. (2009). Use and performances of Web-based portfolio assessment. British Journal of Educational Technology, 40(2), 358-370. http://dx.doi.org/10.1111/j.1467-8535.2008.00885.x
Chau, J. & Cheng, C. (2010). Towards understanding the potential of e-portfolios for independent learning: A qualitative study. Australasian Journal of Educational Technology, 26(7), 932-950. http://www.ascilite.org.au/ajet/ajet26/chau.html
Chou, C. M., Shen, C. H., Hsiao, H. C. & Chen, S. C. (2010a). The influence of innovative organizational management of technological and technological and vocational schools on innovative performance - using e-portfolios acceptance as the mediator variable. World Transactions on Engineering and Technology Education, 8(2), 237-242. http://www.wiete.com.au/journals/WTE&TE/Pages/Vol.8,%20No.2%20(2010)/19-12-CHOU.pdf
Chuang, H. H. (2008). Perspectives and issues of the creation for weblog-based electronic portfolios in teacher education. British Journal of Educational Technology, 39(1), 170-174. http://dx.doi.org/ 10.1111/j.1467-8535.2007.00748.x
Churchill, D. (2009). Educational applications of Web 2.0: Using blogs to support teaching and learning. British Journal of Educational Technology, 40(1), 179-183. http://dx.doi.org/10.1111/j.1467-8535.2008.00865.x
Damanpour, F. & Wischnevsky, J. D. (2006). Research on innovation in organizations: Distinguishing innovation-generating from innovation-adopting organizations. Journal of Engineering and Technology Management, 23(4), 269-291. http://dx.doi.org/10.1016/j.jengtecman.2006.08.002
Davis, E. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319-339. http://www.jstor.org/stable/249008
Fishbein, M. & Ajzen, I. (1975). Belief, attitude, intention and behavior: An introduction to theory and research. Reading, MA: Addison-Wesley.
Gefen, D. & Straub, D. W. (2000). The relative importance of perceived ease of use in IS adoption: A study of e-commerce adoption. Journal of the Association for Information Systems, 1(1), 1-28. http://www.cis.gsu.edu/~dstraub/Courses/MGS%209940/2007/Tim-Mathieu-Betrice%20choice.pdf
Guriting, G., Chunwen, G. & Ndu, N. N. O. (2007). Computer self-efficacy levels, perceptions and adoption of online banking. International Journal of Services Technology and Management, 8(1), 54-61. http://dx.doi.org/ 10.1504/IJSTM.2007.012218
Garrett, N. (2011). An e-portfolio design supporting ownership, social learning, and ease of use. Educational Technology & Society, 14(1), 187-202. http://www.ifets.info/journals/14_1/17.pdf
Hage, J. & Aiken, M. (1970). Social change in complex organizations. NY: Random House.
Hair, J. F. Jr., Anderson, R. E., Tatham, R. L. & Black, W. C. (1998). Multivariate data analysis (5th ed.). Englewood Cliffs: Prentice-Hall.
Hair, J. F. Jr,, Black, W. C., Babin, B. J. & Anderson, R. E. (2010). Multivariate data analysis (7th ed.). New Jersey: Pearson Prentice Hall.
Hu, A. L. & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. http://dx.doi.org/10.1080/10705519909540118
Harvey, L. & Williams, J. (2010). Fifteen years of quality in higher education. Quality in Higher Education, 16(1), 3-36. http://dx.doi.org/ 10.1080/13538321003679457
Hoffer, J. A. & Alexander, M. B. (1992). The diffusion of database machines. Data Base, 23(2), 13-19. http://dx.doi.org/10.1145/141342.141352
Hsu, M. H., Ju, T. L., Yen, C. H. & Chang, C. M. (2007). Knowledge sharing behavior in virtual communities: The relationship between trust, self-efficacy, and outcome expectations. International Journal of Human-Computer Studies, 65(2), 153-169. http://dx.doi.org/10.1016/j.ijhcs.2006.09.003
Ifinedo, P. (2006). Acceptance and continuance intention of web-based learning technologies (WLT) use among university students in a Baltic Country. The Electronic Journal on Information Systems in Developing Countries, 23(6), 1-20. http://www.ejisdc.org/ojs/include/getdoc.php?id=195&article=213&mode=pdf
Igbaria, M. & J. Iivari (1995). The effects of self-efficacy on computer usage. Journal of Management Science, 23(6), 587-605. http://dx.doi.org/ 10.1016/0305-0483(95)00035-6
Jarvenpaa, S. L., Tractinsky, N. & Vitale, M. (2000). Consumer trust in an Internet store. Information Technology and Management, 1(1&2), 45-71. http://dx.doi.org/10.1023/A:1019104520776
Joyes, G., Gray, L. & Hartnell-Young, E. (2010). Effective practice with e-portfolios: How can the UK experience inform implementation? Australasian Journal of Educational Technology, 26(1),15-27. http://www.ascilite.org.au/ajet/ajet26/joyes.pdf
Karsten, R. & Roth, R. M. (1998). Computer self-efficacy: A practical indicator of student computer competency in introductory IS courses. Informing Science, 1(3), 61-68. http://inform.nu/Articles/Vol1/v1n3p61-68.pdf
Koc, M. & Bakir, N. (2010). A needs assessment survey to investigate pre-service teachers' knowledge, experiences and perceptions about preparation to using educational technologies. The Turkish Online Journal of Educational Technology, 9(1), 13-22. http://www.tojet.net/articles/912.pdf
Larsen, T. J., Sørebø, A. M. & Sørebø, Ø. (2009). The role of task-technology fit as users' motivation to continue information system use. Computers in Human Behavior, 25(3), 778-784. http://dx.doi.org/10.1016/j.chb.2009.02.006
Lee, B. D. (2009). Teaching and learning e-portfolios. [viewed 11 May 2011] http://ge.dyu.edu.tw/981criticism/files/2009/12/eP_lee.pdf
Meyer, E., Abrami, P. C., Wade, C. A., Aslan, O. & Deault, L. (2010). Improving literacy and meta-cognition with electronic portfolios: Teaching and learning with ePEARL. Computers & Education, 55, 84-91. http://dx.doi.org/10.1016/j.compedu.2009.12.005
Ministry of Education (2008). Ministry of Education teacher professional development evaluation pilot program. http://140.111.34.34/moe/common/index.php?z=436 [in Chinese; viewed 12 May 2011]
Ministry of Education (2010). Ministry of Education pilot teacher professional development evaluation items. http://140.111.34.34/docdb/files/dma7da0a07101419086.pdf [in Chinese; viewed 14 May 2011]
Mohrman, S. A., Cohen, S. G. & Mohrman, A. M. Jr. (1995). Designing team-based organizations: New forms for knowledge work. San Francisco: Jossey-Bass.
Nicholson, B. (2004). E-portfolios for educational leaders: An ISLLS-based framework for self-assessment. Lanham, Maryland: Rowman & Littlefield.
Pynoo, B., Devolder, P., Tondeur, J., van Braak, J., Duyck, W. & Duyck, P. (2011). Predicting secondary school teachers' acceptance and use of a digital learning environment: A cross-sectional study. Computers in Human Behavior, 27(1), 568-575. http://dx.doi.org/10.1016/j.chb.2010.10.005
Ryu, M. H., Kim, S. & Lee, E. (2009). Understanding the factors affecting online elderly user's participation in video UCC services. Computers in Human Behavior, 25(3), 619-632. http://dx.doi.org/10.1016/j.chb.2008.08.013
Saade, R. G. & Kira, D. (2009). Computer anxiety in e-learning: The effect of computer self-efficacy. Journal of Information Technology Education, 8, 177-191. http://jite.org/documents/Vol8/JITEv8p177-191Saade724.pdf
Seldin, P., Miller, J. E., Seldin, C. A. & McKeachie, W. (2010). The teaching portfolio: A practical guide to improved performance and promotion/tenure decisions. San Francisco, CA: Jossey-Bass. (pp. 62-70)
Shepherd, C. E. & Bolliger, D. U. (2011). The effects of electronic portfolio tools on online students' perceived support and cognitive load. The Internet and Higher Education, 14(3), 142-149. http://dx.doi.org/10.1016/j.iheduc.2011.01.002
Shroff, R. H., Deneen, C. C. & Ng, E. M. W. (2011). Analysis of the technology acceptance model in examining students' behavioural intention to use an e-portfolio system. Australasian Journal of Educational Technology, 27(4), 600-618. http://www.ascilite.org.au/ajet/ajet27/shroff.pdf
Sorensen, E. K., Mathiasen, H., & Dalsgaard, C. (2009). E-learning concepts in higher education. The International Conference on E-Learning in the Workplace 2009, New York 10-12 June. [viewed 10 Sep 2011] http://person.au.dk/fil/16581526/Sorensen_Mathiasen_Dalsgaard_E-learning_concepts.pdf
Sung, Y. T., Chang, K. E., Yu, W. C. & Chang, T. H. (2009). Supporting teachers' reflection and learning through structured digital teaching portfolios. Journal of Computer Assisted Learning, 25(4), 375-385. http://dx.doi.org/10.1111/j.1365-2729.2009.00312.x
Susser, B. & Ariga, T. (2006). Teaching e-commerce web page evaluation and design: A pilot study using tourism destination sites. Computers & Education, 47(4), 399-413. http://dx.doi.org/10.1016/j.compedu.2004.11.006
Swan, G. (2009). Examining barriers in faculty adoption of an e-portfolio system. Australasian Journal of Educational Technology, 25(5), 627-644. http://www.ascilite.org.au/ajet/ajet25/swan.html
Teo, T. (2009a). Examining the relationship between student teachers' self-efficacy beliefs and their intended uses of technology for teaching: A structural equation modelling approach. The Turkish Online Journal of Educational Technology, 8(4), 7-16. http://www.tojet.net/articles/v8i4/841.pdf
Teo, T. (2009b). Modeling technology acceptance in education: A study of pre-service teachers. Computers & Education, 52(1), 302-312. http://dx.doi.org/10.1016/j.compedu.2008.08.006
Tsai, P. S., Tsai, C. C. & Hwang, G. J. (2011). The correlates of Taiwan teachers' epistemological beliefs concerning Internet environments, online search strategies, and search outcomes. The Internet and Higher Education, 14(1), 54-63. http://dx.doi.org/10.1016/j.iheduc.2010.03.003
Vajargah, K. F. & Jahani, S. (2010). Application of ICTs in teaching and learning at university level: The case of Shahid Beheshti University. The Turkish Online Journal of Educational Technology, 9(2), 33-39. http://www.tojet.net/articles/v9i2/924.pdf
Von Konsky, B. R. & Oliver, B. (2012). The iPortfolio: Measuring uptake and effective use of an institutional electronic portfolio in higher education. Australasian Journal of Educational Technology, 28(1), 67-90. http://www.ascilite.org.au/ajet/ajet28/vonkonsky.html
Wilfong, J. D. (2006). Computer anxiety and anger: The impact of computer use, computer experience, and self-efficacy beliefs. Computers in Human Behavior, 22(6), 1001-1011. http://dx.doi.org/10.1016/j.chb.2004.03.020
Wong, E. M. L. & Li, S. C. (2011). Framing ICT implementation in a context of educational change: A structural equation modelling analysis. Australasian Journal of Educational Technology, 27(2), 361-379. http://www.ascilite.org.au/ajet/ajet27/wong.html
Yao, Y., Aldrich , J., Foster, K. & Pecina, U. (2009). Pre-service teachers' perceptions of an electronic portfolio as a tool for reflection and teacher certification. Journal of Educational Research & Policy Studies, 9(1), 25-43. http://www.eric.ed.gov/PDFS/EJ896277.pdf
| Factor | Composition of scales | Items |
| Scale of e-portfolios acceptance | Staff commitment | 1. I will constantly ponder on innovative teaching materials and approaches of e-portfolios. 2. I will use multiple e-instructional approaches to develop students' multiple creations and learning. 3. I can actively promote instructional innovative ideas and attempt to implement e-portfolios. |
| Effort expectancy | 1. E-portfolios of the school have unique innovative ideas on teacher evaluation. 2. I believe that I master e-portfolios terminology and concepts. 3. Supervisors of the school respect and support e-portfolios at work. | |
| Performance expectancy | 1. I have sufficient equipment for e-portfolios teaching. 2. As long as I am in need, professional staff will effectively assist me with e-portfolios. 3. The school provides opportunities for e-portfolios. | |
| Technology training | 1. The school financially supports technology training for e-portfolios. 2. Most teachers of the school are willing to encounter challenges of e-portfolios. 3. The school climate stimulates teachers' e-portfolios thoughts. 4. Members of the school can understand e-portfolios training visions and goals. | |
| Computer self-efficacy | Playfulness | 1. Computer-based instruction is more interesting. 2. Digital data is more likely to attract students. 3. E-portfolios are more interesting. |
| Ease of use | 1. For me, the computer is easy to learn. 2. I can learn how to use computers by observing others. 3. It is easy for me to write all kinds of things on the computer. | |
| Effectiveness | 1. E-portfolios are more flexible. 2. E-portfolios are more likely to increase students' learning motivations. 3. E-portfolios are more likely to increase students' learning outcomes. | |
| Usefulness | 1. It is more useful to make teaching materials by computers. 2. It is easier to assist with students' after-school learning by e-portfolios. 3. It is easier to deal with digital teaching materials after learning computer skills. | |
| Teacher evaluation effectiveness | Classroom management | 1. I will create e-portfolios of classroom management plans in advance. 2. I will become familiar with the contents of different e-portfolios classroom management units. 3. I will prepare teaching materials and tools for different classroom management units. 4. In e-portfolios, I will change instructional activities to maintain students' concentration of classroom management. |
| Instructional design | 1. I will grade students by multiple e-portfolios evaluations. 2. I will have discussions with students regarding their learning results and guide their future learning by e-portfolios. 3. I will adjust instructional schedules, degree of difficulty, and methods, according to e-portfolios evaluation outcomes. 4. I will collect supplementary teaching materials in order to enhance e-portfolios teaching efficacy. | |
| Dedicated attitude | 1. During e-portfolios processes, I can create harmonious learning atmospheres. 2. During e-portfolios processes, I will value students' demands. 3. During e-portfolios processes, I will interact with students and share experiences. 4. During e-portfolios processes, I can maintain order in the class. 5. During e-portfolios processes, I might do something effort while teaching with computers. | |
| Professional development | 1. I can select e-portfolios of professional development according to instructional subjects and teaching materials. 2. I will enhance the operations of e-portfolios media in order to enhance teaching activities. 3. During e-portfolios processes, I will praise and encourage myself professional development progress. 4. At-the-moment solutions while working with e-portfolios are enough for me. |
| Author: Dr Chun-Mei Chou Graduate Institute of Vocational and Technological Education National Yunlin University of Science and Technology Yunlin, Taiwan 64002. Email: choucm@yuntech.edu.tw Please cite as: Chou, C. M. (2012). Influence of teachers' perceived e-portfolio acceptance on teacher evaluation effectiveness in Taiwan. Australasian Journal of Educational Technology, 28(4), 719-739. http://www.ascilite.org.au/ajet/ajet28/chou-cm.html |