Dept of Information Systems, U.C.T., Private Bag, Rondebosch 7700

After reporting two years ago [Eccles & Van Belle, 1998] on a number of "innovative approaches" which the authors adopted in a first-year information systems course, the time has come to evaluate the impact of some of these educational strategies and curriculum options for course delivery on student performance within the same course. Too often educators implement alternative teaching strategies without attempt at an ex-post evaluation of the educational effectiveness of the new course of action.

It has to be recognised that an educational environment almost never allows for laboratory-like circumstances and hence statistical analysis has to be hedged with numerous ifs and buts. Nevertheless, it is hoped that the tentative results presented in this paper will allow educators at other tertiary educational institutions to make more informed decisions on which educational approaches to take and perhaps to spur them on to report on their experiences.

This paper studies the effect of three different educational approaches on student performance in a first-year one semester IS course, the "Foundations of Information Systems" course INF1002F at the University of Cape Town:

• Computer-based training
• The "mastery" method
• The use of enrichment sessions.

In addition, an evaluation of the effect of class attendance a topic of some controversy among both staff and students - is also attempted.

The data for this study is drawn mainly from the departmental records of the INF1002F course for the academic years 2000 and 1999. The final examination results for the 2000 course were not available at the time of writing, but all other course performance results are already available. The course year mark for 2000 is made up of 15 distinct course deliverables: 7 marked practical tutorial assignments (3 spreadsheets and 4 database exercises), 7 "enrichment session hand-ins", 2 theory tests and the final practical examination. Although the number of students registered for the course stand at more than 700, analysis is limited to a subset of the students usually where data from student evaluations has to be used. Students completed an optional course evaluation questionnaire on-line after writing the second theory test during the last week of lectures. The questionnaire was administered and processed using the QuestionMark computer testing software. The main advantage of this approach is that it allows for the unobtrusive identification of each student and thus the linking of student responses to student performance data. About 400 students provided usable course evaluations.

To complement the textbook, lectures and laboratory assistance, students were given access to computer-based training (CBT) packages from two commercially available suites: Net-G and Individual Training. The use of these CBT packages was entirely at the discretion of the student. The availability of the CBT resources was mentioned in the first week of lectures and at occasional intervals later during the course.

The Individual Training software consists of 4 modules covering MS-Windows, MS-Word, MS-Excel and MS-Access. Each module consists of about 8 to 12 hours of interactive training and is followed by an optional assessment test. The modules run as stand-alone tutorials which are loaded directly from a local file server. The Net-G suite is a set of much more comprehensive and in-depth Internet-based tutorials which are accessible both on- and off-campus to all registered UCT students from a central ITS server. However, Net-G requires a fairly elaborate registration and downloading process and the use of an Internet browser.

The author has extensive experience in the use of computer-supported education. In an "ancient" study, supplementing class lectures with computer-based PLATO tutorials was found to have a statistically significant positive impact although that particular study was based on subject matter which required analytical understanding (financial break-even analysis) instead of skill development [Van Belle, 1986].

Due to technical reasons it is not possible to track which students actually used the CBT software. However, two of the questions on the course evaluation questionnaires pertained to the student experience with the CBT packages. It must be stressed that there is no evidence that students who responded to this question, actually did use the packages.

If one hypothesises that CBT contributes positively to learners performance, the corresponding null hypothesis states that there is no difference between the average marks obtained by students who used CBT vis-à-vis those who didnt. Table 1 below lists the average marks obtained in the practical component of the course (the subject area addressed by the CBT), tabulated according to students who indicated that they had used both, one or none of the CBT packages respectively. Note that the "Prac Exam" refers to the final two-hour practical exam written by all students, whereas the "Prac Avge" refers to a weighted average mark of practical exam and 7 marked tutorials.

If the average results of the students who used at least one CBT package are compared to those who did not by means of the Z-test (difference between means of large samples), Z-values of -3.14 (Prac Avge) and 3.60 (Prac Exam) are obtained. This indicates that the chance of both samples being drawn from the same population is statistically significantly less than 1%. However, the sign of the Z-value seems to flout intuitive reason: surely students who make use of additional learning resources would be expected to perform better? The table shows that students who did not make use of CBT actually obtained better scores, on average, than students who did.

Some reflection revealed a possible alternative explanation: possibly the weaker students require the additional resources more than the brighter or better skilled students who either already possess a lot of the required skills, are quicker to pick them up from the text or lectures, or prefer to learn by doing. That the student group who made use of CBT is indeed a weaker group is shown if their average scores for the theory tests are compared: 54.1% (st.dev.= 13.3%) against an average of 59.9% (12.4%) for the students who claim that they didnt make use of CBT (Z-score of 4.40 i.e. significant at 1%).

It must again be stressed that there is no direct evidence that the students did indeed use the CBT packages so the validity of the analysis may be questioned.

Overall, therefore, there is no conclusive statistical support for or against the added value of CBT on student performance.

For interests sake, the actual student evaluations of the CBT packages are listed in table 2.

From the table it appears that students were not overly impressed with either of the two CBT packages. Ratings for other course delivery components were generally much more positive.

After each theory lecture, a second lecture session attempts to make the material "come alive" by means of an "enrichment session". The emphasis of this session is on participatory or active learning. Two examples of the type of activities in these sessions are:

• Small individual or mini-group exercises, in effect requiring the students to apply their knowledge to small cases. Usually the answers have to be summarized on a structured answer sheet that must be handed in. This answer sheet is marked by a tutor and contributes only nominally to the student year mark.
• (Extracts of) educational videos where a number of questions are given prior to showing, to be answered individually.

The intention of the additional materials is to enhance the educational process through participatory learning units: students learn best when doing [Eccles & Van Belle, 1998]. It must be noted that the source data is again not very reliable: enrichment sessions were often done in two or three-person groups, often with a wide range in the actual contribution of various team members. Consequently, some students received marks that they did not really deserve. However, the overall contribution of the enrichment sessions towards the year mark was negligible (0.5%) to reduce the "doing-it-only-for-the-marks ethic prevalent among many students.

Table 3 lists the correlation coefficients of various course component marks with the final examination mark for 1999. Although the overall final course mark is perhaps a better measure of "course performance" than the final examination mark, it includes the weighted component marks, which means that there is a strong (but variable) element of auto-correlation.

Because of the large sample, all these correlation coefficients are statistically significant to the 1% level. However, the strength of the relationship (with the final examination mark) varies (using Behrs [1988] terminology) from a "high and substantial" relationship with the year mark, to a moderate relationship with the two tests or practical exam , to only a "definite but slight" relationship with the enrichment sessions.

A methodological problem, of course, is that the other course components were explicitly intended to be "evaluations" whereas the purpose of the enrichment sessions was not evaluation. Also, they were marked very generously, i.e. with a low discriminatory power. A separate analysis was done on the original data set (which is slightly larger, since not all enrichment sessions were included for mark purposes). But, although the correlation between the marks and final exam mark increases marginally to 40%, the correlation between the number of enrichment sessions handed in and final exam mark is only 29% (N=708)!

This last low correlation coefficient seems, therefore, to reduce the educational argument for the value of the enrichment sessions, definitely in as far as examination preparation goes. However, this does not invalidate any broader educational arguments as to the overall learning experience of the course.

Despite this initial disappointment, some further analysis was conducted. Instead of 1999, the 2000 data was used, because 1999 was an early attempt at introducing enrichment sessions and these sessions were much better developed and validated in 2000. Because final marks were not yet available at the time of writing, the course year mark was used instead. Table 4 shows the results. ("# lectures att./week" indicates the average number of lectures attended by the students using their own estimate as given in the course student evaluations see further.)

This analysis shows a clear pattern: the more hand-ins (and, by implication, the more regular the class attendance?), the higher a students year mark. Students who handed in less than 3 enrichment session deliverables obtained a year mark that was on average a full 18% less than those students who handed in all of the exercises, despite the fact that all these hand-ins together contributed less than 5% to their mark. For statistical analysis, rows were merged as per table 7.

The difference between these averages is statistically significant at the 1% level (the Z-score being an extremely high 8.89). Although it is believed that part of this difference must be explained by the positive benefit of doing the enrichment sessions, another contributing factor is the obviously the motivation and work attitude of the more conscientious student.

It is extremely difficult to monitor class attendance for large groups of students. However, the benefit of actual class attendance for a course where about 80% to 90% of the examinable material is available in the textbooks, is a point of intense debate both among academic staff and students. Hence it was thought important to attempt to obtain even an indication.

A first measure for "class attendance" was through a question on the course evaluation questionnaire: "On average, how many lecture sessions did you attend each week?" with answer options ranging from once to five times per week (no student selected the option "less than once per week"). Since final course marks (including examinations) are not yet available, the course year mark was used as a measure of course performance. Table 6 tabulates the average year marks for various student subgroups based on the number of times per week they claimed to have attended lectures. The "# ER Hand-ins" refers to the number of enrichment session exercises that were submitted (see earlier).

As can be seen from the table, there appears to be little correlation between the average weekly number of lectures students claim to attend and their year mark. The average year mark of the 30 students who admitted to attending on average only one lecture a week is 71.7%, which is almost identical to the average year mark of the 118 students who attended four lectures each week. The only notable exception is the group of 45 students that attended five lectures a week: their year mark is on average 4.4% above the norm for the entire group. It must be noted that these students appear to be exceptionally motivated, since the fifth (Friday) weekly lecture is an optional review session presented by a post-graduate student. This difference is the only one that is statistically significant (at the 1% level) and probably measures the positive effect of the optional Friday sessions.

Nevertheless, the overall lack of correlation seems to negate the value of any lecture attendance. However, there is some doubt about the validity of student claims as to how many lectures they claim to attend. For instance, there is one student who claimed to have attended all lectures but never handed in any of the weekly "enrichment session" deliverables! The last column in table 4 was therefore added to indicate the number of weekly enrichment session hand-ins for each of the sub-groups of students. This shows some but definitely not a very strict correlation: the actual correlation coefficient between the number of lectures students claim to attend and the number of enrichment exercises handed in is a relatively low 37.4%.

Hence, if the analysis was repeated using the number of enrichment session marks obtained by an individual as an alternative proxy for class attendance, one obtains the results as reflected in tables 4 and 5, that do indicate a very strong positive effect. However, the "interpretative leap" required to use "number of enrichment session hand-ins" as a proxy for class attendance is subject to considerable methodological doubt and has to be balanced against the bias of students own statements of class attendance.

For a number of years, until the end of 1999, the mastery method was adopted for the practical component of the course.

The mastery method divides the course material into a number of modules, each with detailed learning objectives. Student performance can then be evaluated on a regular basis against these objectives and students only are permitted to progress once they have "mastered" the relevant material. The main value of the mastery method is that it forces students to take immediate action when they do not understand material presented on the course. Where knowledge and understanding is incremental, it is most important that students having difficulties at the start of the course are given assistance as soon as possible and then given an opportunity to be re-evaluated [Eccles & Van Belle, 1998].

Students had to master all four marked tutorials (obtain a mark of at least 70%). If a student did not pass the tutorial at their first attempt, they had to attend the following Monday revaluation session and rewrite the tutorial. Students only had two revaluation attempts for each of the tutorials. For year mark calculations, lower marks of 50% and 0% respectively were awarded where students mastered (obtain at least 70%) the tutorial on their first or second revaluation attempt. Students who did not master all four tutorials, did not obtain a DP and were barred from writing the final course examinations. In 2000 this mastery requirement was dropped due to extreme workload overheads and administrative burden that the mastery method implies.

The new, much more relaxed requirement for the 2000 academic year is that a student merely had to obtain an average of 50% for all (seven) marked tutorial assignments. It is illuminating to see the effect of this relaxation on actual student performance for the practical examination mark. The standard of the practical examination in 2000 was as close to that of the 1999 examination as possible. In both cases the first semester course was selected who have comparable student bodies. (The analysis was repeated by excluding the small number of non-business science / non-IS major students but no real different results were obtained.)

The a priori hypothesis would be that the mastery method has a positive impact on practical exam marks. Figure 1 shows the actual distributions of the practical examination marks, using 10% intervals, for the 1999 (with mastery method) and 2000 (no mastery method) courses. The 2000 distribution shows a clear shift to the left indicating lower marks, as hypothesised. The c ²-value for the two distributions is 72.84, which is statistically very significant, i.e. the hypothesis that the two distributions come from the same population should be rejected strongly (the critical value for 6 degrees of freedom at the 1% confidence level being a mere 16.81).

Table 7 tabulates the two key statistical measures characterising the differences between the two distributions: the 2000 distribution has a substantially lower average and is much flatter i.e. has a wider spread (standard deviation).

The dropping of the mastery method resulted in an average drop of 5.7% in marks. The Z-score is again an impressive ()8.66 which is naturally significant at the 1% level. Note that, as an independent control variable to ascertain the similar composition and capability of the two student groups, the average for the two theory tests for the two years were 56.0% (2000) and 56.5% (1999) respectively, confirming that the two groups are in fact comparable in other respects.

The above result presents a strong case in favour of the mastery method. However, it must be realised that this approach is very resource intensive in terms of labour, administration, computer laboratory use and course co-ordination. Since the number of students who actually fail the practical exam is relatively small, the real world cost-benefit analysis of the mastery method will depend on the organisational and educational context. As UCT students seem to be much more computer literate now (see also Eccles & Van Belle, 2000), there is less justification for the use of mastery method than there was when it was first adopted in the earlier nineties.

Although the data available is not always pure, i.e. it was not always possible to isolate individual variables within the data sets, the following conclusions seem warranted.

• There is a strong support in favour of the educational benefit of the use of the mastery method in imparting computer skills.
• There seems to be support in favour of the use of well-designed and validated enrichment sessions to enforce the theory component of the course, although other variables such as student attitude and motivation may have played a role.
• There was no conclusive evidence supporting or negating the value of computer-based training packages in the course.
• The case for the value of attending lectures could not be made clearly, due to the lack of an objective measure. Optional review sessions did seem to impart measurable benefits.

It is hoped that colleagues at sister institutions will be able to verify or elaborate on these results and add evidence on their applicability in different contexts.