TEACHING AN INFORMATION SYSTEMS FOUNDATIONS COURSE: WHAT WORKS AND WHAT DOESN'T.

Jean-Paul Van Belle, University of Cape Town

ABSTRACT

This paper looks at three teaching strategies for a foundations information system course and evaluates their impact on actual student performance. The course used is a first year one-semester introductory under-graduate course in Information Systems at the University of Cape Town, taught by the author. In this paper, three delivery options have been evaluated: the use of computer-based training, the "mastery" method and the use of enrichment sessions. It appears as if only the adoption of the "mastery" method seems to have had an unequivocal influence on student performance, although there is some evidence in favour of enrichment session, assuming that they have been validated and refined through repeated use.

INTRODUCTION

Often educators implement alternative teaching strategies without evaluating the educational effectiveness of the new course of action. This paper evaluates the impact of some of these educational strategies and curriculum options for course delivery on student performance in a first year information systems course.

This paper analyses the effect of three different educational approaches on student performance in a first-year one semester IS course, the "Foundations of Information Systems" course INF1002F at the University of Cape Town (UCT): computer-based training; the use of the "mastery" method and the use of enrichment sessions. It must be noted that an educational environment hardly ever allows for controlled laboratory-like circumstances and hence statistical interpretation has certain limitations in evaluating educational strategies.

DATA SOURCE AND COURSE STRUCTURE

The INF102 "Foundations of Information Systems" course at UCT is offered three times each academic year. in the first semester to roughly 750 business science and information systems (IS) majors in the commerce faculty, in the second semester to about 500 B.Com accounting and humanities students, and spread over the entire year with a focus on providing a slower paced small group learning environment to about 100 ADP (academic development program) and post-graduate diploma students. Detailed information about this course (objectives, curriculum, practical organisation, requirements etc.) can be found on the course website at http://www.commerce.uct.ac.za/InformationSystems/courses/INF101/. The course consists of two distinct modules: the practical component (spreadsheet and database skills) and the theory component (basic IS concepts, IT technology and the organisational context of IS).

The data for this study is drawn mainly from the departmental records. This includes optional student evaluations submitted on-line after writing the second theory test during the last week of lectures. The questionnaire was administered and processed using the QuestionMark computer testing software.

THE EFFECT OF COMPUTER-BASED TRAINING

INF102 students are given access to computer-based training (CBT) to complement the textbook, lectures and laboratory assistance Two commercially available suites were installed: Net-G (http://www.netg.com; http://websvm.its.uct.ac.za/Webcal) and Individual Training (http://www.individualsoftware.com). The use of these CBT packages was entirely at the discretion of the student. The availability of the CBT resources was advertised in the first week of lectures and at occasional intervals later during the course.

The Individual Training software consists of 4 modules covering MS-Windows, MS-Word, MS-Excel and MS-Access. Each module consists of about 8 to 12 hours of interactive training and is followed by an optional assessment test. The modules run as stand-alone tutorials which are loaded directly from a local file server. The Net-G suite is a set of much more comprehensive and in-depth Internet-based tutorials which are accessible both on- and off-campus to all registered UCT students from a central ITS server. However, Net-G requires a fairly elaborate registration and downloading process and the use of an Internet browser.

In an "ancient" study, supplementing class lectures with computer-based PLATO tutorials was found to have a statistically significant positive impact although that particular study was based on subject matter which required analytical understanding (financial break-even analysis) instead of skill development [Van Belle, 1986].

It would be expected that CBT contributes positively to learners performance. Chart 1 below lists the average marks obtained in the practical component of the course (the subject area addressed by the CBT), tabulated according to students who indicated that they had used both, one or none of the CBT packages respectively. ("Prac Exam Avge Grade" refers to the grade obtained in the final two-hour practical exam written by all students, whereas the "Tutorial Avge Grade" refers to a weighted average mark of practical exam and 7 marked tutorials.) The standard deviations for the sub-groups ranged from 11.6% to 13.3% (Tutorial Average Grade) and 12.0% to 14.1% (Prac Exam) with the lower values obtained in the group "No CBT" and the highest values in the groups that used both CBT packages.

If the average results of the students who used at least one CBT package are compared to those who did not by means of the Z-test (difference between means of large samples), Z-values of -3.14 (Tutorial Avge Grade) and 3.60 (Prac Exam Avge Grade) are obtained. This indicates that the chance of both samples being drawn from the same population is statistically significantly less than 1%. However, the sign of the Z-value seems to flout intuitive reason: surely students who make use of additional learning resources would be expected to perform better instead of worse?

Some reflection revealed a possible alternative explanation: possibly the weaker students require the additional resources more than the brighter or better skilled students who either already possess a lot of the required skills, are quicker to pick them up from the text or lectures, or prefer to learn by doing. That the student group who made use of CBT is indeed a weaker group is shown if their average scores for the theory tests are compared: 54.1% (st.dev.= 13.3%) against an average of 59.9% (12.4%) for the students who claim that they didnt make use of CBT (Z-score of 4.40 i.e. significant at 1%).

Overall, therefore, there is no conclusive statistical support for or against the added value of CBT on student performance.

THE EFFECT OF "IN-CLASS ENRICHMENT SESSIONS".

Half of the theory lectures are" enrichment sessions": attempts to make the material "come alive". The emphasis of this session is on participatory or active learning. Some activities are small individual or mini-group exercises, in effect requiring the students to apply their knowledge to small cases. Usually the answers have to be summarized on a structured answer sheet that must be handed in. This answer sheet is marked by a tutor and contributes only nominally to the student year mark. Another example is where a video is shown along with a number of questions (given prior to the viewing), to be answered individually. The intention of the additional materials is to enhance the educational process through participatory learning units: students learn best when doing [Eccles & Van Belle, 1998].

Chart 2 shows a clear pattern: the more hand-ins (and, by implication, the more regular the class attendance?), the higher a students year mark. Students who handed in less than 3 enrichment session deliverables obtained a year mark that was on average a full 18% less than those students who handed in all of the exercises, despite the fact that all these hand-ins together contributed less than 5% to their mark. For statistical analysis (c ² test) subgroups were merged into students who submitted 4 or less ER hand-ins (average year mark = 65.2%, st.dev. = 12.4%) and those students who submitted 5 or more (average = 75.9%, st.dev. = 9.6%). The difference between these averages is statistically significant at the 1% level (the Z-score being an extremely high 8.89). Although it is believed that part of this difference must be explained by the positive benefit of doing the enrichment sessions, another contributing factor is the obviously the motivation and work attitude of the more conscientious student.

THE EFFECT OF THE MASTERY METHOD.

From 1993 until the end of 1999, the mastery method was adopted to teach and evaluate the practical component of INF102. Although the mastery method is commonly employed in computer-based training, it is not so common in the traditional class room environment.

The mastery method divides course material into a number of well-defined modules (for INF102: four modules), each with detailed learning objectives. Student performance is then evaluated on a regular basis against these objectives and students only are permitted to progress to the next module once they have "mastered" the relevant material (the required grade for INF102 was set at a relatively high 70%). The main value of the mastery method is that it forces students to take immediate action when they do not understand material presented on the course. Where knowledge and understanding is incremental, it is most important that students having difficulties at the start of the course are given assistance as soon as possible and then given an opportunity to be re-evaluated [Eccles & Van Belle, 1998]. Mastery of all four modules was required in order to be allowed to write the final examinations.

In 2000 this mastery requirement was dropped due to extreme workload overheads and administrative burden that the mastery method implies. The new, much more relaxed requirement for the 2000 academic year is that a student merely had to obtain an average of 50% for all (seven) marked tutorial assignments. It is illuminating to see the effect of this relaxation on actual student performance for the practical examination mark. The standard of the practical examination in 2000 was as close to that of the 1999 examination as possible. In both cases the first semester course was selected who have comparable student bodies.

The a priori hypothesis would be that the mastery method has a positive impact on practical exam marks. Figure 3 shows the actual distributions of the practical examination marks, using 10% intervals, for the 1999 (with mastery method) and 2000 (no mastery method) courses. The 2000 distribution shows a clear shift to the left indicating lower marks, as hypothesised. The c ²-value for the two distributions is 72.84, which is statistically very significant, i.e. the hypothesis that the two distributions come from the same population should be rejected strongly (the critical value for 6 degrees of freedom at the 1% confidence level being a mere 16.81).

The 2000 distribution has a substantially lower averageand is flatter i.e. has a wider spread: 1999 (mastery method) average = 82.2% (st.dev = 13.7%) and 2000 (without mastery method) average = 76.5% (st.dev = 10.9%). The dropping of the mastery method resulted in an average drop of 5.7% in marks. The Z-score is an impressive ()8.66 which is naturally significant at the 1% level. Note that, as an independent control variable to ascertain the similar composition and capability of the two student groups, the average for the two theory tests for the two years were 56.0% (2000) and 56.5% (1999) respectively, confirming that the two groups are in fact comparable in other respects.

The above result presents a strong case in favour of the mastery method. However, it must be realised that this approach is very resource intensive in terms of labour, administration, computer laboratory use and course co-ordination. Since the number of students who actually fail the practical exam is relatively small, the real world cost-benefit analysis of the mastery method will depend on the organisational and educational context. As UCT students seem to be much more computer literate now (see also Eccles & Van Belle, 2000), there is less justification for the use of mastery method than there was when it was first adopted in the earlier nineties.

CONCLUSION

Despite some methodological concerns - the educational environment hardly allows for laboratory-like controlled conditions, the following conclusions seem warranted.

There is a strong support in favour of the educational benefit of the use of the mastery method in imparting computer skills. There seems to be some support in favour of the use of well-designed and validated enrichment sessions to enforce the theory component of the course, although other variables such as student attitude and motivation may have played a role. There was no conclusive evidence supporting or negating the value of computer-based training packages in the course.

Bearing in mind that this is just one isolated case study, this and similar research may nevertheless inform the decision making process of teachers at other tertiary institutions in respect of their teaching strategies and resource allocations.

REFERENCES (abbreviated)

Behr A.L. Empirical Research Methods for the Human Sciences. Durban: Butterworths, 1988.

Eccles M. & Van Belle J.P. Innovative Design and Delivery of an IS Foundation Course. Proceedings of the 28th Conference of the SA Computing Lecturers Association. Stellenbosch: 28-30 June 1998.

Van Belle J.P. The Use of CBE in Business Economics: experiences at UWC. Proceedings of the Conference on "Voortreffelikheid in Onderrig en Leer in Tersiêre Onderwys", 2-4 April 1986, Stellenbosch.