AN EVALUATION OF TEACHING STRATEGIES FOR AN INFORMATION SYSTEMS FOUNDATIONS COURSE.

Jean-Paul Van Belle, University of Cape Town

ABSTRACT

This paper looks at three teaching strategies for a foundations information system course and evaluates their impact on actual student performance. The course used is a first year one-semester introductory under-graduate course in Information Systems at the University of Cape Town, taught by the author. In this paper, three delivery options have been evaluated: the use of computer-based training, the "mastery" method and the use of enrichment sessions. It appears as if only the adoption of the "mastery" method seems to have had an unequivocal influence on student performance, although there is some evidence in favour of enrichment session, assuming that they have been validated and refined through repeated use.

INTRODUCTION

Too often educators implement alternative teaching strategies without attempt at an ex-post evaluation of the educational effectiveness of the new course of action (Nelson, 1999). Two years ago, we reported (Eccles & Van Belle, 1998) on a number of "innovative approaches" adopted in a first-year information systems course. In this paper, we evaluate the impact of some of these educational strategies and curriculum options for course delivery on student performance within the same course. These results have been presented in summery form at a previous conference and are now reported on more fully.

It has to be recognised that an educational environment almost never allows for laboratory-like circumstances and hence statistical analysis has to be hedged with numerous ifs and buts. Nevertheless, it is hoped that the tentative results presented in this paper will allow educators at other tertiary educational institutions to make more informed decisions on which educational approaches to take and perhaps to spur them on to report on their experiences.

This paper analyzes the effect of three different educational approaches on student performance in a first-year one semester IS course, the "Foundations of Information Systems" course INF1002F at the University of Cape Town:

• The "mastery" method
• The use of enrichment sessions.
• Computer-based training

The analysis is confined to comparative statistical analysis, leaving out many other qualitative side-effects (mostly positive?) of the teaching strategies.

DATA SOURCE AND COURSE STRUCTURE

The INF102 course consists of three identical IS foundation courses. INF1002F is offered in the first semester to roughly 750 business science and information systems majors in the commerce faculty, the INF1002S course is offered in the second semester to about 500 B.Com accounting and humanities students, and the INF1002H course is spread over the entire year with a focus on providing a slower paced small group learning environment to about 100 ADP (academic development program) and post-graduate diploma students. Detailed information about this course (objectives, curriculum, practical organisation, requirements etc.) can be found on the course website at http://www.commerce.uct.ac.za/InformationSystems/courses/INF101/. The course consists of two distinct modules: the practical component (spreadsheet and database skills) and the theory component (basic IS concepts, IT technology and the organisational context of IS).

The data for this study is drawn mainly from the departmental records of the INF1002F course for the academic years 2000 and 1999. The course year mark for 2000 is made up of 15 distinct course deliverables: 7 marked practical tutorial assignments (3 spreadsheets and 4 database exercises), 7 "enrichment session hand-ins", 2 theory tests and the final practical examination. Although the number of students registered for the course stand at more than 700, analysis is limited to a subset of the students usually where data from student evaluations has to be used. Students completed an optional course evaluation questionnaire on-line after writing the second theory test during the last week of lectures. The questionnaire was administered and processed using the QuestionMark computer testing software. The main advantage of this approach is that it allows for the unobtrusive identification of each student and thus the linking of student responses to student performance data. About 400 students provided usable course evaluations.

ACTIVE LEARNING: THE USE OF "IN-CLASS ENRICHMENT SESSIONS".

Bloom (1956), in his well-known taxonomy, identified six cognitive domains. Since ex cathedra-based lecturing does not generally stimulate the higher order cognitive activities, an active learning approach was followed to engage the student's mind. After each theory lecture, a second lecture session later during the week (allowing thus for additional student preparation and review) attempts to make the material "come alive" by means of an "enrichment session". The emphasis of this session is on participatory or active learning. Two examples of the type of activities in these sessions are:

• Small individual or mini-group exercises, in effect requiring the students to apply their knowledge to small cases. Usually the answers have to be summarized on a structured answer sheet that must be handed in. This answer sheet is marked by a tutor and contributes only nominally to the student year mark.
• "Action research" exercises whereby students have to determine the best configuration (specifications and cost) for a home business PC or answer a list of questions using the Internet.

The intention of the additional materials is to enhance the educational process through participatory learning units: students learn best when doing (Mills-Jones, A 1999; Bentley et al., 1999). It must be noted that the source data is not very reliable: enrichment sessions were often done in two or three-person groups, often with a wide range in the actual contribution of various team members. Consequently, some students received marks that they did not really deserve. However, the overall contribution of the enrichment sessions towards the year mark was negligible (0.5%) to reduce the "doing-it-only-for-the-marks ethic prevalent among many students.

Table 1 lists the correlation coefficients of various course component marks with the final examination mark for 1999. Although the overall final course mark is perhaps a better measure of "course performance" than the final examination mark, it includes the weighted component marks, which means that there is a strong (but variable) element of auto-correlation.

Because of the large sample, all these correlation coefficients are statistically significant to the 1% level. However, the strength of the relationship between course component and the final examination mark varies, using Behrs (1988) terminology, from a "high and substantial" relationship with the year mark, to a moderate relationship with the two tests or practical exam , to only a "definite but slight" relationship with the enrichment sessions.

A methodological problem, of course, is that the other course components were explicitly intended to be "evaluations" whereas the purpose of the enrichment sessions was not evaluation. Also, they were marked very generously, i.e. with a low discriminatory power. A separate analysis was done on the original data set (which is slightly larger, since not all enrichment sessions were included for mark purposes). But, although the correlation between the marks and final exam mark increases marginally to 40%, the correlation between the number of enrichment sessions handed in and final exam mark is only 29% (N=708)!

This last low correlation coefficient seems, therefore, to reduce the educational argument for the value of the enrichment sessions, definitely in as far as examination preparation goes. However, this does not invalidate any broader educational arguments as to the overall learning experience of the course.

Despite this initial disappointment, some further analysis was conducted. Instead of 1999, the 2000 data was used, because 1999 was an early attempt at introducing enrichment sessions and these sessions were much better developed and validated in 2000. Because final marks were not yet available at the time of writing, the course year mark was used instead. Figure 1 shows the results.

This analysis shows a clear pattern: the more hand-ins (and, by implication, the more regular the class attendance?), the higher a students year mark. Students who handed in less than 3 enrichment session deliverables obtained a year mark that was on average a full 18% less than those students who handed in all of the exercises, despite the fact that all these hand-ins together contributed less than 5% to their mark. For statistical analysis, rows were merged as per table 2. (The exact cut-off value for the grouping does not significantly affect the Z-score.)

The difference between these averages is statistically significant at the 1% level (the Z-score being an extremely high 8.89). Although it is believed that part of this difference must be explained by the positive benefit of doing the enrichment sessions, another contributing factor is the obviously the motivation and work attitude of the more conscientious student.

OJBECTIVE-BASED LEARNING: THE USE OF THE MASTERY METHOD

For a number of years, until the end of 1999, the mastery method was adopted for the practical component of the course.

The mastery method divides the course material into a number of modules, each with detailed learning objectives. Student performance can then be evaluated on a regular basis against these objectives and students only are permitted to progress once they have "mastered" the relevant material. The main value of the mastery method is that it forces students to take immediate action when they do not understand material presented on the course. Where knowledge and understanding is incremental, it is most important that students having difficulties at the start of the course are given assistance as soon as possible and then given an opportunity to be re-evaluated (Eccles & Van Belle, 1998).

Students had to master all four marked tutorials (obtain a mark of at least 70%). If a student did not pass the tutorial at their first attempt, they had to attend the following Monday revaluation session and rewrite the tutorial. Students only had two revaluation attempts for each of the tutorials. For year mark calculations, lower marks of 50% and 0% respectively were awarded where students mastered (obtain at least 70%) the tutorial on their first or second revaluation attempt. Students who did not master all four tutorials, did not obtain a DP and were barred from writing the final course examinations. In 2000 this mastery requirement was dropped due to extreme workload overheads and administrative burden that the mastery method implies.

The new, much more relaxed requirement for the 2000 academic year is that a student merely had to obtain an average of 50% for all (seven) marked tutorial assignments. It is illuminating to see the effect of this relaxation on actual student performance for the practical examination mark. The standard of the practical examination in 2000 was as close to that of the 1999 examination as possible. In both cases the first semester course was selected who have comparable student bodies. (The analysis was repeated by excluding the small number of non-business science / non-IS major students but no real different results were obtained.)

The a priori hypothesis would be that the mastery method has a positive impact on practical exam marks. Figure 2 shows the actual distributions of the practical examination marks, using 10% intervals, for the 1999 (with mastery method) and 2000 (no mastery method) courses. The 2000 distribution shows a clear shift to the left indicating lower marks, as hypothesised. The c ²-value for the two distributions is 72.84, which is statistically very significant, i.e. the hypothesis that the two distributions come from the same population should be rejected strongly (the critical value for 6 degrees of freedom at the 1% confidence level being a mere 16.81).

Table 3 tabulates the two key statistical measures characterising the differences between the two distributions: the 2000 distribution has a substantially lower average and is much flatter i.e. has a wider spread (standard deviation).

The dropping of the mastery method resulted in an average drop of 5.7% in marks. The Z-score is again an impressive ()8.66 which is naturally significant at the 1% level. Note that, as an independent control variable to ascertain the similar composition and capability of the two student groups, the average for the two theory tests for the two years were 56.0% (2000) and 56.5% (1999) respectively, confirming that the two groups are in fact comparable in other respects.

The above result presents a strong case in favour of the mastery method. However, it must be realised that this approach is very resource intensive in terms of labour, administration, computer laboratory use and course co-ordination. Since the number of students who actually fail the practical exam is relatively small, the real world cost-benefit analysis of the mastery method will depend on the organisational and educational context. As UCT students seem to be much more computer literate now, there is less justification for the use of mastery method than there was when it was first adopted in the earlier nineties.

THE USE OF COMPUTER-ASSISTED TRAINING

Many claims and evidence in support of computer-assisted learning/training (CAL) have been made. (Goldberg, 1997). The author has extensive experience in the use of computer-supported education. In an "ancient" study, supplementing class lectures with computer-based PLATO tutorials was found to have a statistically significant positive impact although that particular study was based on subject matter which required analytical understanding (financial break-even analysis) instead of skill development (Van Belle, 1986).

To complement the textbook, lectures and laboratory assistance, students were given access to computer-based training (CBT) packages from two commercially available suites: Net-G and Individual Training. The use of these CBT packages was entirely at the discretion of the student. The availability of the CBT resources was mentioned in the first week of lectures and at occasional intervals later during the course.

The Individual Training software consists of 4 modules covering MS-Windows, MS-Word, MS-Excel and MS-Access. Each module consists of about 8 to 12 hours of interactive training and is followed by an optional assessment test. The modules run as stand-alone tutorials which are loaded directly from a local file server. The Net-G suite is a set of much more comprehensive and in-depth Internet-based tutorials which are accessible both on- and off-campus to all registered UCT students from a central ITS server. However, Net-G requires a fairly elaborate registration and downloading process and the use of an Internet browser.

Due to technical reasons it is not possible to track which students actually used the CBT software. However, two of the questions on the course evaluation questionnaires pertained to the student experience with the CBT packages. It must be stressed that there is no evidence that students who responded to this question, actually did use the packages.

If one hypothesises that CBT contributes positively to learners performance, the corresponding null hypothesis states that there is no difference between the average marks obtained by students who used CBT vis-à-vis those who didnt. Table 4 below lists the average marks obtained in the practical component of the course (the subject area addressed by the CBT), tabulated according to students who indicated that they had used both, one or none of the CBT packages respectively. Note that the "Prac Exam" refers to the final two-hour practical exam written by all students, whereas the "Prac Avge" refers to a weighted average mark of practical exam and 7 marked tutorials.

If the average results of the students who used at least one CBT package are compared to those who did not by means of the Z-test (difference between means of large samples), Z-values of -3.14 (Prac Avge) and 3.60 (Prac Exam) are obtained. This indicates that the chance of both samples being drawn from the same population is statistically significantly less than 1%. However, the sign of the Z-value seems to flout intuitive reason: surely students who make use of additional learning resources would be expected to perform better? The table shows that students who did not make use of CBT actually obtained better scores, on average, than students who did.

Some reflection revealed a possible alternative explanation: possibly the weaker students require the additional resources more than the brighter or better skilled students who either already possess a lot of the required skills, are quicker to pick them up from the text or lectures, or prefer to learn by doing. That the student group who made use of CBT is indeed a weaker group is shown if their average scores for the theory tests are compared: 54.1% (st.dev.= 13.3%) against an average of 59.9% (12.4%) for the students who claim that they didnt make use of CBT (Z-score of 4.40 i.e. significant at 1%); although the gap (as measured by the standardised difference between the two averages) seems to be slightly narrower for the practical component.

It must again be stressed that there is no direct evidence that the students did indeed use the CBT packages so the validity of the analysis may be questioned.

Overall, therefore, there is no conclusive statistical support for or against the added value of CBT on student performance. It is also interesting to note that the students did not seem too greatly impressed with the two CBT packages (see Figure 3); perhaps this is due to the lack of specific emphasis or motivation given by the teacher for the use of the CBT packages? Ratings for other course delivery components were generally much more positive.

CONCLUSION

Although the data available is not always pure, i.e. it was not always possible to isolate individual variables within the data sets, the following conclusions seem warranted.

• There is a strong support in favour of the educational benefit of the use of the mastery method in imparting computer skills.
• There seems to be support in favour of the use of well-designed and validated enrichment sessions to enforce the theory component of the course, although other variables such as student attitude and motivation may have played a role.
• There was no conclusive evidence supporting or negating the value of computer-based training packages in the course.

It is hoped that colleagues at sister institutions will be able to verify or elaborate on these results and add evidence on their applicability in different contexts. It is also important to note that there are also additional positive educational effects from implementing these strategies that are not necessarily measured in student performance in tests and exams. However, these do not lend themselves easily to comparative statistical analysis.

REFERENCES

Behr A.L. (1988) Empirical Research Methods for the Human Sciences. Durban: Butterworths.

Bentley JF, Lowry GR & Sandy GA. (1999) Towards The Compleat Information Systems Graduate: A Problem Based Learning Approach. Proceedings of the 10^th Australasian Confrence on Information Systems, Wellington (New Zealand), p.65-75.

Bloom B et. al. (1956) Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook 1 Cognitive Domain. London: Longmans.

Eccles M. & Van Belle J.P. (1998) Innovative Design and Delivery of an IS Foundation Course. Proceedings of the 28th Conference of the SA Computing Lecturers Association. Stellenbosch (South Africa), p.123-129.

Goldberg M.W. (1996) CALOS: First Results from an Experiment in Computer-Aided Learning. Proceedings of the ACM's 28^th SIGCSE Technical Symposium on Computer Science Education. (Also available from http://hoebrew1.cs.ubc.ca/webct/papers).

Individual Training. http://www.individualsoftware.com.

Mills-Jones A. (1999) Active Learning in IS Education: Choosing Effective Strategies for Teaching Large Classes in Higher Education. Proceedings of the 10^th Australasian Confrence on Information System. Wellington (New Zealand), p.622-633.

Nelson MR. (1999). Effective Teaching Strategies, Teaching Assistant Program, Michigan State University. URL http://www.msu.edu/~taprog/ch4.htm.

Net-G Skill Builder. http://www.netg.com. See also http://websvm.its.uct.ac.za/Webcal/ for how Net-G is used at U.C.T.

Van Belle J.P. (1986) The Use of CBE in Business Economics: experiences at UWC. Conference on "Voortreffelikheid in Onderrig en Leer in Tersiêre Onderwys". Stellenbosch (South Africa).

Jean-Paul Van Belle, 2000.