Expert systems technology has been available to the South African information systems community for a number of years. The early promises and enthusiasm about its capabilities have been countered by the apparent lack of visible systems. This research attempts to establish, by means of a survey, what its current status is amongst organizations who have been associated with the technology. The results are validated against previous empirical research. Overall, a perhaps surprisingly positive response was obtained, revealing a wide variety of fairly large, sophisticated and successful systems.

 

 

In spite of the high hopes, the over-optimistic "final come-back" announcements remained rebutted by the conspicuous lack of actual mainstream business applications. Konsynski [2:4] summarizes it neatly when he notes that:

"The reports of over 1,500 [US] current active expert systems projects seem impressive, until an investigation of a number of these efforts fails to surface more than a handful that have actually validated an organizational impact."

But now again, a more cautious optimism seems to permeate throughout both technical and business literature. These sentiments are echoed by Buckler [3] which acknowledges the steady inroads that expert systems are making into the real business environment: "Expert systems have finally left the hospital and been allowed to come into the office." Is this another premature "break-through" claim or is the comeback of expert systems after what some have called the "AI winter" for real? [4] This survey tries to find out.

2. SCOPE AND OBJECTIVES OF THE RESEARCH

2.1 Scope and Objective of the Survey

A survey of published research shows that the majority of the ES literature can be classified in one of the following categories [5].

- technical literature e.g. knowledge engineering techniques, reviews of system development shells or ES validation techniques;

- work of a theoretical nature e.g. advances in the methods for representing knowledge or new approaches for reasoning under uncertainty; or

- descriptive ES case studies of a "biographical" nature e.g. describing the history of the MYCIN system or du Pont's experience with ES.

But the lack of a sufficiently large critical mass of commercial expert systems has resulted in a relative paucity of survey-type empirical ES research [6]. Nevertheless, three broad "types" of ES surveys can be distinguished in the literature.

1) General surveys that sample a large, non-specific population of organizations with the aim of measuring to what degree ES technology has penetrated the IS environment. An example is Keating [7] who targeted her survey at "South African DP installations". A major objective of this research is to determine to what extent ES are being used; ES users are typically only a small percentage of the population.

2) So-called "saturation" surveys, directed at informed organizations or individuals who are already familiar with ES technology to probe the current state of the art in the ES field. The model survey for this type is Philip & Schultz [8] who surveyed only recipients of the AI Expert magazine. Here a substantial proportion of the sample develops or uses ES.

3) More focused or targeted surveys which are aimed at a specific type of ES or group of users. This type of survey focuses solely on e.g. strategic ES, hybrid ES, or a particular industry. The series of bi-annual surveys conducted by Coopers & Lybrand in the US insurance industry exemplifies this type of survey.

In the South African context, a successful survey of the first type [Keating, 1991] has recently been published. It indicated a low level of ES use. A personal interest in computerized decision support systems and market signals indicating a pick-up in ES activity, prompted the authors to revisit the issue. Consequently, it was decided to conduct a new, comprehensive survey with the objective to investigate the current state of ES application and usage within South African organizations that have expressed an interest in ES technology i.e. a "type 2" survey, the first of its kind in South Africa.

In the light of the above, it must be recognized that the research findings can not be extrapolated to all South African organizations. Only organizations who are known to be informed about ES technology were targeted for this research.

In addition, the general lack of empirical research data may hamper the ability to draw statistically meaningful comparisons. In particular, the lack of a previous large scale South African survey prevents the identification of historical trends. Hopefully, this study will remedy this situation for future research.

Finally, it is conceivable that isolated "pockets of ES excellence" may have been missed through the selection of a particular sampling frame. That this is not inconceivable, even in the relatively small South African IS industry, can be illustrated by the following (paraphrased) comment from ES consultant K.Carden:

"There may be things going on that hardly anyone knows about. For instance, everyone working in the [ES] field was surprised at the last [Expert 90] conference when a couple of speakers turned up to present a big, quite advanced system while no one even suspected that their organization was active in ES. And their work was really good, by any international standards. It was not that they had kept it specifically secret or so, they had just been quietly working on their own." (personal communication, March 1992)

The primary research hypothesis postulated in this research is as follows:

H_o: "Generally, very little ES activity in terms of system development and systems in actual use, is taking place in South Africa."

A number of secondary hypotheses operationalize the empirical verification of the primary hypothesis. They are specific enough to support or reject them on basis of the survey results. Many questions in the survey questionnaire have been formulated to relate directly to a specific secondary hypothesis.

Low ES activity implicates low project priorities, resulting in turn in a relative small amount of corporate resources being available. Hence it is expected that many systems will be relatively small in size. Although there are many ways in which the size of an ES can be measured, the length of a questionnaire usually limits the number of dimensions that can be measured.

H₁: The majority of ES are small. Development effort is measured in person-months rather than years; rule-based systems rarely exceed 100 inference rules.

In addition, many systems can be expected to be in the prototype or testing phase.

Because of the small amount of resources allocated, it may be expected that quick-and-easy, simple software development tools will be used. The hardware can also be expected to be at the bottom of the range.

H₂: Most ES are developed using low-end shells. These shells, as well as the systems developed with them, run on personal computers. Mainframe-based systems and ES custom-coded in traditional programming languages are the exception rather than the rule.

The interfacing of ES with other applications is a resource intensive and relatively sophisticated process.

H₃: The majority of ES are stand-alone systems. Systems that do interface, typically do so on a relatively low and safe level, i.e. they read but do not modify data from the corporate databases.

When interest and awareness is low, applications tend to be limited to areas with proven success stories where "innovators" have taken the lead.

H₄: The spread of ES applications will be relatively narrow, limited mainly to research environments or fields with historic success stories (medicine, computer systems, chemistry, electronics, and military).

When new technologies are introduced, the majority of the reported problems tend to be of a technical nature. Aspects such as financial viability, effectiveness, user interface etc. are a typical sign of a more mature technology.

H₅: Most problems that are experienced are developmental and relate to the technological aspects of the systems.

Organizations will be loath to develop strategic systems which depend on a relatively new, unproven technology.

H₆: Very few strategic ES are in place. Not many organizations have developed a clear role for future ES in their strategic plans.

The population identified for the survey was defined as "those organizations who are informed about the capabilities of ES technology." To determine an appropriate and manageable sampling frame, it is assumed that members of these organizations can be expected to have participated in conferences and user groups. Hence the following mailing lists were obtained through the kind cooperation of the various local Special Interest Groups for Artificial Intelligence (SIG-ART) of the Computer Society of South Africa (CSSA): SigArt Cape; SigArt Transvaal; SigArt Natal. In addition, a number of individuals kindly assisted in providing the following two valuable address lists: attenders of the EXPERT 90 conference; speakers for the EXPERT 92 conference. Duplicate names were removed manually. Quite a "nixies" were present in the some of the more dated lists. Some respondents passed copies of the questionnaire on to colleagues working in the field.

After a few informal talks with academics (UCT), users (Old Mutual, NBS) and a ES shell vendor (Level 5), two interviews were arranged with respected and experienced consultants who are active in the ES market: Mr. Rob Simpson (branch manager, Productivity Software, Cape Town; now with UC-2 Technologies) and Mr. Kenneth Carden (management consultant, Performance and Systems Improvement, Cape Town). These interviews provided the author with a much appreciated insight into the dynamics of the South African ES market and the valued time that these individuals made available is hereby gratefully acknowledged.

A questionnaire was designed around the secondary research hypotheses but in such a way as to allow maximum compatibility with the literature, specifically the Keating [1991] and Philip & Schultz [1990] surveys, whilst remaining as short as practically possible. The practical guidelines for questionnaire design suggested by Steenekamp [9] proved invaluable, as was the input gathered through the exploratory interviews. The questionnaire was in the form of a 4 page A5 booklet, sent with a postage paid reply envelope. There were various types of questions including open-ended, 5-point Likert scale, "list of statements" and "tick-the-applicable-box" type.

A total of 457 questionnaires were sent out by 6 April 1992, of which 37 were returned marked "undelivered". By April 27th, a total of 156 replies were received. This translates into a gross response rate of more than 37% which can be considered as extremely successful. This compares to the already high 20% response rate of Keating [1991:67] and the 14,7% recorded by Philip & Schultz [1990:57].

80 out of 153 respondents (or 52%; 3 replies could not be used) provided information on one or more expert systems in use or under development. These respondents will hereafter be referred to as "ES users". A total of 128 unique systems were reported. Of these, 72 (or 56%) are currently in use and 38 (30%) are under development; 15 (12%) are not in use and the fate of the remaining 3 (2%) is unknown. The questionnaire provided space for a maximum of 3 ES; 8 users indicated that they knew of more ES within their organization, adding up to another 26 undocumented ES.

4.2.1 Size Measured in Number of ES Rules

The number of rules of an ES is not an ideal measure of system size, especially not in the light of the growing importance of object-orientated and neural network-based ES. However, it allows easy comparison with other surveys. Table 1 lists the results which were obtained.

Table 2: Comparison of ES Size measured by Number of Rules

4.2.2 Development Time in "Person-months"

To gain a measure of corporate investment in the ES, the development time seemed the most appropriate measure. It was envisaged that overall financial investment would be more difficult to quantify by a respondent. Table 3 illustrates that there is a reasonable spread between the different categories and provides a comparison with the USA survey.

Table 3: Development time of ES

4.2.3 Relationship between Number of Rules & Development Time

Harmon & King [1985:201] claim that ES with 50 to 350 rules, developed using an ES tool, take approximately 3 to 6 months to develop as opposed to larger systems of 500 to 3000 rules which take 1 to 2 years. Given the data provided by the survey, this claim can be tested empirically through cross-tabulation. Note that some classes had to be grouped to ensure that each cell in the table would have an "expected frequency" of at least 5, which is desirable from a statistical viewpoint.

Table 4: Relationship # Rules - Development Time

4.3.1 Hardware platforms

The PC remains the most popular hardware platform for ES development (103 ES), although there are quite a few (17) mainframe-based ES around; against only 8 ES on mini (table 5).

4.3.2 Software system development tools

The respondents were given the opportunity to specify the software tools used in the development of their ES. The results (table 6) are very interesting: a wide variety of shells (87 ES) and traditional languages (34 ES) are being used with only a handful (14) employing "artificial intelligence" languages. 11 respondents did not answer this question, and 14 users indicated 2 or 3 categories of tools.

Philip & Schultz do not provide a detailed break-down, but they do report that 58% of their respondents use shells, 43% conventional languages, 38% Lisp and 33% Prolog. Although statistical comparisons suffer from similar drawbacks as in 4.3.1, they are more easily overcome. The statistical null-hypothesis that the respective proportions using a shell are different (RSA: 68%; USA: 58%), cannot be rejected at a 5% significance level although a visual inspection seemed to indicate this fact.

4.3.3 Use of ES Prototypes

The use of prototypes in IS development is a recognized step in IS development. This seems to be especially true for ES as table 7 indicates.

Table 7: Use of Prototypes for ES Development

A bias may have resulted from the fact that Philip & Schultz used the organization as the basis for analysis, against the "finer grain" ES-basis of this survey. Unfortunately this bias cannot be avoided without giving up the detail information in this survey although its effect is considered relatively minor. A potentially better explanation was hinted at by two respondents who indicated the use of a prototype for their oldest ES, but not for their more recently developed ES. Hence, it may be assumed that a prototype is usually necessary to motivate a first ES in order to overcome organizational scepticism; but less so when the technology has been proved and thus becomes more acceptable. This would indicate a more mature attitude. Yet another explanation take the opposite attitude: South African DP shops are less sophisticated (or more relaxed) and often cut corners in the SDLC, skipping prototyping "en route". Perhaps the most plausible explanation refers back to the observed greater occurrence of small ES: it does not make much sense in building a prototype for a (rule-based) ES with less than 50 rules!

4.3.4 The Knowledge Acquisition Process

Since no previous surveys provided any information on the knowledge acquisition process, it is interesting to note that most ES are being developed using a combination of methods. [10] [11] Of the 122 ES for which this question was completed, 43% used 1 method, 27% 2 methods, 24% 3 methods, 6% 4 methods and 2 systems used 5 different methods (table 8).

Perhaps the only surprise is the low priority given to computer-automated case analysis or induction. There are powerful statistical methods available to aid this process. This is also a built-in facility provided with the more powerful shells. Finally, neural technology could also make a significant contribution to this process. If maintenance of the knowledge base is an important concern for the future (as evident from a part of the survey not discussed here), then automated knowledge acquisition should receive more prominence. A final comment concerns the do-it-yourself ("other") approach, whereby the system developer doubles as knowledge engineer and subject expert. This is facilitated by the more user-friendly development shells which are GUI-based and/or incorporate OOPS technology. However, it remains a topic of controversy whether larger ES should be developed in this way.

Overall, the preceding results neither strongly confirm nor reject H₂.

4.4 Interfacing with Other Applications

The growing integration of ES with other applications is one of the important ES trends cited in the literature. [12] It was found that a considerable 57% of ES exchange data directly with other programs. This is an increase on the findings by Keating where only 35% (8 out of 23 answers) of the ES were integrated. Table 9 provides a comprehensive overview of the type of external programs with which the ES interface. These findings seem to contradict H₃.

The following is a list of application areas in which ES are being used. The figure between brackets indicates the number of ES that could be identified on basis of the descriptions given.

Functional Business Management (27)

- Personnel/payroll/ind.relations (6)

- Production management or costing (6)

- Customer support/product selection (5)

- Procurement (3)

- Project management, scheduling, optimization (3)

- Strategic planning (3)

- Other (1)

Finance (20)

- Creditworthiness/loan application screening (10)

- Insurance claim assessment (4)

- Stock market/portfolio planning (2)

- Other (4)

Engineering/Manufacturing (41)

- Diagnostics/fault finding (15)

- Continuous process control (12)

- Building/design (8)

- Equipment selection (3)

- Other (3)

Agriculture/Forestry/Ecology (9)

- Advise/planning/assist (8)

- Modelling (1)

Computer Systems (8)

- Network management/control (3)

- Software support/diagnostics (3)

- Database tuning (1)

- Training (1)

Medical (4)

- Diagnostics (3)

- Interpretation (1)

Military (4)

- Situation analysis (2)

- Diagnostics (1)

- Planning (1)

Other (4)

- Geological interpretation (2)

- Transportation (1)

- Curriculum advisor (1)

This is indeed an impressively wide spread of applications, especially when compared to the literature. [13] [14] Hypothesis H₄ must therefore be rejected. Areas that appear to be relatively under-represented in South Africa are chemistry, legal and medical ES. Follow-up surveys to e.g. the appropriate research institutions could verify whether this is a methodological bias due to the sampling frame.

Table 10 lists the ratings given to problems possibly experienced by the respondents. It proved to be quite difficult to provide a methodologically justifiable ranking, so the relative position of the problems must therefore not be seen as definite.

The overall impression is very encouraging: the general level of problems experienced is reasonable low with only a small percentage of respondents experiencing critical or major problems. Another interesting observation is that the more specifically ES technology-related issues are less problematic than the "institutional" or generic IS project problems: only the "technical limitations" and "selection" of ES tools make the top 10 problems. The other top eight problems seem to be problems associated with many other IS projects. This finding thus contradicts hypothesis H₅.

The comparison of the ranking of problems highlighted in the Keating and Philip & Schultz surveys is detailed in table 11. There is little correlation (-0.01) between the ranking of problems listed in Keating and in table 10. This is even more the case when comparing with Philip & Schultz' survey where the "real" correlation is even "worse" than the -0,70 correlation coefficient) calculated according to the table: the five problems listed here came nearly all at the absolute bottom of table 10 (positions 4, 12, 15, 16 and 17 out of a list of 18) whereas they constituted the absolute top 5 out of 12 listed by Philip & Schultz. The possible cause for this is the fact that the US survey seemed to concentrate on technical, ES-specific problems. However, both studies confirmed the low severity of problems experienced by users in absolute terms.

4.7.1 Strategic ES Currently in Place

Users were queried whether they had developed any ES which are of strategic importance to their organization. This question was formulated identically in Keating's survey and the comparison follows in table 12.

Table 12: Occurrence of Strategically Important ES

4.7.2 Future Development of Strategic ES

Respondents were also asked whether they envisaged the future development of strategically important systems in their organization. Table 13 tabulates and compares the results in a slightly different format than table 12, since Keating also asked this question to non-users.

Table 13: Future of Strategically Important ES

4.8 Overall ES Success Rate

Although not directly addressing any of the research hypotheses, perhaps the single most important question in the questionnaire relates to the degree of success as perceived by the respondents. Table 14 tabulates the results for all ES where the question was answered, as well as only the active systems. The latter exclude the 15 ES who are not in use; 38 ES under development and 3 ES with unknown status. The last column provides the comparable data from Keating's survey; although she provided the option "Too soon" instead of "Don't know" which may have affected the responses.

Table 14: Perceived Success of ES

It is clear that the systems are perceived to be highly successful, whichever way it is measured. There are no real statistically significant differences between the various distributions when the methodologically problematic bottom two rows are excluded. It can also be investigated to what extent "success" is dependent on other measured variables (e.g. system size), but this falls outside the scope of this research objectives of this report.

In summary, the survey has documented 128 ES, of which 76 are in actual use, the majority for more than 2 years, many of a relatively large size and representing a fair development investment, spread over a wide area of applications. Although many are admittedly developed on personal computers using shells, a substantial number were developed in traditional languages. Many ES interface on a relatively sophisticated level with external applications. They are generally considered successful. Only about one-fifth is strategically important but a more strategic role of ES is expected in the future. The problems experienced are typical for most IS developments and not specific to the ES technology. Many current non-users indicated anticipated ES development in the near future.

In the light of the strong rejection of the majority of the secondary hypotheses, it is difficult to support Ho. Consequently, and contrary to the findings of Keating [1991], H_o must be rejected on basis of the survey results i.e. both a significant amount and fairly sophisticated level of ES activity appears to be taking place in South Africa.

Results are generally compatible although not always perfectly agreeable with the Philip & Schultz (USA, 1990) survey, but regularly contradict Keating's findings (RSA, Nov.1990) which seem to have been based on too small a sample. The overall validity of the research is considered to be fairly high and the report can thus be said to represent a fair picture of the status of ES technology in South Africa as of 1992.

The following limitations of the research must be recognized, as referred to in 2.2.

The scope of the survey was limited to organizations who have already been identified positively with ES technology. The findings therefore cannot be extrapolated to all South African organizations. In particular, no conclusions can be drawn with respect to the overall penetration rate of ES in the overall corporate world.

Although the survey was intended to be as comprehensive as possible, there must be a considerable number of South African ES and users which have not been included in this survey. Areas which seem particularly under-represented are the retail, medical, legal, natural science and military applications. It is possible that these may have been missed due to the particular sampling frame.

This survey could form the basis of future, similar follow-up questionnaires intended to assess the dynamics of the ES industry and identify historical trends among ES users.

Secondly, some of the problems experienced with particular questions could be addressed by further theoretical and empirical research. The areas which need specific attention are:

- further investigation of the reasons for abandoning of ES projects;

- further exploration of barriers to ES technology among non-users;

- empirically-based exploration and validation of an ES classification scheme;

- a structured framework for benefits and problems experienced with ES.

Thirdly, this survey has proven that the extent of ES usage is sufficient to allow for more focused ES surveys. There seems to be a sufficiently large "critical mass" (population) of ES to ensure success of the following types of targeted ES surveys:

- specific use of ES as a component of DSS or EIS;

- use of ES in the management, finance and/or manufacturing environment.

Fourthly, the 128 ES which have been identified could form the basis of a comprehensive South African ES "case study" collection (book?) for use by practitioners and educators.

Lastly, it is recognized that not all potentially significant findings may have been extracted from the survey database. Further statistical analysis of the data obtained through this survey, especially regarding the correlation of the different variables, might provide further valuable empirical insights.

6 REFERENCES

[1] AMBROSIO J. "Expert Systems Make Their Mark in Corporations." Computerworld, Vol.24 No.32 (6 Aug 1990), p.10.

[2] KONSYNSKI B.R. "Editorial to Special Issue: Decision Support and Knowledge-Based System." Journal of Management Information Systems, Vol.5 No.1 (Summer 1988), pp.3-5.

[3] BUCKLER G. "Expert Systems Making Commercial Inroads: AI Now Viewed as Another Class of Development Tools." Computing Canada, Vol.16 No.15 (19 July 1990), pp.28-29.

[4] ANON. "Expert Systems Shrug Off Slow Start for Good Growth According to Frost & Sullivan Report." Information Today, Vol.7 No.5 (May 1990), pp.43-44.

[5] WATERMAN D.A. A Guide to Expert Systems. Reading, MA: Addison-Wesley, 1986.

[6] BENBASAT I. & NAULT B.R. "An Evaluation of Empirical Research in Managerial Support Systems." Decision Support Systems, Vol.6 No.3 (1990), pp.203-226.

[7] KEATING L.M. The Extent of Expert System Usage. MBA Research Report. Johannesburg: University of the Witwatersrand, 1991.

[8] PHILIP G.C. & SCHULTZ H.K. "What's Happening With Expert Systems? A Survey of Expert System Users." AI Expert, Vol.5 No.11 (Nov.1990), pp.57-62.

[9] STEENEKAMP C.S. 1984. Praktiese riglyne vir vraelyskonstruksie. Geleentheidspublikasie No.16. Pretoria: Raad vir Geesteswetenskaplike Navorsing, 1984.

[10] KIM J. & COURTNEY J.F. "A Survey of Knowledge Acquisition Techniques and Relevance to Managerial Problem Domains." Decision Support Systems, Vol4 No3, pp.269-284.

[11] AGARWAL R. & TANNIRU M.R. "Knowledge Acquisition Using Structured Interviewing: An Empirical Investigation." Journal of Management Information Systems, Vol.7 No.1 (Summer 1990), pp.123-140.

[12] MENTZ M. The Integration of Expert Systems and Decision Support Systems to Solve Business Problems. Technical Report (Hons). University of Cape Town, May 1988.

[13] HARMON P. & KING D. Expert Systems. Artificial Intelligence in Business. New York: J.Wiley & Sons, 1985.

[14] SPRAGUE, R.H. & WATSON H.J. (eds) 1986. Decision Support Systems. Putting Theory into Practice. Englewood Cliffs, NJ: Prentice-Hall, 1986.

Jean-Paul Van Belle emigrated to South Africa after obtaining his Licentiate in Economic Sciences (Rijksuniversiteit Ghent, Belgium) to lecture in the Business Economics department at the University of the Western Cape (UWC) in 1984. He specialized in Financial Management and Management Science whilst building up considerable experience in computer-based education and personal computers. He obtained his M.B.A. at the University of Stellenbosch in 1988. Whilst on secondment to UWC's Unit for Computer-Supported Education to oversee the "downsizing" from mainframe Plato to PC-based Socrates, he obtained a B.Comm (Hons) in Information Systems at the University of Cape Town. He is currently in charge of the Information System courses offered by both the Business Economics and Accounting departments. His main computer interests are "shareware" and AI. He is happily married to Eva and has a young daughter and a son. He is a keen outdoors person and long-distance runner. <Jean-paul.VanBelle@uct.ac.za>