Jean-Paul Van Belle 's  Life

Work & Study Publica- tions Family & Home Travel Running & Sports Other

A survey of the current status of expert systems in South Africa. 

Jean-Paul Van Belle & University of Cape Town

Author: Jean-Paul Van Belle

Submitted as part of the examination requirements for the B.Com (Hons), IS
 at the University of Cape Town, 1992.

Jean-paul.VanBelle@uct.ac.za

CHAPTER 1: INTRODUCTION

1.2.3 Limitations of the Study

1.2.4 Objectives of the Research

- Opinions in respect of the future of ES technology.

- Miscellaneous issues: interface with other systems, strategic ES.

1.2.5 Research Hypotheses

First primary research hypothesis HI:

Second primary research hypothesis HII:

1.4 STRUCTURE OF THE RESEARCH REPORT

CHAPTER 2: LITERATURE SURVEY

2.2 DEFINITION OF EXPERT SYSTEMS

2.3 EXPERT SYSTEM FUNDAMENTALS

2.3.1 Structure of an ES.

2.3.2 Categories of ES

2.3.2.1 Based on Technical Characteristics.

2.3.2.2 Based on Problem Area.

2.3.2.3 Based on Application Area

2.3.2.4 Based on System Complexity

2.3.3 ES Development Life Cycle and Knowledge Acquisition

Manual Methods

. working through example problems (cases);

. discussion of cases in context of an ES prototype;

. the Repertory Grid Approach;

. directed interviews (usually supplementing the above);

. informal interviews (useful in the early stages).

Computer-Aided Methods

2.3.4 Other ES Considerations

2.4 EXPERT SYSTEMS AS DECISION SUPPORT SYSTEMS

2.4.1 Introduction and Definitions

2.4.2 The DSS context

2.4.3 Similarities and Differences Between DSS and ES

2.4.4 Towards an Integration of ES and DSS

2.5 RECENT TRENDS IN EXPERT SYSTEMS

2.5.2 Further Integration with Other Systems

2.5.3 Other Trends

2.6 OVERVIEW OF CURRENT ES APPLICATIONS

2.6.1 Accounting & Financial ES

- Prognosis: AUDITOR (debtors' risk);

- Monitoring: WATCHDOG (investment/portfolio).

2.6.2 Agricultural ES

- Diagnosis: PLANT/DS (soyabean disease).

- Prediction: PLANT/CD (damage from black cutworm).

2.6.3 Chemistry ES

- Planning: SPEX (molecular biology experiments).

- Remedy: TQMSTUNE (tune mass spectrometer)

2.6.4 Electronics ES

- Prognosis: CRITTER & DFT (VLSI performance).

2.6.5 Engineering ES

- Instruction: STEAMER (steam powerplant).

2.6.6 Geological ES

2.6.7 Information Systems ES

- Monitor/Control: YES/MVS (IBM MVS).

- Planning: ISA & IMACS (order scheduling, DEC).

2.6.8 Legal ES

2.6.9 Management ES

2.6.10 Medical ES

- Prognosis: DRUG INTERACTION CRITIC (drug interactions)

2.6.11 Military ES

2.6.12 ES in Other Application Areas

- Monitoring: LES (shuttle liquid oxygen).

- Prognosis: WILLARD (thunderstorms).

2.7 RESULTS FROM PREVIOUS ES SURVEYS

2.7.1 General Comments

2.7.2.1 Behestian-Ardekani [1988]: USA

Usage

Reasons for not using ES

- not cost-effective;

- user resistance;

- no necessity;

- limited technological capacity;

- too early;

- low priority;

- ES solve only simple problems.

2.7.2.2 Price-Waterhouse: UK

Usage

Responsibility for initiation and development

Success ratio

Reasons for not using ES

- lack of corporate awareness (66%);

- finding suitable applications (53%);

- cost justifying applications (37%);

- availability of technical skills (31%);

- integration with existing systems (29%);

- acceptance by users (25%);

- delivering practical systems (24%);

- capturing expert's knowledge (22%);

- maintaining captured knowledge (14%).

 

Expectations about the future

2.7.2.3 Keating [1991]: R.S.A.

Usage

Organizational demographics

Strategic importance

Responsibility for initiation and development

Success

Development platform

Operational status

Issues for successful ES implementation

* Issues with mode = Very significant

- Acceptance by users (89%)

- Capturing expert's knowledge (83%)

- Delivering practical systems (78%)

- Ease of use (67%)

- Availability of technical skills (50%)

- Maintaining captured knowledge (39%)

- Corporate awareness of potential (39%)

* Issues with mode = Significant

- Finding suitable applications (50%)

- Task is of a manageable size (50%)

- System will provide a high return (50%)

* Issues with mode = "Neutral"

- Integration with existing systems (33%)

Potential/Experienced

- Identifying suitable ES building tools 61% / 28%

- Commitment from a human expert 56% / 22%

- Lack of understanding about ES 56% / 11%

- Validating ES knowledge 50% / 22%

- Finding a suitable topic 50% / 22%

- Cost justification 50% / 17%

- Capturing human expertise 50% / 17%

- Maintaining knowledge 50% / 11%

- Interfaces to other systems 44% / 22%

- Long development cycle 33% / 22%

Benefits expected and actually experienced

Expected/Experienced

- Improving productivity 83% / 56%

- Achieving competitive advantage 67% / 39%

- Storing of valuable information 67% / 39%

- Improved consistency 67% / 28%

- Substituting for human expertise 56% / 28%

- Training new experts 56% / 11%

- Personnel savings 28% / 22%

Keating her research as follows:

2.7.2.4 Other Reported Surveys

2.7.2.5 Conclusion

2.7.3 Surveys Directed at Current ES Users

2.7.3.1 Philip & Schultz [1990]: USA

Usage

Organizational demographics

Responsibility for initiation

Reasons for developing ES

- improve decision making/problem solving 67%

- improve productivity 63%

- improve service 53%

- improve product quality 51%

- improve diagnostics 45%

- preserve knowledge 44%

Areas of application

Development platform

Integration with other applications

Size of systems

Success of systems

Problems experienced

- Performance speed 17%

- Interface with external program 16%

- Documentation 13%

- Interface with external files 14%

- Debugging aids 12%

- Ease of use 10%

- Ease of maintenance 10%

General opinions of respondents

2.7.3.2 The JIPDEC survey: Japan.

Usage

Areas of application

- diagnosis 29%

- plant operation and production control 19%

- consulting 14%

- designing 9%

- investment management 6%

- distribution management 6%

2.7.3.3 The Frost & Sullivan Report

USA

Europe

1988 (1994 forecast)

United Kingdom $ 81 million ($ 584 million)

Germany $ 86 million ($ 643 million)

France $ 63 million ($ 468 million)

Italy $ 16 million ($ 218 million)

Total "big four" $246 million ($1913 million)

2.7.3.4 Conclusion

2.7.4 Surveys Targeted at Specific ES Types or Users

2.7.4.1.1 The 1986 Survey [Crofts, 1989]

2.7.4.1.2 The 1988 Survey [O'Leary, 1988]

Level of Activity

Application Areas

Responsibility for Initiating & Developing ES

Development Environment

Obstacles to Development

4. Complexity of (use of) ES tools.

5. Availability of "Off the Shelf" ES.

6. Identifying potential applications.

7. Cost of delivery.

8. Availability of knowledge engineers.

9. Cost of ES development.

Benefits

% Respondents: Using ES/Developing ES

Increased Profits 56% / 15%

Broader Distribution of Scarce

Resources 33% / 14%

Improved Quality / Consistency

of Employee Output 22% / 7%

Improved Training 11% / 14%

Increased Experience with ES 11% / 29%

No Benefits Derived Yet 22% / 57%

Strategic Importance

2.7.4.1.3 The 1990 Survey [Jones, 1990]

Level of Activity

Application Areas

L&H / P&C

Underwriting 40% / 48%

Claims 17% / 19%

Investment NA / 10%

Sales support 18% / 10%

Personal financial planning 7% / NA

Development Environment

Obstacles to Development

1. Lack of technical staff.

2. Availability of funds.

3. Other projects have priority.

4. User acceptance or scepticism.

5. Technical uncertainties.

Benefits

The key ES benefits were identified by respondents:

1. Improved quality and consistency.

2. Improved employee productivity.

3. Broader distribution of scarce expertise.

4. Better definition of professional activities.

5. Improved training.

Strategic Importance

2.7.4.1.4 Comments

2.7.4.2 Doukidis: Use of DSS concepts in ES

About the Role of ES

Design features

                      - 15% of ES is rated as "user-driven", 67% as system-driven, 13% a combination of both and 5% are non-interactive.

                      - Most systems are quite user-friendly with only 4% using a command language, although also only 7% sporting natural language interfaces.

                      - Only very few ES implemented non-procedural commands or fourth generation language query facilities.

                      - Almost all ES undergo an adaptive development process.

Components

              55% of the ES can be said to consist explicitly of the three sub-systems conceptually present in a DSS:

                      - Data sub-system: 60% of ES; the other 40% rely on the user to supply all data..

                      - Models sub-system: all systems, although only 48% formally.

                      - Dialog sub-system: 97% i.e. all the interactive ES.

Prescriptive versus Descriptive

              All ES employ a descriptive (informal) view in the form of heuristics, as opposed to a much smaller proportion that also support a prescriptive (formal) view in the form of normative models (33%) and regulations or work instructions (19%).

Stages of Decision

              Most ES are concerned with providing information or intelligence (72%) and a conclusion or choice (63%) with only 31% providing alternatives. Only 16% support all three stages of decision-making.

Organizational Level of Activities

              Unlike DSS the majority of ES (78%) focus on operational tasks; 24% on management work and only 3% on strategic issues.

Conclusion

              Although this survey cannot be seen as definitive because of its rather limited sampling frame (historically described ES cases), it offers an interesting perspective on ES which is worthy of further empirical investigation.

              Three fundamental DSS issues are explicitly applied in the majority of ES sampled: the semi-structured problem/task domain, its supportive rather than prescriptive role and its emphasis on effectiveness rather than efficiency.

              On the other hand, although "both DSS and ES have similar aims, they achieve them in completely different ways. The main differences are the boundary of the problem-space and the way to tackle problems." [Doukidis, 1990:353]

2.7.4.3 Meyer & Curley: Classifying ES by Complexity

              Meyer and Curley [1991] attempt to present a cohesive framework that would allow the differentiation and comparison of ES projects across different industrial settings or applications from the perspective of the IS manager. The primary focus of the survey is an attempt to describe and empirically validate a classification methodology based on the relative degree of complexity of an ES.

              This framework, described in more detail in 2.3.2.4, was empirically validated against a sample of 50 successfully developed ES. Figure 12 present the scatterplot of the systems according to the knowledge and technology complexity framework. A forthcoming publication of the application of the classification scheme to the German situation by von Dobschutz [to be published in Wirtschaftsinformatik, 1992] is also mentioned.

              To illustrate a practical IS management use for this classification methodology, additional data considered relevant from an IS management point of view, was gathered. Figure 13 gives the summary information for the sample.

              As part of the investigation, the responsibility for initiation and control of development was investigated. This does not seem contingent on the classification and the overall influence of DP/MIS remained important. A full two-thirds of ES were initiated by an existing (62%) or new (4%) business department with only 34% originating from DP/MIS. Identical proportions were obtained for the locus of the organizational control over the project.

2.7.4.4 Miscellaneous Smaller Surveys

              A number of other surveys were encountered in the literature but have a insufficiently large sampling frame to be representative.

2.7.4.4.1 Liang: ESS bridging DSS and ES

                  Although ES has already proven fairly successful in various technical areas, its successful employment in strategic management areas has proved more difficult because of the multi-disciplinary approach that is needed in the decision-making process. Liang [1990] proposes the use of ESS as a pragmatic intermediate solution and analyzed nine ES in the management area to develop his arguments. The analysis is aimed at investigating how the traditional limitations of ES can be overcome while still retaining their general benefits.

2.7.4.4.2 Anderson & Stach: Insurance Industry

                  Anderson and Stach [1990] report an in-depth telephone interview-based survey among 25 senior underwriting, claims and MIS executives that were known to be technically advanced in the use of ES.

                  The results are "impressive" i.e. a very wide range of application areas is described, ambitious future ES development plans are envisaged and the strategic role of ES is underlined, the survey is severally biased by the fact that the survey was conducted by an established vendor amongst its clients and prospects. The Cooper & Lybrand surveys [2.7.1] are more scientifically valid and more respresentative.

2.7.4.4.3 Barbara: New York CPA agencies

                  This unpublished survey, which was reported by O'Leary [1988], drew 28 responses from 148 NY CPA agencies. None of the firms had an ES in use although 53% had heard about ES, 46% were aware of potential uses, 20% was monitoring usage and 18% contemplating use.

                  The data from this survey underscores the need to assess the general status of ES before embarking on a more focused ES survey.

2.7.4.5 Conclusions

              A more substantial body of overseas empirical ES is available than commonly assumed. Unfortunately, not many studies conform to a format that would easily allow comparison over time or geographically.

              In general, though, there seems to be some evidence of a maturing ES technology. This is demonstrated by the use of more standard development platforms, integration with other IS applications and, to a lesser extent, the movement towards more management-oriented problem areas.

              Lastly, the literature illustrates the need for a more comprehensive survey to assess the overall status of ES technology in South Africa. This could then be used as a basis for more focused South African surveys.

CHAPTER 3: METHODOLOGY

3.2 PRIMARY RESEARCH HYPOTHESES

      Based on the South African study by Keating [1991] and a number of exploratory interviews with practitioners, it was felt that ES activity in South Africa is still generally at a low level. This motivates the first major research hypothesis.

First primary research hypothesis HI:

          "Generally, very little ES activity in terms of system development and systems in actual use, is taking place in South Africa."

      HI is stated in absolute terms. Fortunately, is a sufficient amount of empirical research available to allow comparisons to be made. The literature survey (2.7) generally indicated a recent acceleration in ES activity in North America and Europe. The following relative hypothesis can therefore also be formulated.

Second primary research hypothesis HII:

          "The general level of ES activity in South Africa is lower than that overseas"

      Note that this may also reflect the vision believed to be held by some practitioners or managers that "(IS) technology-wise, South Africa is generally a couple of years behind the USA, Japan and Europe", perhaps as a result of the international sanctions or supposed organizational conservatism. Although the author certainly does not share this generalization, it may perhaps hold truth for ES technology in particular.

3.3 SECONDARY RESEARCH HYPOTHESES

      In order to make the empirical testing of the primary research hypotheses possible, a number of secondary hypotheses are formulated. These secondary hypotheses are logical deductions from the primary hypotheses but also specific enough to support or reject them on basis of the survey results. The questions in the survey questionnaire have been formulated to relate directly, but not necessarily exclusively, to a specific secondary hypothesis.

      Each secondary hypothesis can be stated in an absolute and a relative way, similar to the two primary research hypotheses. Below follow the "absolute" versions, although the statistical test hypotheses will usually take a relative format, i.e. comparing it to the results of another survey. Whenever the term "ES" is used in the following, it is meant to pertain specifically to South African systems.

3.3.1 Size of ES

          Low ES activity implicates low project priorities, resulting in turn in a relative small amount of corporate resources being available. Hence it is expected that many systems will be relatively small in size. Although there are many ways in which the size of an ES can be measured, the length of a questionnaire usually limits the number of dimensions that can be measured.

              H1: The majority of ES are small. Development effort is measured in person-months rather than years; rule-based systems rarely exceed 100 inference rules.

          In addition, many systems can be expected to be in the prototype or testing phase.

3.3.2 Development Platform

          Because of the small amount of resources allocated, it may be expected that quick-and-easy, simple software development tools will be used. The hardware can also be expected to be at the bottom of the range.

              H2: Most ES are developed using low-end shells. These shells, as well as the systems developed with them, run on personal computers. Mainframe-based systems and ES custom-coded in traditional programming languages are the exception rather than the rule.

3.3.3 Level of External Interfacing

          The interfacing of ES with other applications is a resource intensive and relatively sophisticated process.

              H3: The majority of ES are stand-alone systems. Systems that do interface, typically do so on a relatively low and safe level, i.e. they read but do not modify data from the corporate databases.

3.3.4 Use of Consultants

          In a relatively small market, there is limited number of reputable consultants with a proven track record. Also, when awareness is low, project ideas come from enthusiastic users or technically-minded IS staff who are likely to propose and champion a simple in-house demonstration prototype. Projects with a high perceived risk (read: new technology) may also be confined in-house to "save face" in case things go wrong.

              H4: Most ES are developed in-house. Few are proposed or developed by consultants.

3.3.5 Spread of Application Areas

          When interest and awareness is low, applications tend to be limited to areas with proven success stories where "innovators" have taken the lead.

              H5: The spread of ES applications will be relatively narrow, limited mainly to research environments or fields with historic success stories (medicine, computer systems, chemistry, electronics, and military).

3.3.6 Nature of Problems

          When new technologies are introduced, the majority of the reported problems tend to be of a technical nature. Aspects such as financial viability, effectiveness, user interface etc. are a typical sign of a more mature technology.

              H6: Most problems that are experienced are developmental and relate to the technological aspects of the systems.

3.3.7 Opinions about the Future

          Although unrealistic or extreme individual expectations may be expected in the early stages of technology implementation, consensus opinion will tend to display neither a strong positive nor negative inclination.

              H7: A wide spread of opinions about the future of ES must be expected, with many people voicing radically opposed views. However, the consensus views on the main prospects of the technology will tend to be indecisive.

3.3.8 Strategic ES

          Organizations will be loath to develop strategic systems which depend on a relatively new, unproven technology.

              H8: Very few strategic ES are in place. Not many organizations have developed a clear role for future ES in their strategic plans.

3.3.9 Barriers to ES

          Reasons for not adopting ES technology are mainly a consequence of low organizational awareness, lack of staff skills and the untested nature of the technology.

              H9: Barriers cited by non-users of ES technology relate to technological and human issues, many of which could be addressed by education.

3.4 POPULATION AND SAMPLE DEFINITION

      Based on the rough categorization developed in 1.2.2, the Keating [1991] survey can be termed a "penetration" survey with all RSA DP installations as a target. Before a survey can be "focused" on a specific aspect of ES, it is necessary to conduct a survey to determine the actual state of ES technology in South Africa.

      Hence the population identified for this survey will be "all organizations who have been positively identified with ES technology."

      To determine an appropriate and manageable sampling frame, it is assumed that members of these organizations can be expected to participate in conferences and user groups. Hence the following mailing lists were obtained through the kind cooperation of the various local Special Interest Groups for Artificial Intelligence (SIG-ART) of the Computer Society of South Africa (CSSA):

- SigArt Cape;

- SigArt Transvaal;

- SigArt Natal.

      In addition, a number of individuals kindly assisted in providing the following two valuable address lists:

- Attenders of the EXPERT 90 conference;

- Speakers for the EXPERT 92 conference.

      A number of important remarks are in order.

          - A certain overlap between the conference and SigArt list was identified and duplicate addresses were removed manually.

          - The author has been warned that the SigArt lists and the Expert 90 conference list were relatively dated and a considerable number of non-deliverable questionnaires could be expected. Although a return address was provided, the full effect on the response rate cannot be measured exactly.

          - In at least two known cases, questionnaires were copied by the addressees and passed on to another ES user. It is not known in how many other instances this may have occurred; although this practice enhances the representativeness of the results, it creates an upward bias in the response rate.

3.5 EXPLORATORY INTERVIEWS WITH PRACTITIONERS

      After a few informal talks with academics (UCT), users (Old Mutual, NBS) and a ES shell vendor (Level 5), two interviews were arranged with respected and experienced consultants who are active in the ES market: Mr. Rob Simpson (branch manager, Productivity Software, Cape Town; now with UC-2 Technologies) and Mr. Kenneth Carden (management consultant, Performance and Systems Improvement, Cape Town).

      The following areas were discussed in the interview: the general status of the South African market, the major dynamic forces, expectations with respect to the immediate and long term future, aspects of ES that needed specific attention and investigation, appropriate sampling frames, determinants of response rate.

      These interviews provided the author with a much appreciated insight into the dynamics of the South African ES market and the valued time that these individuals made available is hereby gratefully acknowledged.

3.6 QUESTIONNAIRE DESIGN

      The questionnaire has been designed around the secondary research hypothesis but in such a way as to allow maximum compatibility with the Keating [1991] and Philip & Schultz [1990] surveys, whilst remaining as short as practically possible. The practical guidelines for questionnaire design suggested by Steenekamp [1984] proved invaluable.

The overall structure of the questionnaire is as follows:

Section 1: To be answered by all respondents.

- Personal background 1

- Organization information

- Exposure to expert systems

Section 2: To be answered by ES users (have developed or implemented ES)

- Description 2 3

- Nature of systems 3

- System usage 3

- System development 3

- Benefits and problems 4

- Opinion of the future of ES 5

- Strategic ES

Section 3: To be answered by non-users

- Future plans re ES

- Barriers to ES use 6

Section 4: To be answered by all

- Comments 2

Notes:

          (1) Name and address information optional.

(2) Open-ended.

(3) Information to be provided for up to three ES.

          (4) Measured on a 5-point scale with additional "NA" option.

(5) Measured on a 5-point Likert scale.

(6) Measured with a list of statements.

      All other questions were closed-end questions with multiple choice selection. The questionnaire consists of a total of 33 questions.

      The following remarks should also be considered:

Question 10: The Primary Activity of the ES.

Based on the typology in 2.3.2.2

Questions 11 & 18: Measuring size of ES.

      There is no universally accepted "standard" to measure ES size. Buchanon [1989] lists at least 7 measures, but including all of them would have been impractical. Although the "number of rules" exhibits a strong rule-based ES bias, it allows comparison to other surveys.

      Question 22: The Knowledge Acquisition Process.

Based on Turban [1990] pp.453-482.

Question 28: Opinion on Statements about the Future of ES.

      These statements are a selection from Keating [1991] and Philip & Schultz [1990], with a few omissions, changes and additions.

Questions 32 & 33: Barriers to ES usage.

      It seemed important to investigate why organizations who are "au fait" with ES technology have NOT implemented it (yet). However, there were no previous surveys to base these questions on and a number of important omissions may have been made.

      The full 8-page survey with covering letter can be found in Addendum 8.1. It was sent to the addressees in a reduced format as a 4 page A5 booklet with postage paid reply envelope.

      3.7 DATA ANALYSIS TECHNIQUES

The following analysis techniques will be employed in this research:

- descriptive listing: for open-ended questions;

          - tables: for visual display and enumeration of frequencies - single bordered tables indicate a simple listing of survey results, double border tables are used for comparisons;

- summary statistics: mode and median values for categorical data;

CHAPTER 5: INTERPRETATION OF THE RESULTS

5.1 VALIDITY OF THE RESULTS

5.1.1 Validity Issues

          The fundamental value of the findings which have been reported in chapter 4 dependS on how representative the responses are of the true state of ES usage in South Africa, i.e. what is the validity of this survey?

          There are basically three issues to be considered in assessing the validity of this report.

              1) How well does the survey instrument (i.e. the questionnaire) measure the variables which are needed to assess the research hypotheses?

              2) How representative are the respondents of the sample?

              3) How appropriate is the sampling frame for the true population?

          Question 1 relates to the internal validity and questions 2 and 3 are concerned with the external validity of the survey.

5.1.2 Internal Validity of the Questionnaire

          The following specific measures were intended to increase the internal validity of the questionnaire.

Literature survey.

          The extensive literature research covered all aspects of the questionnaire. Chapter 2 only reports on selected areas of interest, but other areas such as benefits, problems, management and general IS were also researched in preparation of the questionnaire design. The benefits and occasional weakness of this approach is illustrated by question 22 concerning the knowledge acquisition methods which proved comprehensive enough to satisfy most users except for the omission of the option where the subject expert doubles as the ES developer.

Use of previously validated instruments

          The relative paucity of ES surveys imply that no ES survey questionnaires have been tested and validated in a comprehensive and accepted way. On the other hand, there seems to be a consensus about the different areas that need to be assessed in evaluating the status of ES. The instrument has benefited from the author's opportunity to assess the weaknesses of previous instruments and has tried to improve upon them. Examples of this are the omission of "extreme statements" from the opinion statements in question 28 and the "expansion" of potential benefits and problems in question 26 and 27.

          Design of specific questions

          Comments about the specific validity of certain questions have been made in 3.6. In addition, many of the other questions have drawn upon the results of previous surveys. Specific examples the questions on development platforms and the strategic use of ES. The questions in section 3 (barriers to ES from non-users) did not benefit from such a process and can thus be assumed to have a lower validity.

          Open-ended questions

          Including an open-ended "Other, please specify" option in what would otherwise have been a pure multiple option type question is a double-edged sword. Although it makes statistical processing of the data more cumbersome, it also gives the researcher a good feel for the completeness of the options provided in the question. The relative low use that has been made of this option [see 8.4] indicates again that most questions were felt to be fairly complete by the respondents.

          Internal consistency

          Where possible, references were made to internally related results when discussing specific findings. In general, the conclusions seemed consistent with each other. The following can serve as a typical example: 21% of users indicate that they have strategic ES in place [4.3.7.1] and a corresponding 20% feels (in another type of question) that discontinuing current ES would create severe difficulties for the organization [4.3.9]. Another example is the investigation of the correlation between question 11 (number of rules) and question 18 (development time) which proved statistically significant.

          External consistency

          Comparing results of specific questions with results of other surveys can also serve to validate parts of the questionnaire. A good example of this is the almost unreal correspondence of the ranking of opinions about the future between this and the Philip & Schultz survey [8.3.18 and 4.3.9].

          Definitions

          Providing the respondent with a comprehensive list of definitions can also improve the validity of the questionnaire. Apart from the definition of expert systems in the covering letter, no other terminology was defined. Although this made the questionnaire shorter, quicker to complete and thus more attractive, it may be seen as having lowered the validity of at least one questions namely question 10 about the primary activity of the ES.

          Overall conclusion

          Most of the questions can be considered fairly to very valid, excepting perhaps questions 10 and 15. In addition, the validity of questions 32 and 33 cannot be assessed at this moment. Overall, the questionnaire seems a valid and comprehensive instrument. This was underscored by the many extremely positive comments received from respondents about the design and contents of the questionnaire.

          5.1.3 External Validity of the Survey Results

          The following remarks should demonstrate that the survey results are valid and representative for the population which was targeted.

          High response rate

          The extremely high response rate of 37% strongly increases the validity of the study in as such that it indicates in this case how closely the sampling frame matches the population. This is further discussed in 4.1 and 4.2. However, it must still be borne in mind that the response rate of interested people is substantially higher than that for the non-interested; one study quoted the figures 75% versus 13% respectively. This means that the proportion of users versus non-users cannot safely be extrapolated to the entire sample or population (as Keating seems to suggest implicitly).

Keating's penetration ratio

          Bearing the above remarks in mind, the following interesting calculation can be made. It must firstly be noted that Keating's survey was conducted less than 18 months ago and 72% of the "active" ES in this study have been in use for more than one year [table 12]. Keating sampled one in every five South African "DP installations" which resulted in a list of 18 users and 26 ES. Since the DP installation list concerned has virtually no duplications and her sampling can be assumed to be random, it could be expected that the results of a saturation survey to all South African DP installations would have been five-fold, i.e. 90 users and 130 ES. This compares extremely well with the 80 users and 128 ES obtained in this survey,

          General external validity measured against Keating

          Generally, the results of this survey do not agree with the conclusions of Keating as is evident from the many statistical comparisons which have been made in chapter 4. It is the opinion of the author that this is mainly due to the small number of users upon which Keating's findings are based. However, the validity of the general sample population seems to be vindicated by the almost identical geographical spread of "all" responses received by both surveys [table 5].

General external validity measured against Philip & Schultz

          Overall, the results of this survey correspond closely with Philip & Schultz who used a much larger sampling frame but, because of the lower response rate, ended up with a comparable number of users. Unfortunately Philip & Schultz did not collect any data on the level of the individual ES, so results are not always directly comparable. However, results generally correspond closely enough to increase the confidence in the validity of both surveys.

General external validity measure against other surveys

          Not much attention has been paid to comparisons with other surveys because of various methodological problems. It is interesting to note, however, that in the four main areas where statistically significant differences with the Keating or Philip & Schultz survey were encountered, the survey results seem to be supported by the Coopers & Lybrand 1990 survey! This pertains to the favouring of shells as opposed to AI languages; ES problems' similarity to mainstream IS; ranking of ES benefits and the "paradoxal" drop in strategically important ES.

Overall conclusion

          The very positive response rate and the comparisons with Philip & Schultz indicate strongly that this survey has a high external validity and can be considered very representative for the status of the South African ES technology. However, the limitations of the study as indicated in 1.2.3 must be kept in mind, i.e. the findings of this survey are not meant to apply to all South African organizations and it is conceivable that a number of "pockets of ES excellence" have been missed due to the selection of this particular sampling frame.

5.2 DISCUSSION OF THE SECONDARY RESEARCH HYPOTHESES

          5.2.1 Size of ES

              H1: The majority of ES are small. Development effort is measured in person-months rather than years; rule-based systems rarely exceed 100 inference rules.

          Although a significant proportion of ES is small, 48% has more than 200 rules and 20% more than 500 rules. This compares almost exactly with the USA status [4.3.3.3]. Most systems [56%] are operational and used relatively frequently [4.3.4.2]. They have been doing so for quite some time: more than 2 years in 43% of the cases [4.3.4.1]. One quarter of ES took more than one "person-year" to develop [4.3.5.3].

          The above hypothesis is not supported at all by the results of this survey and must therefore be rejected.

5.2.2 Development Platform

              H2: Most ES are developed using low-end shells. These shells, as well as the systems developed with them, run on personal computers. Mainframe-based systems and ES custom-coded in traditional programming languages are the exception rather than the rule.

          Most ES are developed on powerful stand-alone computers, although 20% of the systems runs on mini or mainframe [4.3.5.4]. Heavy use is made of ES shells (88 ES), the top-end shells seem to be doing well (DmX, Leonardo, Level 5, Synapse) with the low-end VP-Expert capturing less than 10%. Although little use is made of AI languages, conventional languages were used for 34 ES [4.3.5.5]. A number of systems use more advanced techniques e.g. neural technology [8.2].

          These findings do not support nor strongly reject the secondary hypothesis. However, there is a strong indication that the use of shells and powerful PCs is predominantly a sign of the maturity of the market.

5.2.3 Level of External Interfacing

              H3: The majority of ES are stand-alone systems. Systems that do interface, typically do so on a relatively low and safe level, i.e. they read but do not modify data from the corporate databases.

          Only 43% of ES are stand-alone. Three quarters of the interfaced systems have "write" access to data files. Although levels of interfacing are not as high as in the USA, about one quarter of ES interfaces with in-house developed applications and a similar number with DBMS [4.3.5.8].

          The secondary hypothesis is contradicted by the survey results and must be rejected.

5.2.4 Use of Consultants

              H4: Most ES are developed in-house. Few are proposed or developed by consultants.

          Although end-users are responsible for initiating 30% of ES, consultants account for 18% of project proposals [4.3.5.1]. 41% of ES are wholly developed by outsiders with a significant proportion of another 26% being co-developed by consultants [4.3.5.2].

          The secondary hypothesis is not supported by the survey results and must be rejected.

5.2.5 Spread of Application Areas

              H5: The spread of ES applications will be relatively narrow, limited mainly to research environments or fields with historic success stories (medicine, computer systems, chemistry, electronics, and military).

          ES are used in a very wide range of areas, with the majority of expert systems are used in functional business management (27 ES), finance (20) and engineering & manufacturing (41). The areas mentioned in the hypothesis are represented by only a few ES [4.3.3.2 and 8.2].

          The hypothesis is completely contradicted by the empirical evidence and must be rejected.

5.2.6 Nature of Problems

              H6: Most problems that are experienced are developmental and relate to the technological aspects of the systems.

          Only two of the top ten problems identified by users relate to the ES technology specifically. All the others appear common to any general IS project [4.3.8.2]. This is in fact completely contradictory to the USA results. In addition, the general level ("severity") of the problems was substantially lower than that of the benefits which were experienced.

          The secondary hypothesis must be rejected on basis of the survey evidence.

          5.2.7 Opinions about the Future

              H7: A wide spread of opinions about the future of ES must be expected, with many people voicing radically opposed views. However, the consensus views on the main prospects of the technology will tend to be indecisive.

          The opinions about the future of ES show a reasonable consensus and strongly support the trends identified in the ES literature: the need to interface with DBMS, using ES in DSS, integration with traditional systems, usefulness for strategic IS. There is almost perfect correlation with the opinions expressed in the USA. Quite a few statements attracted a neutral consensus view [4.3.8.3]

          The hypothesis is contradicted by the results, especially with respect to the wide spread. Where the consensus view is neutral, this is a sign of maturity and stabilization of the ES technology as well as a realism with respect to expectations of the potential of ES.

5.2.8 Strategic ES

              H8: Very few strategic ES are in place. Not many organizations have developed a clear role for future ES in their strategic plans.

          21% of the users have strategically important ES in place, substantially less than the ratio reported by Keating [4.3.7.1]. 49% envisage strategic ES in the future against 23% who do not - many (28%) don't know [4.3.7.2]. Only 20% feel (8% strongly) that discontinuing the current ES would create severe difficulties for the organization although 60% feels that ES are an excellent tool for developing future strategic ES [4.3.9].

          Although the first part of the hypothesis cannot be rejected completely on the basis of the available data, the part about the future role of ES cannot be supported.

5.2.9 Barriers to ES

              H9: Barriers cited by non-users of ES technology relate to technological and human issues, many of which could be addressed by education.

          The barriers identified by non-users are not based on ES technology or any other external issues. They are solely an internal problem of lack of manpower and the higher priorities of other information systems [4.3.10.3]. Non-users in general also seem fairly knowledgeable about ES technology [4.3.2.1].

          The technological part of the hypothesis must be rejected. However, the main barrier against further ES use indeed seems to be a human problem.

5.3 DISCUSSION OF THE PRIMARY RESEARCH HYPOTHESES

5.3.1 First Primary Research Hypothesis

              HI: "Generally, very little ES activity in terms of system development and systems in actual use, is taking place in South Africa."

          In the light of the strong rejection of almost all secondary hypotheses, it is difficult to support HI. In summary, the survey has documented 128 ES (addendum 8.2), of which 76 are in actual use, most for more than 2 years, many of a relatively large size and representing a fair development investment, spread over a wide area of applications. Although many are admittedly developed on personal computers using shells, a substantial number were developed in traditional languages. Many interface on a relatively sophisticated level with external applications. They are generally considered successful. Only about one-fifth is strategically important but a more strategic role of IS is expected in the future. The problems experienced are typical for most IS developments and not ES-specific. Many current non-users indicate ES development in the future.

          Consequently, and contrary to the findings of Keating [1991], HI must be rejected on basis of the survey results.

5.3.2 Second Primary Research Hypothesis

              HII: "The general level of ES activity in South Africa is lower than that overseas"

          This survey was mailed to a sample of less than half the size of Philip & Schultz, yet it yielded almost the same number of organizations actively involved in ES. The spread of the applications areas in which ES are being used rivals the listing in 2.6 which is based on comprehensive U.S.A. data. Although we have a slightly higher proportion of small systems than the U.S.A., there is an identical proportion of large systems measured both in terms of number of rules and development time. Equivalent hardware platforms are being used as in the USA but development in artificial intelligence languages is virtually ignored in favour of shells and conventional languages. A relatively smaller proportion of RSA ES interfaces with external programs when compared to the USA. The general expectations of South African users with respect to the future of ES matches both the view expressed by USA practitioners and the (mainly overseas') ES literature.

          In many areas, South Africa is "on par" with the USA 1990 status although it is slightly lacking in a few isolated areas. If the Frost & Sullivan report [2.7.3.3] is valid, then these conclusions should hold equally well for Europe. In terms of the JIPDEC survey, we seem to be more advanced than Japan: i.e. we have at least one-quarter of the number of systems for an economy which is proportionally much smaller.

          HII can therefore not be supported on basis of the available empirical research, although the existence of a time lag of one or two years cannot be excluded.

CHAPTER 6: CONCLUSION

6.1 SUMMARY OF IMPORTANT FINDINGS

      There is a significant amount of ES activity taking place in South Africa: out of 420 questionnaires, 156 responses were received with 87 respondents identifying themselves as developers or users of ES and 80 of them provided a list of 128 expert systems [8.2] with at least another 26 not being documented. The overall response is better than any other ES survey encountered in the literature. There is no significant secrecy surrounding ES development as illustrated by the fact that more than 90% provided contact details.

      Activity is taking place in all sectors of the economy with the potential exception of retail/wholesale sector. Although more than a third of the users and developers have more than 2000 employees, another third have less than 100. As could be expected, 60% of users are concentrated in the PWV area; with the rest more or less equally distributed between the Cape Province and Natal.

      Most systems are addressing interpretation, diagnosis and prediction. The popular application areas are engineering/manufacturing (41 ES), finance (20) and other functional business management areas (27) with the classic "technical" areas such as medicine and geology trailing far behind. This spread of applications is as wide as anywhere else in the world.

      43% of the 72 active ES have been in use for more than 2 years, against 28% which have been operational for less than a year. 38 ES are under development and 15 "not in use". Of the active ES, 30% are known to be used once or more daily but the usage frequency for another 30% is unknown.

      Of the rule-based ES, one quarter are tiny systems (less than 50 rules), another quarter is fairly small (50 to 200 rules) and, just like in the USA, half has more than 200 rules. Almost half of ES took up to half a year to develop, a quarter between 6 and 12 months and more than a quarter longer than a year. This compares well with the USA, although we have a slightly larger proportion of small ES. A very strong correlation is found between ES size measured by the number of rules and development time.

      The majority (30%) of ES is initiated by end-users only, the balance more or less evenly spread between upper management, IS/DP department, consultants and others. ES are mainly developed by consultants (31%) and IS/DP (26%) with end-users, sub-contractors, outsiders and various combinations of the above accounting for about 10% each.

      High-end IBM-compatible PCs are the favourite hardware development platform although there are 17 mainframe ES around. A very wide variety of commercial ES shells is the favourite development tool (89 ES) although many (34) are also developed in conventional programming languages. In contrast to the USA, few South Africans make use of the so-called artificial intelligence languages such as Lisp or Prolog.

      Although a prototype is used for more than two-thirds of the ES, this is still a lower proportion than in the USA. A combination of different knowledge acquisition methods are used but the individual (70%) and multiple (29%) expert interview as well as analysis of written documents (34%) rate as the most popular ones. A surprisingly low number of ES use computer-automated rule induction.

      43% of ES have write-access to external data files, and a further 14% read-only access. 26% of ES interface with in-house developed applications written in conventional languages while 21% interfaces with a DBMS; these figures are lower than those identified in a USA survey.

      More than 70% of ES in use are rated successful or very successful; only 6% are considered unsatisfactory or failure. There is some suggestion but no statistical evidence that ES size or investment correlates with perceived success.

      21% of ES currently in use are of strategic importance, although 49% of users envisages development of strategically important ES in the future.

      Respondents rated the benefits experienced from ES generally very high with improved decision-making, improved reliability & consistency, storing and distributing expert knowledge and improving productivity receiving the highest ratings. Very few critical or major problems are experienced, and the top problems are generic to any IS development: commitment from human expert, cost justification, maintenance, long development cycle and user resistance. The two specific ES technology-related problems that make the top 10 are the technical limitations and proper selection of ES tools, vendors should take note! Unfortunately, the ranking of top benefits and problems experienced by users does not compare well with other surveys.

      The expectations of respondents with respect to the future of ES reflect the trends identified by USA users and those mentioned in the academic ES literature very closely: integration with DBMS, DSS, traditional and strategic IS. Few users harbour unrealistic expectations although most expect ES use to increase and remain a cost-effective technology.

      27% of non-users expect ES to be developed within 3 years, while 11% does not expect any ES development, the majority is not sure: "possibly": 48%; "don't know": 5%; no answer: 8%. Further probing reveals that most non-users, who are generally fairly ES literate, are aware of potential applications areas. The need to first put other systems in place and the critical lack of skilled (wo)manpower are the main barriers to pursuit of these areas. Non-users have no intrinsic problems with the ES technology itself.

      Results are generally compatible although not always agreeable with the Philip & Schultz survey, but regularly contradict Keating's findings (RSA, Nov.1990) which seem to have been based on too small a sample. The overall validity of the research is considered to be fairly high and the report can thus be said to represent a fair picture of the status of ES technology in South Africa as of early 1992.

6.2 OVERALL ACHIEVEMENT OF THE RESEARCH OBJECTIVES

      The research objectives have all been achieved to a more than acceptable degree and the study can be considered a success. This was made possible by the excellent cooperation of a large number of individuals and organizations which resulted in an unexpectedly high response rate.

      In particular, the level of ES seems to be on a much higher level than was assumed by the author and many others. This led to the convincing refutation of the conservatively stated research hypotheses. A number of hypotheses were tested statistically and proved significant at high levels of confidence.

      However, the success of the study must be seen in the context of its scope and the limitations mentioned below should be kept in mind. This report could form the basis of a number of interesting future research projects.

6.3 LIMITATIONS OF THE RESEARCH

      The following limitations of the research must be recognized.

      The scope of the survey was limited to organizations who have already been identified positively with ES technology. The findings therefore cannot be extrapolated to all South African organizations. In particular, no conclusions can be drawn with respect to the overall penetration rate of ES in the overall corporate world. In fact, even amongst the responses, marked differences between certain perceptions of users and non-users were already observed.

      Although the survey was intended to be as comprehensive as possible, there must be a considerable number of South African ES and users which have not been included in this survey. Areas which seem particularly under-represented are the retail, medical, legal, natural science and military applications. It is possible that these may have been missed due to the particular sampling frame.

      Another limitation can be traced through the job descriptions given by the respondents (8.4.1). Although the author assumed that the sampling frame would ensure a large response from end-users and functional managers, the overwhelming majority of respondents are in fact directly linked to the IS/DP department or represent the ES industry. This may have introduced a certain bias in the findings of this research.

      Finally, although the survey measurement instrument (the questionnaire) can be considered to possess a fairly high validity, some grey areas remain with respect to a suitable ES classification scheme, reasons for non-usage of completed ES and reasons for non-usage of ES technology. In addition, the lack of statistically significant correlation of benefits and problems experienced by this survey's respondents with the findings of other surveys requires special attention in future empirical research.

6.4 SUGGESTED AREAS FOR FUTURE RESEARCH

      This research should open a number of interesting avenues for future research projects.

      Firstly, it could form the basis of future, similar follow-up questionnaires intended to assess the dynamics and identify historical trends among ES users. This option was specifically considered in the design of the questionnaire through the incorporation of the question "May we contact you in the future for a follow-up survey", to which 90% of users and 81% of non-users responded positive.

      Secondly, some of the problems experienced with particular questions could be addressed by further theoretical and empirical research. The areas which need specific attention are:

- further investigation of the reasons for abandoning of ES projects;

- further exploration of barriers to ES technology among non-users;

- empirically-based exploration and validation of an ES classification scheme;

- structure framework for benefits and problems experienced with ES.

      Thirdly, this survey has proven that the extent of ES usage is sufficient to enable more focused ES surveys to take place. There seems to be a sufficiently large "critical mass" of ES to ensure success of the following types of targeted ES surveys:

- specific use of ES as a component of DSS or EIS;

- use of ES in the management, finance and/or manufacturing environment;

- integrating ES in the traditional SDLC or CASE process;

- ES as a component of strategic IS planning;

- hybrid ES; etc.

      Fourthly, the list of 128 ES which have been identified could form the basis of a comprehensive South African ES "case study" collection (book?) or use by practitioners and educational purposes alike.

      Lastly, it is recognized that not all potentially significant findings may have been extracted from the survey database. Further statistical analysis of the date obtained through this survey, especially regarding the correlation of the different variables, might provide further valuable empirical insights.

CHAPTER 7: REFERENCES

AGARWAL R. & TANNIRU M.R. 1990. Knowledge Acquisition Using Structured Interviewing: An Empirical Investigation. Journal of Management Information Systems, Vol.7 No.1 (Summer 1990), pp.123-140.

AMBROSIO J. 1990. Expert Systems Make Their Mark in Corporations. Computerworld, Vol.24 No.32 (6 Aug 1990), p.10.

ANDERSON C. & STACH R. 1990. The Bright Future of Expert Systems. (Technology for the Times) Best's Review - Property-Casualty Insurance Edition, Vol.91 No.1 (May 1990), pp.76-82.

ANON. 1985. What's Happening With Expert Systems? EDP Analyzer, Vol.23 No.12 (Dec.1985), pp.1-14.

ANON. 1987. A Three-Year Strategy for Expert Systems. EDP Analyzer, Vol.25 No.3 (Mar.1987), pp.1-14.

ANON. 1988. Silicon Expertise. Knowledge-based Systems. Computer Mail, 29 Jan. 1988, pp.39-46.

ANON. 1989. Game for Experts: A computerised expert system developed in SA has beaten 10 teams of businessmen at their own game. Computer Mail, 29 Sept. 1989, pp.48-49.

ANON. 1990a. Figuring Payback. Financial Advantages of Using Expert Systems Software. Government Computer News, Vol.9 No.21 (1 Oct.1990), p.66.

ANON. 1990b. Widening Expert Systems Applications in Japan Offer Market Opportunities for U.S. Vendors. Business America, Vol.111 No.4 (26 Feb.1990), p.14.

ANON. 1990c. Expert Systems Shrug Off Slow Start for Good Growth According to Frost & Sullivan Report. Information Today, Vol.7 No.5 (May 1990), pp.43-44.

BEAU S. 1987. Thinking about artificial intelligence. Harvard Business Review, Jul-Aug 1987, pp.91-97.

BEHESHTIAN-ARDEKANI M. & SALCHENBERGER L.M. 1988. An Empirical Study of the Use of Business Expert Systems. Information & Management. Vol.15 (1988), pp.183-190 as reported in Keating 1991.

BENBASAT I. & NAULT B.R. An Evaluation of Empirical Research in Managerial Support Systems. Decision Support Systems, Vol.6 No.3 (1990), pp.203-226.

BENCHIMOL G., LÉVINE P. & POMEROL J-C. 1987. Developing Expert Systems for Business. London: North Oxford Academic, 1987.

BEULENS A.J.M. & VAN NUNEN J.A.E.E. The Use of Expert System Technology in DSS. Decision Support Systems, Vol.4 No.4 (1988),pp.421-431.

BLACKMAN M.J. 1990. CASE for Expert Systems. AI Expert, Vol.5 No.2 (Feb.1990), pp.26-35.

BLANNING R.W. & KING D.R. 1989. Editorial to Special Issue: Decision Support and Knowledge-Based System. Journal of Management Information Systems, Vol.6 No.3 (Winter 1989-90), pp.4-6.

BRODIE M.L. & MYLOPOULOS J. (ed) 1986. On Knowledge Base Management Systems. Integrating Artificial Intelligence and Database Technologies. New York: Springer-Verlag, 1986.

BROUSELL D.R. 1990. Integration with Existing Systems Is Key: AICorp.'s Harris on Artificial Intelligence. Datamation, Vol.36 No.6 (15 Mar.1990), pp.40-49.

BUCHANON B.G. & SMITH R.G. 1989. Fundamentals of Expert Systems in BARR A., COHEN P.R. & FEIGENBAUM E.A. The Handbook of Artificial Intelligence. Volume IV. Reading, MA: Addison-Wesley, 1989, pp.149-192.

BUCKLER G. 1990a. Expert Systems Making Commercial Inroads: AI Now Viewed as Another Class of Development Tools. Computing Canada, Vol.16 No.15 (19 July 1990), pp.28-29.

BUCKLER G. 1990b. Expert Systems Tackle Corporate Decision-making: the Role of AI Is to Reduce Uncertainty. Computing Canada, Vol.16 No.9 (26 Apr.1990), p.21.

CASHMORE S. 1989. Hot Technologies for the Nineties. Computer Mail, 24 Nov. 1989, pp.15-23.

CHAPNICK P. 1990. The Art of Manufacturing. AI Expert, Vol.5 No.4 (Apr.1990), pp.5-6.

CROFTS A.E., CIESIELSKI V.B., MOLESWORTH M. et al 1989. Bridging the Gap Between Prototype and Commercial Expert Systems - A Case Study in QUINLAN J.R. Applications of Expert Systems. Volume 2. Sydney: Addison-Wesley, 1989, pp.93-105.

DENG P-S., HOLSAPPLE C.W. & WHINSTON A.B. 1990. A Skill Refinement Learning Model for Rule-based Expert Systems. IEEE Expert, Vol.5 No.2 (Apr 1990), p.15(14).

DOUKIDIS G.I. 1988. Decision Support System Concepts in Expert Systems: An Empirical Study. Decision Support Systems, Vol.4 No.3 (1988),pp.345-354.

DREYFUS H. & DREYFUS S. 1986. Why Expert Systems Do Not Exhibit Expertise. IEEE Expert, Summer 1986, pp.86-90.

DYSON E. 1990. Machine Learning. PC-Computing, Vol.3 No.4 (Apr.1990), p.17.

ELIOT L.B. 1990. Loose LIPS Sink Ships. Expert System Developers Need to Employ Security Techniques. AI Expert, Vol.5 No.11 (Nov.1990), pp.11-15.

FINLAY P.N. 1990. Decision Support Systems and Expert Systems: a Comparison of Their Components and Design Methodologies. Computers & Operations Research, Vol.17 No.6 (Nov.1990), pp.535-543.

FRIEDLAND J. 1990. The Expert Systems Revolution: a new generation of "smart" computers is transforming the financial services industry. in TRIPPI R.R. & TURBAN E. Investment Management: Decision Support and Expert Systems. Boston, MA: Boyd & Fraser, 1990, pp.6-18.

GIARRATANO J. & RILEY G. 1989. Expert Systems. Principles and Programming. Boston, MA: PWS-Kent, 1989.

HAMMOND K.R. 1987. Toward a Unified Approach to the Study of Expert Judgement in MUMPOWER J.L., PHILLIPS L.D., RENN O. & UPPULURI V.R.R. 1987.(ed.) Expert Judgment and Expert Systems. Berlin: Springer-Verlag, 1987, pp.1-16.

HARMON P. & KING D. 1985. Expert Systems. Artificial Intelligence in Business. New York: J.Wiley & Sons, 1985.

HAYES-ROTH F., WATERMAN D.A. & LENAT D.B. (ed.) 1983. Building Expert Systems. Reading: Addison-Wesley, 1983.

HIGA K. & LIU SHENG O.R. 1989. An Object-Oriented Methodology For Database/Knowledgebase Coupling: An Implementation of the Structured Entity Model in Nexpert System. Database, Spring 1989, pp.25-29.

HILLMAN D. 1990. Integrating Neural Nets and Expert Systems. AI Expert, Vol.5 No.6 (June 1990), pp.54-60.

HOLTZMAN S. 1989. Intelligent Decision Systems. Reading MA: Addison-Wesley, 1989.

HU D. 1987. Programmer's Reference Guide to Expert Systems. Indianapolis, IN: Howard W. Sams, 1987.

JACOBSON M.A. 1989. The Use of Commercial Expert SYstems in the Business Environment. Technical Report (Hons). Cape Town: University of Cape Town (Accounting Department), October 1989.

JONES D.C. 1990. Expert System Activity Level Up. National Underwriter Property & Casualty-Risk & Benefits, 1990 No.44 (29 Oct), p.11.

KEATING L.M. 1991. The Extent of Expert System Usage. MBA Research Report. Johannesburg: University of the Witwatersrand, 1991.

KERSCHBERG L. (ed) 1989. Expert Database Systems. Proceedings From the Second International Conference. Redwood City, CA: Benjamin/Cummings, 1989.

KIM J. & COURTNEY J.F. A Survey of Knowledge Acquisition Techniques and Their Relevance to Managerial Problem Domains. Decision Support Systems, Vol.4 No.3 (1988),pp.269-284.

KONSYNSKI B.R. 1988. Editorial to Special Issue: Decision Support and Knowledge-Based System. Journal of Management Information Systems, Vol.5 No.1 (Summer 1988), pp.3-5.

KOPCSO D., PIPINO L. & RYBOLT W. 1988. A Comparison of the Manipulation of Certainty Factors by Individuals and Expert System Shells. Journal of Management Information Systems, Vol.6 No.3 (Winter 1989-90), pp.66-81.

LIANG T-Y. 1990. Expert support systems: an intermediate reality for management. Behaviour & Information Technology, Vol.9 No.4 (1990), pp.297-305.

LIEBOWITZ J. 1989. How much `artificial stupidity do expert systems possess? Information Age, Vol.11 No.4 (Oct.1989) pp.225-227.

MENTZ M. 1988. The Integration of Expert Systems and Decision Support Systems to Solve Business Problems. Technical Report (Hons). Cape Town: University of Cape Town (Accounting Department), May 1988.

METHLIE L.B. 1987. On Knowledge-based Decision Support Systems for Financial Diagnostics in HOLSAPPLE C.W. & WHINSTON A.B.(ed.) Decision Support Systems: Theory and Application. Berlin: Springer-Verlag, 1987, pp.335-372.

MEYER M.H. & CURLEY K.F. 1991. An Applied Framework for Classifying the Complexity of Knowledge-Based Systems. MIS Quarterly, Dec.1991, pp.455-472.

MONEY A., TROMP D. & WEGNER T. 1988. The Quantification of Decision Support Benefits Within the Context of Value Analysis. MIS Quarterly, June 1988, pp.222-236.

MOTODA H. 1990. The Current Status of Expert System Development and Related Technologies in Japan. IEEE Expert, Vol.5 No.4 (Aug.1990), pp.3-12.

MYKYTYN K., MYKYTYN P.P. & SLINKMAN C.W. 1990. Expert Systems: a Question of Liability? MIS Quarterly, Vol.14 No.1 (Mar.1990), pp.26-43.

NEL P.A., RÄDEL F.E. & LOUBSER M. 1988. Researching the South African Market. Pretoria: University of South Africa, 1988.

O'LEARY D.E. & WATKINS P.E. 1988. Prior Surveys on Expert Systems in Accounting, Auditing and Related Areas. Expert Systems Review for Business & Accounting. Vol.I No.4 (Sep.1988), pp.11-12.

PHILIP G.C. & SCHULTZ H.K. 1990. What's Happening With Expert Systems? A Survey of Expert System Users. AI Expert, Vol.5 No.11 (Nov.1990), pp.57-62.

PRICE WATERHOUSE, 1989. Artificial boom in artificial brains. IT Review, 1988/89, pp.12-17.

SCHOONMAKER, P.M. 1898. The Best Laid Plans: Troubleshooting An Expert System. Mechanical Engineering-CIME, Vol.111 No.12 (Dec.1989), pp.56-58.

SPRAGUE, R.H. & WATSON H.J. (eds) 1986. Decision Support Systems. Putting Theory into Practice. Englewood Cliffs, NJ: Prentice-Hall, 1986.

STEENEKAMP C.S. 1984. Praktiese riglyne vir vraelyskonstruksie. Geleentheidspublikasie No.16. Pretoria: Raad vir Geesteswetenskaplike Navorsing, 1984.

TRIPPI R. & TURBAN E. 1989. The Impact of Parallel and Neural Computing on Managerial Decision Making. Journal of Management Information Systems, Vol.6 No.3 (Winter 1989-90), pp.85-98.

TRIPPI R. & TURBAN E. 1990. Investment Management: Decision Support and Expert Systems. Boston (MA): Boyd & Fraser, 1990.

TURBAN E. 1990. Decision Support and Expert Systems: Management Support Systems. New York: Macmillan, 1990.

TURBAN E. & WATKINS P.R. 1986. Integrating Expert Systems and Decision Support Systems in SPRAGUE, R.H. & WATSON H.J. (eds). Decision Support Systems. Putting Theory into Practice. Englewood Cliffs, NJ: Prentice-Hall, 1986, pp.138-152.

WATERMAN D.A. 1986. A Guide to Expert Systems. Reading, MA: Addison-Wesley, 1986.

WEBER G. 1991. The Application of Expert Technology in an Image Processing Environment. Proceedings of the Whitehead Morris/Automated Reasoning conference on Technical Application of Expert Systems, 25-26 June 1991, RSA.

WEITZEL J.R. & KERSCHBERG L. 1989. Developing Knowledge-based Systems: Reorganizaing the System Development Life Cycle. Communications of the ACM, Vol.32 No.4 (Apr.1989), pp482-490.

WHITTEN J.L., BENTLEY L.D. & BARLOW V.M. 1989 Systems Analysis & Design Methods. Homewood, IL: Irwin, 1989.

YOUNG L.F. 1990. Knowledge-Based Systems for Idea-Processing Support. Database, Fall 1990, pp27-33.