This paper investigates a number of ways in which enterprise models can be compared and proposes a framework for assessing the quality of generic enterprise models.

The paper starts with an overview of various qualitative and quantitative yardsticks, taken from related disciplines, that can be used to evaluate generic enterprise models. It then uses some of these to develop a suitable framework for assessing or comparing the quality of enterprise models. The resultant framework is essentially an extension of Bart-Jan Hommes framework for analysing the quality of a business modelling technique, to include some criteria suggested for evaluating the genericity of enterprise reference architectures.

To illustrate how the proposed framework can be operationalised, it is then applied to two well-known "general enterprise models": the reference models underlying two ERP solutions namely those of Baan and SAP AG.

A number of vendors offer enterprise resource planning solutions (ERPs): SAP, BAAN, Oracle and PeopleSoft are some of the better known. These systems are based on a generic enterprise model that is claimed to have universal applicability, though some customisation or configuration of the model may be required. The question arises as to how generic the reference model underlying the ERP systems actually is and how much customisation of the model is required. The systems also require that the underlying enterprise model is an accurate model of the real enterprise.

Other vendors also provide "ready-to-use" enterprise models or model templates for use with popular CASE and/or modelling tools. An example is the BOMA™ (Business Object Management Architecture) library developed by SES Software for Rational Rose developers. Other models are published in reference works for the professional developers, e.g. Silverston, Inmon & Graziano (1997).

As these products become available, there is understandably some debate within the industry as to which enterprise model is more generic or which model is of better quality. Decision-makers and practitioners have thus a need for comparative i.e. evaluative analysis. Which model is "better" (for a given purpose and context)? To our knowledge, no explicit frameworks exist to compare models although a large number of candidates can be identified from related research e.g. linguistic analysis, IT methodology frameworks, ontology research and the like (which look at similar intellectual artefacts). Note that it is not just necessary to develop criteria to compare models within a given modelling approach with each other, there is a need to compare models across different methodologies too. The challenge is akin to selecting e.g. a Nobel prize in literature: how to compare an obscure Greek political play with a popular Chinese historical novel?

The authors ultimate aim is to develop one single contingency-based evaluation framework. A first iteration, presented below, is a heuristic combination of elements from a number of different disciplines. This has been developed by extending Bart-Hommes' framework for evaluating the quality of a modelling technique to include the requirements for a generic enterprise reference architecture.

An enterprise model has been described as: "A computational representation of the structure, processes, information, resources, goals and constraints of an enterprise. It can be both definitional and descriptive - spanning what should be and what is. The role of an enterprise model is to achieve model driven enterprise design, analysis and operation." [Gruninger et al, 1996]

Gale & Eldred [1996] outline the purpose of enterprise modelling as being to enable understanding and improvement of the enterprise; that is, to improve the symbiosis of the individual business objects. "In this sense improvement means any change in co-ordination among the business objects that increases the benefits of symbiosis. The methodology of enterprise modelling is the construction of a set of views of the enterprise considered as a system. The resulting enterprise model contains representations of facts, objects and relationships that occur in the enterprise." [Gale & Eldred, 1996: 11]

Hay [1995] believes that the underlying structure of enterprises is similar, or that they have similar components. Understanding these similarities provides the analyst with a starting model, which can then be adjusted as necessary to match the specific circumstances of a particular company.

Silverston et al. [1997] believe that in general, one third of a data model (corporate or logical) consists of common constructs that are applicable to most organisations, and the other two thirds of the model are either industry - or enterprise-specific. This implies that new data modelling efforts are recreating some data modelling constructs that have already been recreated many times before in other organisations.

Similarly, Fowler takes a process view of enterprise modelling, suggesting that: "There are a small number of highly generic processes that cut across traditional boundaries of systems development and business engineering. The diagnosis and treatment model is one; another is the accounting and inventory model. Many diverse businesses can use a set of very similar abstract process models." [Fowler, 1997: 10]

These views are converging towards defining a generic model that underlies all enterprises. This idea is increasing gaining prominence as academics and practitioners come to realise the underlying similarities in enterprises [Van Belle, 1999].

It is certainly possible to make superficial quantitative comparisons between actual enterprise models, although the comparison becomes more difficult when the underlying modelling approaches differ widely. A simple example of such a comparison is given in table 1. Note that this example is given mainly to illustrate the different types of modelling approaches.

Table 1: Some quantitative measures for generic enterprise models

More detailed comparisons are equally possible but the selection of models then often has to be restricted to those of similar modelling approaches. E.g. when comparing object-oriented models, it is possible to compare a large number of "complexity" measures.

The following represent a selection of frameworks for evaluating the quality of modelling approaches and methodologies. Many of these can be partially adapted to also evaluate the outcome or product of a methodology or modelling process, namely the resultant enterprise model. Many more frameworks have been proposed but there is a high degree of overlap between many of them. The following frameworks are those from which the authors have drawn their data model quality evaluation criteria.

The Seligman Framework

The Seligman (1989) framework for analysing information systems development methodologies proposes that there are five ways or elements that should be considered when analysing Information Systems Development Methodologies (ISDM) in general.

Figure 1 illustrates the structure of the Seligman Framework. Included within the original framework is the way of designing which encompasses both the way of modelling and the way of working.

Figure 1 Seligman's Framework for understanding IS Development Methodologies

Each element can be described as follows:

• Way of Thinking: The way of thinking refers to the (often-implicit) assumptions applied to the perception of the empirical object (ontology), and to the way this can be studied or designed. (Epistemology)
• Way of Modelling: This describes the models used and their relationship, as used during systems development.
• Way of Working: The way of working describes the tasks and sub-tasks that must be carried out during the development of the models. This views modelling as a process based activity which occurs over a period of time.
• Way of Controlling: The monitoring of the activities (project management tasks) described in the way of working is the focus of the way of controlling.

• Way of Supporting: This refers to the tools that support the systems development.

Williams' Criteria for a Good Reference Architecture

Williams [1996] gives a detailed list of the requirements for an Enterprise Reference Architecture (ERA). The following have been selected in view of their later integration into the proposed framework.

1. The ERA should present a method for the breakdown of all system functions to their inherent generic functions and tasks. These should be accomplished with a minimum number of types of basic building blocks;
2. The method should be able to show all existing relationships of those entities represented by the blocks or modules used for modelling;
3. The place of the human in the enterprise should be able to be explicitly shown, functionally and organisationally.

Fox' criteria for Ontologies

Fox (1996) highlighted a number of characteristics which he believes should be used to evaluate a generic ontology. These include:

• Functional Completeness Is all the information necessary for a function to perform its task, represented?
• Perspicuity Is the model easily understood by users so that it can be consistently applied across the enterprise? Does the representation "document itself"?
• Generality Is the model specific to a sector such as retailing, finance etc., or applicable to all sectors?
• Precision/Granularity Is there a core set of ontological primitives? Does the model support reasoning at various levels of abstraction and detail?
• Efficiency Is some sort of transformation required?
• Minimality Does the model contain a minimum number of objects necessary?

Grazier's criteria for comparing ontologies.

Grazier (1998) suggests the following criteria to compare ontologies.

These criteria have been developed outside the normal information systems development paradigm and could be used to add a number of extensions to the proposed model, especially if more knowledge-intensive models will be considered.

Hommes' framework

Bart-Jan Hommes framework for analysing the quality of a business modelling technique (1998) was used as a basis for developing our framework. Hommes bases his framework in turn on Seligmans framework. In his framework, the modelling technique is influenced by two elements: the way of modelling and the way of working.

The way of modelling is influenced by the level of expressiveness, which can be described as how well the conceptual system corresponds to the business system. The way of working is influenced by the level of arbitrariness, which is the degree of freedom the modeller has when modelling a given domain.

The two elements are in turn influenced by four different properties of the resulting model: consistency, correctness, comprehensibility, and usability. The expressiveness of a modelling technique is affected by the consistency and correctness of the model. These are described as follows:

• Consistency is the overlap in modelled concepts between the individual sub-models of a modelling technique. A certain degree of overlap is necessary in order to obtain a coherent model cycle while too much overlap increases the danger of the models becoming inconsistent. Thus a fine balance needs to be struck to ensure that the overlap is sufficient but not overwhelming.

• Correctness refers to the way in which the model describes the corresponding business system i.e. the model should be a clear representation of the business system.

The arbitrariness of a modelling technique is affected by the correctness, comprehensibility and usability of the model. These are described as follows:

• Correctness relates to the determinism (the degree of freedom) of the modelling technique. When the models do not describe the corresponding business system in a clear way, the result is that several different models can describe the same business system, increasing the arbitrariness.
• Comprehensibility also leads to determinism. Modellers that do not fully comprehend the business system and its representation can be forced into choices between options on arbitrary motives and thus towards arbitrariness.
• Usability influences the arbitrariness of a modelling technique since complex models (models that demand a lot of knowledge from a modeller) might not be fully comprehended by the people using them. This will also lead to forced choices between options on arbitrary motives.

The framework is depicted in Figure 2.

It is believed that Hommes' model for evaluating techniques can also be used to evaluate the final products of methodologies i.e. to evaluate the quality of their outputs: (generic) enterprise models. Three requirements were added to the framework to extend its evaluation criteria from that of a quality modelling technique to a generic model.

There is more than one way to represent or model the same knowledge, each representation is of a differing complexity. This also relates to the degree to which the modelling technique is capable of improving the productivity of design and ensuring that there is a degree of consistency is applied to all the models. An efficient modelling technique should use the available resources to leverage generality.

Since an efficient model would be correspond more to the business system, the efficiency attribute of the enterprise reference architecture would be evaluated as being an aspect of the Way of Modelling, and would thus fall under the expressiveness property.

GERAM notes that many of the existing ERP solutions and architecture methodologies are industry specific (such as SAP and CIM-OSA for the manufacturing industry), and specifies that a generic enterprise reference architecture should, by definition, be applicable to all organisations and encompass all industries or sectors.

If the aim of a generic enterprise reference architecture is to model business concepts that are applicable to any and all business systems, then generality would be defined as being a Way of Modelling aspect of the modelling technique, and consequently a characteristic of the techniques arbitrariness.

The generic enterprise reference architecture should encompass all aspects of the organisation. The methodologies that were investigated emphasised this aspect to varying degrees and used different approaches to being a complete architecture.

PERA places more emphasis on the human involvement in the business system than other architectures. GERAM requires that a generic architecture should consider a business system from the following three aspects: the human factors, the customer product or service aspect, and the information systems aspect. In considering the human aspect, GERAM elaborates on the importance of including the human perspective: the tasks, skills, structures and other human-related factors that form part of the holistic business system.

The completeness of the architecture is indicated by the various views that the modelling methodology incorporates: CIM-OSA uses an organisational view, a resource view, an information view, and a functional view; the R/3 architecture uses a process view, a data view, an information flow view, a function view, and an organisational view.

The completeness attribute would be defined as a prerequisite of the modelling technique, and would consequently be grouped under the arbitrariness property of the Way of Working.

Figure 3: Proposed framework for evaluating a generic business model

To illustrate the actual application of the proposed model, it is applied to the underlying reference models of two popular enterprise resource planning packages: SAP AG's R/3 reference model and Baan's Dynamic Enterprise Model(s) (DEM).

Efficiency

Many similarities are apparent in a comparison of DEM and R/3 in terms of the degree of efficiency, as both systems tend to focus the user in the configuration of the system through the use of the process models. However, while DEM supports this through the BPM, SAP requires that the switches in the system be flicked to configure the system. Both systems support the generation of additional code through the ABAP Workbench and BAAN Tools.

Generality

Both R/3 and BAAN were initially developed with a manufacturing focus and then extended. SAP has developed eleven different application models, which are sold as different modules, although a "wall-to-wall" SAP implementation would encompass all the modules. In contrast, BAAN sells into six vertical industries with six major applications, but is still deficient in areas like finance and human resources. DEM integrates the entire value chain, and consequently the system is marketed as components rather than modules. DEMs hybrid model promises to be the most generic, encompassing the entire enterprise in contrast to the earlier industry solutions.

Consistency

The degree of overlap between the models within the architecture should be sufficient enough to provide continuity without leading to any inconsistencies between the models. Within the event driven process chain of R/3 it is possible to view all integration relationships and to map the relationships of processes to each other. Within BAAN there is a significant amount of overlap between the BCM, BFM, BPM, and BOM that results in inconsistency. However the BPM is consistent within itself as it enable the viewing of all relationships.

Correctness

Both R/3 and DEM achieve the purpose that the models aim to achieve by facilitating a rapid and integrated implementation of the system. Within SAP, models were created in terms of four views: function, event, organisation, and information which were then developed into the task, information and organisation models which maps the system in a holistic manner. Within DEM, (the BCM, BFM, BOM and BPM), there is a synchronous relationship between the models which enables the provision of one view of the enterprise merely in different ways.

Usability

Usability is more difficult to evaluate and measure as a consequence of the subject nature of the assessment. However, both SAP and BAAN appear to have usable modelling techniques. R/3 models down to a level where all events, decisions and resources are depicted. At this level the model becomes fairly self-explanatory given the clear definitions provided in the legends. Similarly, within BAAN, the lucid syntax and clear structure enable the user to learn the modelling technique within a relatively short space of time.

Comprehensibility

Both SAP and BAAN enable aspects of the system to be viewed in isolation thus reducing the complexity of the modelling technique. As a consequence of the application of the technique of system reticulation (breaking a system or processes down to a lower level), the ambiguity and arbitrariness of the modelling technique is reduced.

Completeness

Although both R/3 and DEM purport to model the entire business system, it is clear that neither of the models meet this aim. SAP has four different views (process, data/information, organisation, and function) which encompass most aspects of an enterprise. DEM also has four different models (control, function, process and organisation). The event driven process chain of R/3 tends to focus the architecture towards a process orientation, while DEM has an information flow orientation.

Neither DEM or R/3 provide a clear illustration of the place of the actors or human beings in relation to the application. This omission results in a model which provides an inaccurate perception of the business system, devoid of humanness.

Overall evaluation

The application of the proposed framework shows that neither SAP nor BAAN fully meet all the criteria outlined in this framework. The area in which both of these models are particularly lacking is in their ability to show human interaction within the application. The relationship between the people within an enterprise and the business solution is an important one, not only in terms of the application but also in terms of the purpose of a reference architecture modelling the holistic business system.

However, both SAP and BAAN demonstrated higher levels of those requirements associated with a quality modelling technique consistency, correctness, usability and comprehensibility than those of a generic model efficiency, generality, and completeness. This could be attributed to the fact that both solutions were initially manufacturing sector solutions and have only recently begun to extend their applicability to other industries, increasing their levels of generality.

This paper outlined the development of a framework to evaluate a proposed generic enterprise model. An analysis of the requirements for a generic enterprise reference architecture, and a comparison of these requirements with Hommes' framework for evaluating the quality of modelling techniques, indicated that the framework should be extended to cover efficiency, generality, and completeness in order to evaluate the generality of a specific enterprise model.

The framework developed was then applied to two leading ERP solution providers, SAP and BAAN. The results showed that both the ERP solutions encompassed those aspects of quality modelling techniques, but lacked in some of the aspects of "genericity" of their enterprise models. This finding could be expected due to the fact that both vendors have only recently focused their products on becoming more generic solutions, rather than being manufacturing industry-specific solutions. Both vendors had also limited the representation of human interactions and system involvement in their models.

The untried and initiatory nature of the framework, coupled with the nature of the research conducted, resulted in qualitative and somewhat subjective results. The use of the framework itself should ideally be of a much more deterministic nature.

Moreover, some of the requirements describing generality should be revised to include additional ones proposed in ontology research.

Barker, 1995 Foreword : Data Model Patterns - Conventions of thought By Hay, DC : Dorset House Publishing, New York, 1996 pp xvii - xviv

Bernus, P & Nemes L : Enterprise integration - engineering tools for designing enterprises : In Bernus, P & Nemes, L [Ed] Modelling and Methodologies for Enterprise Integration Chapman & Hall, London, 1996

Bernus, P; Nemes, L; & Morris, R : The Meaning of an Enterprise Model ; In Bernus, P & Nemes, L [Ed] Modelling and Methodologies for Enterprise Integration Chapman & Hall, London, 1996.

Cook, M A : Building Enterprise Information Architectures - Re-engineering Information Systems, New York, 1996.

Curran T., Keller G. & Ladd A. SAP R/3 Business Blueprint: Understanding the Business Process Reference Model. Prentice-Hall, Englewood Cliffs (NJ), 1997.

Doumeingts, G & Chen D : State-of-the-art on models, architectures and methodologies; In Bernus, P; Nemes, L, Williams TJ [Ed] : Architectures for Enterprise Integration (First Edition), Chapman & Hall, London, 1996

Dietz, J L G & Mulder, H B F : Organisational transformation requires constructional knowledge of business systems, Unpublished Paper, 1998

Fowler, M : Analysis Patterns - Reusable Object Models, Addison Wesley Longman, Massachusetts, 1997.

Fox, M S and Gruninger, M : Enterprise Modelling, Research paper, Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, 1996.

Gale, T & Eldred, J : Getting Results with the Object-Oriented Enterprise Model, SIGS Books, New York, 1996.

Hay, DC : Data Model Patterns - Conventions of Thought, Dorset House Publishing, New York, 1996.

Hommes, B-J : Analysing the Quality of a Business Modelling Technique, Unpublished Masters Thesis, 1998.

Jayaratna N. Understanding and Evaluating Methodologies. NIMSAD: A Systemic Framework. McGraw-Hill, London, 1994.

Keller, G & Detering S : Process-Oriented Modelling and the Analysis of Business Processes using the R/3 Reference Model : In Bernus, P & Nemes, L [Ed] Modelling and Methodologies for Enterprise Integration (First Edition) Chapman & Hall, London, 1996.

MacGregor, B & Heynes, E : The Contribution of an Information Architecture to the success of BPR Projects, Technical Report, Dept Information Systems, University of Cape Town, 1996.

Martin, RJ : "Explicit Acts of Architecture" Journal of Systems Management Vol. 45, July 1994 pp.28-29.

Nemes, L & Bernus P: Introduction; In Bernus, P; Nemes, L, Williams TJ [Ed] : Architectures for Enterprise Integration, Chapman & Hall, London, 1996.

Niederman, F; Brancheau, JC & Wetherbe, JC : Information Systems Management Issues for the 1990s MIS Quarterly VOL 7 No 4 (December 1991) pp 475-500.

Perkins, A : Enterprise Information Architecture [On-line] Available: http://www.ies.aust.com/~visible/ papers/Architecture.html, 1998.

Price, B; Boxal, L & Walsh, S. Characteristics of Model-based Reference Architectures. Technical Report, Dept of Information Systems, University of Cape Town, 1998.

Sheer A.-W. Business Process Engineering. Reference Models for Industrial Enterprises. Springer-Verlag: Berlin, 1994.

Silverston, L; Inmon, WH; Graziano, K : The Data Model Resource Book - A library of logical data models and data warehouse designs, John Wiley & Sons Inc., New York, 1997

Ushold M., King M., Moralee S. & Zorgios Y. The Enterprise Ontology. Working Paper, AIAI, University of Edinburgh, 1995.

Van Belle, JP : Moving Towards Generic Enterprise Models: from Pacioli to Cyc. Proceedings of the 10^th Australasian Conference on Information Systems (ACIS), Wellington (New Zealand), Dec. 1999, pp. 1084-1094.

Williams, TJ : The needs of the field of Integration : In Bernus, P; Nemes, L, Williams TJ [Ed] : Architectures for Enterprise Integration (First Edition), Chapman & Hall, London, 1996

Zachman, J : Enterprise Architecture and Legacy Systems [On-line] Available: http://www.ies.aust.com /~visible/papers/zachman1.htm, 1998

Zachman, JA : Enterprise Architecture: The issue of the century [On-line] Available: http://www.zifa.com.zifajz01.htm, 1998