An evaluation framework for large-scale ontology-based biomedical data integrated systems

Abstract

There has been an emergence of various ontologies describing data from either the clinical or biological domains. Associated with this has been the development of biomedical ontologies using various strategies to integrate biological and clinical data across scope, process and differing levels of granularity. However, biomedical ontologies still find little use and adoption in distributed computing applications. This is largely attributed to: (i) lack of knowledge about user needs for biomedical data integration systems; and (ii) the absence of a general framework with tools and metrics to assess their relative suitability for specific applications. In an attempt to bridge the gap this research developed a flexible framework for user evaluation of biomedical ontologies. The study adopted a mixed method research design to generate requirements for the evaluation framework. Requirements for the framework were tested in a descriptive survey using 450 medical doctors and biologists as the study population. Concepts from systems theory, basic formal ontology, set theory and multicriteria evaluations were exploited in order to provide a unifying design of the evaluation framework based on user requirements. The framework extends the Ontometric ontology evaluation method while providing new features namely: (i) a reference ontology model for biomedical data integration and (ii) scope, granular density and process density as new metrics for biomedical ontology evaluation. To test the utility of the framework an ontology evaluation tool was built as an application of the design. The tool was used to evaluate the infectious disease ontology and the results validated using a questionnaire based study. The results revealed a strong positive correlation (Pearson's r) between those where the tool was used and the corresponding ones from the questionnaire based study. Since the tool is an application of the framework design, the strong positive correlation provided empirical proof of the validity of the approach using the derived scope, granular and process density as evaluation metrics. The framework contributes to the wide adoption and reuse of biomedical data integration ontologies in the following ways: (i) generating requirements for use as criterion for biomedical ontology integration and evaluation; (ii) a tool for use to gather requirements for extending existing ontologies, resulting into new ones that address current needs for biomedical data integration; (iii) a reference model and metrics for evaluating biomedical ontologies significantly contribute to integrating information systems and to scientific knowledge. The novelty of this approach lies in the ability to combine concepts from systems theory, basic formal ontology, set theory and multicriteria evaluations into a flexible framework for evaluating biomedical ontologies in the dynamic environment of biomedicine. This framework has therefore potential to be extended and reused in other dynamic environments, besides biomedicine.