OWL2Bench Documentation

This document provides documentation for the first version of our benchmark OWL2Bench (Accepted at the ISWC'20 Resources Track). OWL2Bench can be used to benchmark three aspects of the reasoners - support for OWL 2 language constructs, scalability in terms of ABox size, and the query performance.

Table of Contents

1. Introduction

   1.1 TBox Details

   1.2 ABox Details

   1.3 SPARQL Query Details

2. About the Repository

3. Usage Instructions

   3.1 Direct execution using executable jar (with default configurations)

   3.2 Using Source Code (with or without default configurations)

4. Experiments Performed

5. Future Work

6. References

1. Introduction

OWl 2 is gaining popularity in a variety of domains because of its high level of expressivity. OWL 2 has several profiles such as OWL 2 EL, OWl 2 QL, OWL 2 RL, and OWL 2 DL that vary in terms of their expressivity and reasoning performance. There are several OWL 2 reasoners (such as Hermit, JFact, Openllet, Pellet, Konclude and ELK) and some SPARQL query engines (such as Stardog and GraphDB) that are backed by OWL 2 Reasoners so as to help answer queries that involve reasoning. Our benchmark, OWL2Bench, is an extension of well known University Ontology Benchmark (UOBM). OWL2Bench includes fixed TBox for each profile covering the set of constructs supported by that profile, generation of ABox of varying sizes and a separate set of SPARQL queries for each profile to be executed over generated data for performance evaluation of several reasoners and SPARQL query engines.

The hierarchy among some of the classes, including the relations between them, is shown in the figure below. All the four TBoxes (one for each profile) of OWL2Bench consist of classes such as University, College, CollegeDiscipline, Department, Person, Program, and Course. They are related to each other through relationships such as enrollFor, teachesCourse, and offerCourse. The labeled (dashed) edges represent the properties. The unlabeled edges represent the subclass relation.

1.1 TBox Details

The OWL2Bench TBoxes have been built by enriching the existing UOBM ontology with the supported OWL 2 constructs in each profile. Each TBox follows the syntactic restrictions of the particular profile.

OWL 2 DL : UNIV-BENCH-OWL2DL.owl

OWL 2 RL : UNIV-BENCH-OWL2RL.owl

OWL 2 QL : UNIV-BENCH-OWL2QL.owl

OWL 2 EL : UNIV-BENCH-OWL2EL.owl

1.2 ABox Details

ABox axioms are generated by OWL2Bench based on two user inputs, the number of universities and the OWL 2 profile (EL, QL, RL, DL) of interest. The instance data that is generated complies with the schema defined in the TBox of the selected OWL 2 profile. The size of the instance data depends on the number of universities. The axioms generated from the code are the assertion axioms (class, object property and data property). First of all, the instances (class assertion axioms) for the University class are generated and their number is equal to the number of universities specified by the user. Then for each University class instance, instances for College, Research Group and Publication are generated. For each College, Department instances are generated. For each Department, instances of Man, Woman, Program, and Course are generated. The number of instances that are generated for each class is selected automatically and randomly from the range specified in the configuration file. The other Classes are defined (in the TBox) in such a way that the reasoner can draw inferences for them. For example, a Person class is a union of Man and Woman classes.

Property assertion axioms are created using these instances. For example, an object property isDepartmentOf links a Department instance to a College instance. Similarly, a data property hasName is used to connect a department name to a Department instance. The number of instances of each class (other than University) and the number of connections between all the instances are again selected automatically and randomly from the range specified in the configuration file.

1.3 SPARQL Query Details

OWL2Bench consists of twenty-two SPARQL queries to test the query performance of the OWL 2 reasoners. The SPARQL Queries are available at https://doi.org/10.5281/zenodo.3838735

2. About the Repository

The project repository consists of 2 main directories: OWL2Bench and Experiments.

OWL2Bench is a java source code directory of our benchmark that generates the varying size datasets (TBox + ABox) to compare the reasoning and querying performance of OWL 2 Reasoners (see section 3.2 for source-code usage instructions).

Experiments directory consists of details about the experiments and results that were reported in our paper accepted at the ISWC'20 Resources Track.

The repository also consists of four different TBox for each OWL 2 Profiles (EL, QL, RL and DL): UNIV-BENCH-OWL2EL.owl, UNIV-BENCH-OWL2QL.owl, UNIV-BENCH-OWL2RL.owl, UNIV-BENCH-OWL2DL.owl, and a java executable jar file : OWL2Bench.jar.

3. Usage Instructions

Requirements: The user must have java and maven installed in the system. 24 GB RAM for all the experiments. Operating System-Ubuntu

The user needs to provide two inputs, the number of universities and the OWL 2 profile (EL, QL, RL, DL) of interest. The ABox axioms are generated using OWL2Bench code, complying with the schema defined in the TBox of the selected profile. The size of the ABox depends on the number of universities. The final dataset on which the reasoners are evaluated consists of both TBox and ABox axioms.

3.1. Direct execution using executable jar :

We have provided a java executable jar OWL2Bench.jar that generates the datasets using the default configurations that were used for the experiments reported in the paper. In order to execute this Jar file, user need to give the inputs (in the same order): Number of Universities (mandatory), Required OWL 2 Profile (mandatory), and Seed (optional). If user wishes to generate the exact same datasets that is reported in the paper, use seed value of 1.

For eg. :

java -jar OWL2Bench.jar 10 EL 20 (where 10 is the number of universities, EL is OWL 2 profile and 20 is the seed value)

java -jar OWL2Bench.jar 1 DL (where 1 is the number of universities, DL is OWL 2 profile and the default seed value)

Number of universities makes the ABox scalable. By default, the number of ABox axioms for 1 university is approximately 50,000 that reaches upto 14,000,000 for 200 universities.
To execute OWL2Bench.jar, make sure the TBox for all profiles (UNIV-BENCH-OWL2EL.owl, UNIV-BENCH-OWL2QL.owl, UNIV-BENCH-OWL2RL.owl, UNIV-BENCH-OWL2DL.owl) and excel file for random names RandomNames.xlsx is present in the same directory as jar file. RandomNames.xlsx is used to generate real like names for University and Person instances.

3.2. Using Source Code :

We are also providing the java source code for ABox generation. Using the source code, user can also modify the default configurations (such as generated ontology Format, and density of each node in the generated dataset) in the config.properties file. In order to run the source code, user need download the project repository owl2bench-master. Extract it and save it in a folder. There is a maven project OWL2Bench inside the owl2bench-master directory. Open command line and change to the directory that contains the pom.xml of this OWL2Bench project. Execute the maven command:

mvn compile

mvn install

Now, using maven's exec plugin, run the main class Generator and pass the list of arguments Number of Universities, Required OWL 2 Profile and Seed (same as above) using exec.args. For example-

mvn exec:java -Dexec.mainClass=ABoxGen.InstanceGenerator.Generator -Dexec.args="1 QL 1" , or

mvn exec:java -Dexec.mainClass=ABoxGen.InstanceGenerator.Generator -Dexec.args="1 QL"

The output files are stored in files with names such as "OWL2"+ Profile + "-" + Number of Universities + ".owl" . For example. OWL2DL-1.owl, OWL2QL-1.owl, OWL2EL-10.owl, OWL2RL-100.owl.

Note:

Since we are providing the seed value as one of the inputs to the ABox generation algorithm along with the number of Universities and the desired OWL2 profile, the same set of instances is generated if the seed value remains the same across multiple runs. If the user does not specify the seed value then the default seed value of 1 will be used. For all our datasets that were used in the experiments, we use seed value of 1. Similar principle was used in UOBM as well.

In order to change the default size of the generated ABox as well as to control the density of each node (number of connections between different instances), the range (maximum and minimum values of the parameters) can be modified in the configuration file. Moreover, the output ontology format can also be specified in the configuration file (owx for OWL_XML, ofn for OWL_Functional, omn for OWL_Manchester, ttl for Turtle, rdf for RDF_XML). By default, the generated ontology format is RDF/XML.

4. Experiments Performed

The Experiments directory consists of scripts required for evaluations. The details about the files and experiments performed are provided in the README.

5. Future Work

For the next version of OWL2Bench, we plan to be able to customize the TBox for each profile rather than having a fixed TBox. We also plan to extend this by providing an option to the users to choose the desired hardness level (easy, medium, and hard) of the ontology with respect to the reasoning time and OWL2Bench will then generate such an ontology.

6. References

1. Bail, S., B., Sattler, U.: JustBench: A Framework for OWL Benchmarking. In: The Semantic Web - ISWC 2010 - 9th International Semantic Web Conference, ISWC2010, November 7-11, 2010, Revised Selected Papers, PartI. Lecture Notes in Computer Science, vol. 6496, pp. 32–47. Springer (2010).
2. Glimm, B., Horrocks, I., Motik, B., S., G., Wang, Z.: Hermit: An owl2 reasoner. Journal of Automated Reasoning53(3), 245—-269 (Oct 2014).
3. Guo, Y., Pan, Z., Heflin, J.: Lubm: A benchmark for owl knowledgebase systems. Journal of Web Semantics3(2-3), 158–182 (Oct 2005).
4. Hitzler, P., Kr ̈otzsch, M., Parsia, B., F. Patel-Schneider, P., Rudolph, S.: OWL2 Web Ontology Language Primer (Second Edition) (2012).
5. Kazakov, Y., Kr ̈otzsch, M., Simanˇc ́ık, F.: The incredible elk. Journal of AutomatedReasoning53(1), 1–61 (Jun 2014).
6. Link, V., Lohmann, S., F., H.: Ontobench: Generating custom owl 2 bench-mark ontologies. In: International Semantic Web Conference. pp. 122–130 (2016).
7. Ma, L., Yang, Y., Qiu, Z .and Xie, G., Pan, Y., Liu, S.: Towards a complete owl ontology benchmark. In: The Semantic Web: Research and Appli-cations. pp. 125–139. Springer Berlin Heidelberg, Berlin, Heidelberg (2006).
8. Parsia, B., Matentzoglu, N., Gon ̧calves, R.S., Glimm, B., Steigmiller, A.:The owl reasoner evaluation (ore) 2015 resources. In: The Semantic Web –ISWC 2016. pp. 159–167. Springer International Publishing, Cham (2016).
9. Parsia, B., Matentzoglu, N., Gon ̧calves, R.S., Glimm, B., Steigmiller, A.: The owlreasoner evaluation (ore) 2015 competition report. Journal of Automated Reasoning, 59(4), 455––482 (Dec 2017).
10. Sakr, S., Wylot, M., Mutharaju, R., Le Phuoc, D., Fundulaki, I.: Linked Data -Storing, Querying, and Reasoning. Springer (2018).
11. Sirin, E., Parsia, B., Cuenca Grau, B., Kalyanpur, A., Katz, Y.: Pellet:A practical owl-dl reasoner. Journal of Web Semantics5(2), 51–53 (2007).
12. Steigmiller,A.,Liebig,T.,Glimm,B.:Konclude:System Description.Journal of Web Semantics. 27-28, 78–85(2014).
13. Tsarkov, D., Horrocks, I.: Fact++ description logic reasoner: System descrip-tion. In: Third International Joint Conference on Automated Reasoning ,IJCAR. pp. 292–297. Springer Berlin Heidelberg, Berlin, Heidelberg (2006).