ParlBench: experimental run


This page describes an experimental run of the ParlBench benchmark on a collection of 1% of proceedings together with paragraphs and tagged entities on the Virtuoso Open-Source Edition (Native RDF Storage).

Table of Content

  1. Introduction
  2. Experimental Run
  3. Benchmark Resources


I Introduction

ParlBench is a scalable RDF benchmark modelling a large scale electronic publishing scenario. The benchmark suite consists of the data sets and a set of queries.

The benchmark data sets include (1) the Dutch parliamentary proceedings, (2) political parties, (3) politicians, (4) paragraphs with textual content of the proceedings and (5) tagged entities which provide links from the paragraphs to DBpedia. The data is real, but free of intellectual property rights issues.

The benchmark defines 19 analytical queries covering wide range of SPARQL constructs. The queries can be viewed as coming from one of two use cases: create a report or perform a scientific research. To enable more comprehensive analysis of performances of RDF stores', the benchmark queries are grouped into four micro benchmarks with respect to the their analytical aims: Average, Count, Factual and Top 10.

The benchmark data sets are described in the ParlBench benchmark in more detail.



II Experimental Run

This section describes an experimental run of the benchmark.

Experimental Setup

Experimental Machine

For the experiment we used a personal laptop Apple MacBook Pro with the following specification:

The System Under Test

The benchmark was run on the OpenLink Virtuoso RDF native store (Open Source Edition). We used Virtuoso v.06.01.3127 compiled from source for OS X. Default RDF index scheme configuration of Virtuoso, i.e., the scheme consisted of the following indices:

We configured the Virtuoso Server to handle load of large data sets:

Performance metrics

In the experimental run we measured the following performance metrics:
  1. Loading Time
  2. Loading time is the time to load an RDF data set to an RDF store system.

    All the data sets are in RDF/XML representation. We measured time in seconds by loading one data sets from the test collection at a time. Members and parties are loaded entirely using the Virtuoso RDF bulk load procedure. Proceedings, Paragraphs and Tagged Entities are loaded using the Virtuoso function to load large RDF/XML text DB.DBA.RDF_LOAD_RDFXML_MT.

  3. Query Response Time
  4. Query response time is the time it takes to execute a SPARQL query.

    We measured the query response time in warm runs, i.e., after executing the benchmark queries several times on the Virtuoso server, in order for Virtuoso to store data in its cache memory. We set the number of warm runs to be 5. Each run consisted of the 10 permutations of the benchmark queries, i.e., 190 queries in total. The total number of queries executed in the warm up stage is 950.

    After warming up Virtuoso, we run the permutations for 3 times and measured the average execution time of each query across these 3 runs. Each query was run 30 times in total.

Procedure

To perform the experimental run, we used the scripts which are available for downloading here .

Software:

The procedure of the experimental run constisted of the following steps:
  1. Generate and Load a Test Collection
  2. The benchmark data sets can be scaled by the number of included proceedings and by including/excluding the paragraphs and tagged entities data sets. The scaling of the benchmark data sets is discussed here in more detail.

    For the exeprimental run, we generated the test collection that consists of the members and parties data sets and 1% of proceedings together with paragraphs and tagged entities. Note, that the members, parties and proceedings data sets are required for answering the benchmark queries.

    Table 1. The sizes of the data sets of the test collection.
    dataset members parties 1% of proceedings togetder witd paragraphs and tagged entities
    number of triples 33,892 510 953,703
    size 14M 612K 152.54M

    To generate the test collection and load it into Virtuoso, we used the script loadRDF.sh. The input of the script is the path to the Benchmark data sets. For the correct run of the script it is important to keep the structure of the folders as it is after unpacking the downloaded data sets.

    The following command was used to run the script on the experimental machine:

    $ ./loadRDF.sh ./input-data/

    The following is the output of running the command above:

    1 $ Do you want to load members? (y/n) y
    2 $ Do you want to load parties? (y/n) y
    3 $ Do you want to load proceedings? (y/n) y
    4 $ Enter the number (integer) indicating what percent of proceedings you want to load? 1
    5 $ Do you want to load paragraphs? (y/n) y
    6 $ Do you want to load tagged entities? (y/n) y
    7 $ Do you want to load proceedings? (y/n) y

    Note, that the number indicating what percent of proceedings you want to load (line 4 of the output above) must be integer.

  3. Generate Permutations of the Benchmark Queries
  4. We created 10 permutations of the benchmark queries. Each permutation consists of all 19 ParlBench queries.

    To create permutations we used the script createPermutations.sh. This script takes as input the number of permutations to be created. The script requires the folder queries with the benchmark queries to be in the same directory. The script creates the folder permutations in the current directory.

    The following is the command we run to generate 10 permutations.

    $ ./createPermutations.sh 10

  5. Execute test runs
  6. To run the queries, we used the script run-queries.sh. The script takes two input parameters: the number of runs of permutations to warm up, and the number of runs to measure average execution time of queries. The script requires the folder permutations, created by the previous step to be in the same directory.

    The following command was used to run the script run-queries.sh on the experimental machine:

    $ ./run-queries.sh 5 3

    The script output is:


Results

  1. Loading Time
  2. Table 1. Loading Time of the Test Collection in seconds.
    data setmembersparties1% of proceedingsparagraphstagged entities
    number of triples33,885510442,849147,830363,024
    loading time (sec)0.1905.42022.17921854.030 2424.558

  3. Query Response Time
  4. Table 2. Loading Time (sec) of the Test Collection in seconds.
    A0-A1-A2-C0-C1-C2-C3-
    0.4694000.4072000.5302000.0601000.0069000.0071000.006900
    C4-C5-C6-C7-C8-C9-C10
    0.0102000.0256000.0106000.0066000.0073000.0265000.147500
    C11C12C13F0-F2-F4-F5-
    0.0203000.0220000.2131000.0308000.5728000.3386000.303800
    F6-F8-T0-T1-T2-T3-T4-
    0.0440000.3736000.0405000.1370000.4244000.0226001.928300

  5. Execution times of all the queries
  6. Figure 2. Execution time in seconds of ParlBench queries on the collection with 1% of proceedings, paragraphs and tagged-entities.
    Execution time in seconds of ParlBench queries.
  7. Raw results
  8. The query execution time for each query in each permutation and run can be downloaded here.

Public SPARQL-endpoint

SPARQL-endpoint provides access to the test collection desdcribed in this page, i.e., with the complete members and parties data sets, 1% of proceedings together with paragraphs and tagged entities sampled according to the selected proceedings.

III Benchmark Resources