How to Reproduce Results?

Download the pre-configured VM here and follow the remaining instructions on this page. The VM contains a pre-configurated installation of qCORAL and RealPaver. Note that the VM doesn't come with Mathematica 9.0 installed - you will need to do this manually (using your own license file).

Reproducing Table 1 : Variance reduction for the example case.

Inside the 'scripts' folder, run 'runTable1.sh'. The results will be printed on the screen.

  • Summary: 
$qcoral:~/qcoral> cd scripts
$qcoral:~/qcoral/scripts> ./runTable1.sh

Reproducing Table 2 : Microbenchmarks.

Inside the 'scripts' folder, run 'runTable2.sh'. Two files will be generated: one csv file with the raw data from the executions, and a report in txt format. The location of both files will be printed on the screen.

  • Summary: 
$qcoral:~/qcoral> cd scripts
$qcoral:~/qcoral/scripts> ./runTable2.sh
>> Table2: running qCORAL 30 times for each microbenchmark <<
>> Preparing csv file with raw data... (../output/data-table2.csv) <<

%%%%%%%% Running for 1000 samples %%%%%%%%%%%%%%
[...]

Reproducing Table 3 (part 1): VolComp

Requirements:

To reproduce VolComp results, please visit the link above and follow the instructions on the link above to download/run the tool. All benchmarks from the VolComp release used in this project (in qCORAL input language) can be found at the 'inputs/pldi/' folder. Check the README in this folder for more info.

Reproducing Table 3 (part 2): Mathematica

Requirements:

Setup for Mathematica:

  • Set the location of the JLink library (part of the Mathematica distribution) at the file "scripts/variables"
  • Make sure that the 'math' command (to start the Mathematica kernel) is in your $PATH variable
  • Summary: 
$qcoral:~/qcoral> vi scripts/variables #change variable "JLINK"
$qcoral:~/qcoral> math
Mathematica 9.0 for Linux x86 (64-bit)
Copyright 1988-2013 Wolfram Research, Inc.

In[1]:=

To reproduce Mathematica results, run the script 'runTable3Mathematica.sh'. The location of the files containing the raw csv data and the report will be printed on the screen. This experiment should take 2 hours. 

$qcoral:~/qcoral> cd scripts
$qcoral:~/qcoral/scripts> ./runTable3Mathematica.sh
>> WARNING: VOL benchmark is *not* included by default. To include it, re-execute this script with '--with-vol'
>> Preparing csv file with raw data... (../output/data-table3-math.csv) <<
[...]
>> ok, done. The report can be found at ../output/report-table3-math.csv <<

Reproducing Table 3 (part 3): qCORAL

To reproduce qCORAL results, run the script 'runTable3.sh'. To reproduce Mathematica results, run the script 'runTable3Mathematica.sh'. Like before, the location of the files containing the raw csv data and the report will be printed on the screen. This experiment should take around 6 hours to complete.

  • Summary: 
$qcoral:~/qcoral> cd scripts
$qcoral:~/qcoral/scripts> ./runTable3.sh
>> Table3: running qCORAL 30 times for each VolComp benchmark<<
>> WARNING: VOL benchmark is *not* included by default. To include it, re-execute this script with '--with-vol'
>> Preparing csv file with raw data... (../output/data-table3.csv) <<
[...]
>> ok, done. The report can be found at ../output/report-table3.csv <<

Notes

In both cases, the VOL benchmark will not be executed by default because it takes a long time by itself (around 14 hours for qCORAL/+15h for Mathematica). To include it, execute the respective script with the option '--with-vol'.

Reproducing Table 4 (part 1): Mathematica

Requirements:

  • Mathematica 9 (check above how to configure it)

To reproduce Mathematica results, run the script 'runTable4Mathematica.sh'. Once more, the location of the files containing the raw csv data and the report will be printed on the screen. The experiment should take around 16 hours.

  • Summary: 
$qcoral:~/qcoral/scripts> ./runTable4Mathematica.sh
>> Preparing csv file with raw data... (../output/data-table4-math.csv) <<
=========== 1000 ===========

../inputs/jpf/turnlogic_cons_leaves_sample70
[...]
>> ok, done. The report can be found at ../output/report-table4-math.csv <<

Reproducing Table 4 (part 2): qCORAL

To reproduce qCORAL results, run the script 'runTable4.sh'. Once more, the location of the files containing the raw csv data and the report will be printed on the screen. The first experiment should take around 30 hours

  • Summary: 
$qcoral:~/qcoral/scripts> ./runTable4.sh
>> Preparing csv file with raw data... (../output/data-table4.csv) <<
=========== 1000 ===========

../inputs/jpf/turnlogic_cons_leaves_sample70
[...]
>> ok, done. The report can be found at ../output/report-table4.csv <<