Summary of all benchmark classes in the SCore competition
This section presents the results for the benchmark classes of the First
Answer Set Programming System Competition.
Each single run was limited to 600 seconds execution time and 448 MB RAM memory
usage.
For the sake of completeness, we provide similar data for all benchmarks used in
each competition class. In addition, the tables give the number of instances
selected from each class ("#") and provide additional information on the
satisfiable and unsatisfiable instances.
Results for SAT and UNSAT instances
| Benchmark Class | # | Solved | % | SAT | % | UNSAT | % | min | max | avg |
|---|
| 15-Puzzle | 10 | 236/260 | 90.77 | 121/130 | 93.08 | 115/130 | 88.46 | 0.74 | 480.13 | 25.49 |
| Factoring | 5 | 114/130 | 87.69 | 46/52 | 88.46 | 68/78 | 87.18 | 1.21 | 554.76 | 50.35 |
| RLP-150 | 14 | 306/364 | 84.07 | 21/26 | 80.77 | 285/338 | 84.32 | 0.34 | 205.03 | 22.01 |
| RLP-200 | 14 | 287/364 | 78.85 | 0/0 | | 287/364 | 78.85 | 0.39 | 581.98 | 75.21 |
| Schur Numbers | 5 | 99/130 | 76.15 | 88/104 | 84.62 | 11/26 | 42.31 | 2.76 | 561.20 | 49.82 |
| EqTest | 5 | 93/130 | 71.54 | 0/0 | | 93/130 | 71.54 | 0.66 | 592.24 | 75.02 |
| Hamiltonian Path | 14 | 219/364 | 60.16 | 201/338 | 59.47 | 18/26 | 69.23 | 0.24 | 559.88 | 64.74 |
| Random Non-Tight | 14 | 216/364 | 59.34 | 38/52 | 73.08 | 178/312 | 57.05 | 0.57 | 598.20 | 121.87 |
| Blocked N-Queens | 14 | 167/364 | 45.88 | 55/156 | 35.26 | 112/208 | 53.85 | 13.70 | 587.40 | 110.59 |
Results for SAT instances only
| Benchmark Class | # | Solved | % | min | max | avg |
|---|
| 15-Puzzle | 5 | 121/130 | 93.08 | 0.81 | 332.30 | 18.80 |
| Factoring | 2 | 46/52 | 88.46 | 1.35 | 298.80 | 54.61 |
| Schur Numbers | 4 | 88/104 | 84.62 | 2.76 | 561.20 | 39.70 |
| RLP-150 | 1 | 21/26 | 80.77 | 0.56 | 38.49 | 11.75 |
| Random Non-Tight | 2 | 38/52 | 73.08 | 1.04 | 450.38 | 70.50 |
| Hamiltonian Path | 13 | 201/338 | 59.47 | 0.24 | 559.88 | 61.88 |
| Blocked N-Queens | 6 | 55/156 | 35.26 | 13.93 | 587.40 | 129.19 |
Results for UNSAT instances only
| Benchmark Class | # | Solved | % | min | max | avg |
|---|
| 15-Puzzle | 5 | 115/130 | 88.46 | 0.74 | 480.13 | 32.52 |
| Factoring | 3 | 68/78 | 87.18 | 1.21 | 554.76 | 47.46 |
| RLP-150 | 13 | 285/338 | 84.32 | 0.34 | 205.03 | 22.76 |
| RLP-200 | 14 | 287/364 | 78.85 | 0.39 | 581.98 | 75.21 |
| EqTest | 5 | 93/130 | 71.54 | 0.66 | 592.24 | 75.02 |
| Hamiltonian Path | 1 | 18/26 | 69.23 | 1.24 | 399.79 | 96.69 |
| Random Non-Tight | 12 | 178/312 | 57.05 | 0.57 | 598.20 | 132.84 |
| Blocked N-Queens | 8 | 112/208 | 53.85 | 13.70 | 471.36 | 101.46 |
| Schur Numbers | 1 | 11/26 | 42.31 | 24.97 | 278.94 | 130.81 |
|