Artifact of Balancing Soundness and Efficiency for Practical Testing of Configurable Software

Abstract

Testing configurable systems is important and challenging due to the enormous space of configurations where errors can hide. Existing approaches to test these systems are often costly or unreliable. This paper proposes S-SPLat, a technique that combines heuristic sampling with symbolic search to obtain both breadth and depth in the exploration of the configuration space. S-SPLat builds on SPLat, our previously developed technique, that explores all reachable configurations from tests. In contrast to its predecessor, S-SPLat sacrifices soundness in favor of efficiency. We evaluated our technique on eight software product lines of various sizes and on a large configurable system – GCC. Considering the results for GCC, S-SPLat was able to reproduce all five bugs that we previously found in a previous study with SPLat but much faster and it was able to find two new bugs in a recent release of GCC. Results suggest that it is preferable to use a combination of simple heuristics to drive the symbolic search as opposed to a single heuristic. S-SPLat and our experimental infrastructure are publicly available.

Sabrina Souto, Marcelo d'Amorim, Rohit Gheyi

Preprint: s-splat-preprint-icse-2017.pdf

Artifact Evaluation - ICSE 2017 submission

The artifact evaluation contains all the artifacts used in this paper, including the source code, the data set and scripts to generate tables and plots. We explain how to reproduce the results using a single virtual machine. If you have any question, please contact sabrinadfs@gmail.com.

Artifact Setup

Download Link: s-splat.tar

Prerequisite: In order to set up the VM, you need to install a version of Docker.

Setup: Start the Docker default machine and load the image containing our VM image (created on Ubuntu 15.04). After start Docker, start the terminal and run the following commands:

  > sudo docker-machine create --driver virtualbox default (create a new Docker VM)
  > sudo docker-machine ls (list your available machines)
  > sudo docker-machine env default (get the environment commands for your new VM)
  > sudo eval "$(docker-machine env default)" (connect your shell to the default machine)
  > sudo docker load < /yourpath/s-splat.tar (load our image)
  > sudo docker images (check the available images)
  > sudo docker run -i -t s-splat /bin/bash (run the image)

Artifact Organization

  The important folders and files in the VM image are listed as follows: 
  * 1) For SPLs
        * /spls/s-splat: contains the source code of S-SPLat, scripts and library dependencies
        * /spls/s-splat/src-subjects: contains the source code of the SPLs used in our experiments
        * /spls/s-splat/data: the raw data from results of S-SPLat execution will go to this folder
        * /spls/s-splat/plots-figs: the dataset processed from results goes to this folder
  * 2) For GCC
        * /gcc6: contains the instrumented and installed version of GCC 6.1.0 scripts and library dependencies
        * /gcc6/s-splatgcc: contains the source code the source code of S-SPLat specific for GCC, scripts and library dependencies
        * /gcc6/s-splatgcc/data: the raw data from results of S-SPLat execution will go to this folder, and the tests executed are also in this folder
        * /gcc6/s-splatgcc/plots-figs: the dataset processed from results goes to this folder
  * 3) Datasets
        * /dataset/data-plots: contains a folders for each figure generated from results, where each one of these folders contain the dataset, the correspondent plot(s), and the scripts to generate the plots
        * /dataset/rawdata-results: contains the results from the execution of S-SPLat with SPLs(/dataset/rawdata-results/spls), GCC-6.1.0 (/dataset/rawdata-results/gcc6), and GCC-4.8.2 (/dataset/rawdata-results/gcc4)

Replication

1) Experiments for SPLs

Here is how to run the S-SPLat evaluation against all the subjects we used (companies, desktopsearcher, email, gpl, jtopas, notepad, sudoku, and zipme). The instructions are for our VM image:

Running S-SPLat

 1. > cd spls/s-splat/
 2. > chmod +x ssplat 
 3. > chmod +x runssplat.sh 
 4. > ./runssplat.sh (run S-SPLat for all subjects, this may take about 5 days)

The output of S-SPLat execution is stored in /spls/s-splat/data/ (this output contains all configurations sampled by each technique for all tests, and the test results). It will produce a file for each subject and technique, for example: /spls/s-splat/data/gpl_pairwise.txt.

Reproducing Results

 1. > chmod +x genstats.jar (inside /spls/s-splat/)
 2. > java -jar genstats.jar (this may take about 10 minutes)
 Remark: these commands must be executed after running S-SPLat.

The output of genstats.jar execution is stored in /spls/s-splat/plots-figs/samplesize-versus-numfailures/.txt. These sample sets are used to generate Figures 6, 7, and 8 of the paper.

To generate the plots, run ./genplot.sh inside the corresponding plot folder in /spls/s-splat/plots-figs/. For example: run ./genplot.sh in /spls/s-splat/plots-figs/samplesize-versus-numfailures/ and it will generate grid.pdf with the corresponding plot.

2) Experiments for GCC

Running S-SPLat

 1. > cd /gcc6/s-splatgcc/
 2. For each technique run this command: 
    > ./ssplat.sh /gcc6/ <technique> 
    (<technique>: med, oe, od, pw - example ./ssplat.sh /gcc6/ oe)
    The execution of S-SPLat for all 4 techniques may take about a week.

The output of S-SPLat execution is stored in /gcc6/s-splatgcc/data/ (this output contains all configurations sampled by each technique for all tests, and the test results). It will produce a file for each technique, for example: /gcc6/s-splatgcc/data/gcc.pw.txt.

Reproducing Results

 1. > chmod +x genstats.jar (inside /gcc6/s-splatgcc/)
 2. > java -jar genstats.jar (this may take about 10 minutes)
 Remark: these commands must be executed after running S-SPLat.

The output of genstats.jar execution is stored in /gcc6/splat/plots-figs/. These sample sets are used to generate Figures 10a, 10b, 10c of the paper. The rest of figures were manually built.

To generate the plots, run ./genplot.sh inside the corresponding plot folder in /gcc6/s-splatgcc/plots-figs/. For example: run ./genplot.sh in /gcc6/s-splatgcc/plots-figs/gcc-samplesize-versus-numfailures/ and it will generate gcc.pdf with the corresponding plot.