Faster Bug Detection for Software Product Lines with Incomplete Feature Models
Sabrina Souto, Divya Gopinath, Marcelo d'Amorim, Darko Marinov, Sarfraz Khurshid, Don Batory


Abstract

A software product line (SPL) is a family of programs that are differentiated by features— increments in functionality. Systematically testing an SPL is challenging because it requires running each test of a test suite against a combinatorial number of programs. Feature models capture dependencies among features and can (1) reduce the space of programs to test and (2) enable accurate categorization of failing tests as failures of programs or the tests themselves, not as failures due to illegal combinations of features. In practice, sadly, feature models are not always available. We introduce SPLif, the first approach for testing SPLs that does not require the a priori availability of feature models. Our insight is to use a profile of passing and failing test runs to quickly identify failures that are indicative of real problems in test or code rather than specious failures due to illegal feature combinations. Experimental results on five SPLs and one large configurable system (GCC) demonstrate the effectiveness of our approach. SPLif enabled the discovery of five news bugs in GCC, three of which have already been fixed.



SPLC 2015 submission - Artifacts
If you have any question, please contact Sabrina Souto. 

1. Experiments for SPLs

2. Experiments for GCC



1. Experiments for SPLs 

Evaluation Environment

The SPLs evaluation was performed against a machine with the following specification:

- Architecture: X86_64

- OS: Mac OS X Lion 10.7.5

- Processor: 2,4 GHz Intel Core i5

- Memory: 4 GB 

- Java version: 1.7.0_65 


Running SPLif

Here is how to run the SPLif evaluation against one of the subjects we used <subject_name> (replace occurrences of <subject_name> with one of the subjects we used: companies, desktopsearcher, gpl, notepad, zipme).

The instructions are for Unix/MacOS, but can easily be changed for Windows:


A. Download the tool and subjects: splif.tar.gz

1) tar -zxvf splif.tar.gz

2) cd splif

B. Run the tests from the specified subject with SPLif, producing a file ../splif/data/<subject_name>.txt, containing the profiles of passing and failing test runs:
1) chmod +x runtests.sh
2) ./runtests <subject_name>
3) Here are the files <subject_name>.txt generated for each subject that we obtained in for the paper results:
- companies.txt
- desktopsearcher.txt
- gpl.txt
- notepad.txt
- zipme.txt

C. From the profiles of passing and failing test runs, generate data for both statistics on tests and the ranking of tests (used in figures 4 and 5):
1) chmod +x genstats.sh
2) ./genstats <subject_name>
3) Here are stats generated for each subject that we obtained in for the paper results (from the results of step B):
- stats_companies.txt
- stats_desktopsearcher.txt
- stats_gpl.txt
- stats_notepad.txt
- stats_zipme.txt

D. From the the ranking of tests, generate configuration rankings considering the 4 prioritization techniques cited in the paper, and then generate a plot comparing the progress on inspecting tests and failing configurations using each technique (the plot, for each subject, is showed in figure 6):
(Prerequisite: R statistical package)
1) chmod +x inspection_progress.sh
2) ./inspection_progress <subject_name>
3) Here are both the data and plots generated for each subject:
- companies.data companies.pdf
- desktopsearcher.data desktopsearcher.pdf
- gpl.data gpl.pdf
- notepad.data notepad.pdf
- zipme.data zipme.pdf

Remarks:
-The steps have to be followed in sequence;
-The step B may take several time (more than one day) to run for all the subjects. GPL and ZipMe are smaller subjects and can be used to obtain results quickly;
-Please, ignore warning messages and test failures.



2. Experiments for GCC 

Ground Truth Model
The models are in Z3 format:
- Garvin's model
- Our model
- Garvin + Our (used in the paper)

Evaluation Environment
The GCC evaluation was performed against a machine with the following specification:
- Architecture: X86_64
- OS: Ubuntu precise (12.04.5 LTS)
- Processor: 4 GHz Intel Core i7
- Memory: 8 GB
- Java version: 1.7.0_65

Running SPLif
A.
Instrument and install GCC
B.
Run tests with SPLif
C.
How to reproduce failures

A. Instrument and install GCC
You can use a virtual environment, already configured, to run SPLif or you can manually build your environment, here are instructions for both:

A.1. Instrument and install GCC Virtual Machine
1) Download and install VirtualBox specific for your operational system: https://www.virtualbox.org/wiki/Downloads
2) Load an image of our environment (link) using VirtualBox
3) Follow the steps: B.3, C.1, C.2.

A.2. Manual
1) Create a directory for GCC.  This will be your main dir.   
> mkdir gcc
   > cd gcc   
   > MAIN_GCC=$PWD
2) Download GCC version 4.8.2
   For example, consider using the mirror from Austria:
   - Click: https://gcc.gnu.org/releases.html
   - Then click: mirror sites (at the top of the page) and pick a mirror (e.g, Austria) 
   - Then choose: "releases", next "gcc-4.8.2", and finally download file "gcc-4.8.2.tar.gz"
3) Decompress ".tar.gz" file
   > tar -zxvf gcc-4.8.2.tar.gz
4) Install prerequisites in $MAIN_GCC dir
   > sudo apt-get install libgmp-dev libmpfr-dev libmpc-dev dejagnu tcl expect
   > sudo pip install regex
5) Download these files under $MAIN_GCC:
   instrument_splat.py, type_to_var, matches_changed.txt, system.h, gcc-dg.exp, mfdg.exp
6) Replace original files: system.h, gcc-dg.exp and mfdg.exp. From $MAIN_GCC:
   > mv system.h $MAIN_GCC/gcc-4.8.2/gcc/system.h
   > mv gcc-dg.exp $MAIN_GCC/gcc-4.8.2/gcc/testsuite/lib/gcc-dg.exp
   > mv mfdg.exp $MAIN_GCC/gcc-4.8.2/libmudflap/testsuite/lib/mfdg.exp
7) From $MAIN_GCC, run the following command (to instrument GCC):
   > python instrument_splat.py gcc-4.8.2 type_to_var matches_changed.txt
8) Validate instrumentation.  From $MAIN_GCC/gcc-4.8.2 run these commands: 
   > find . -name "*.c" | xargs grep "splat_log(" 2>/dev/null | wc -l
   1686
   > find . -name "*.c" | xargs grep "splat_log_h(" 2>/dev/null | wc -l
   2
   > find . -name "*.c" | xargs grep "splat_log_s(" 2>/dev/null | wc -l
   0
   > find . -name "*.c" | xargs grep "splat_log_cs(" 2>/dev/null | wc -l
   21
   > find . -name "*.c" | xargs grep "splat_log_b(" 2>/dev/null | wc -l
   35
9) Configure and build GCC.  From $MAIN_GCC:
   > mkdir objdir
   > cd objdir
   > ./../gcc-4.8.2/configure --prefix=$HOME/gcc-4.8.2 --enable-languages=c,c++
   > make -j $(grep processor /proc/cpuinfo | wc -l)
- If the execution show an error message, follow these two steps:
   > wget http://pan.cin.ufpe.br/splif/gcc/files/libstdc++-symbols.ver
   > mv libstdc++-symbols.ver objdir/x86_64-unknown-linux-gnu/libstdc++-v3/src/

B. Run tests with SPLif
1) Download the following files under $MAIN_GCC: 
   runtest.sh 
   splifgcc.tar.gz
2) Uncompress file splifgcc.tar.gz:
   $MAIN_GCC> tar -zxvf splifgcc.tar.gz
3) Run SPLif on crashing tests reported in the paper
   > cd splif
   > ./splif.sh $MAIN_GCC
   - The output is a raw file, with the profile of pass/fail configurations for each test, generated in:
     $MAIN_GCC/splif/data/gcc.txt

C. How to reproduce failures
1) Run the following script to generate a list of pairs test/configuration that cause crashes:      
   $MAIN_GCC/splifgcc> ./genCrashPairs.sh gcc
   - The output contains lines with this format <script to reproduce crash, test suite, test name, configuration>, for example:
   ./runtest_opt.sh dg pr39794.c /-fno-sanitize=address/-fcheck-data-deps/-fno-tree-vectorize/-fno-tree-pta//-fno-omit-frame-pointer/-fno-ipa-pta/-fno-fat-lto-objects/-fno-strict-aliasing/ 
2) Reproduce a failure reported for one configuration
   Download the script runtest_opt.sh in MAIN_GCC$
   Run this script, just by copying and executing one of the lines generated in previous step, for example:
   ./runtest_opt.sh dg pr39794.c /-fno-sanitize=address/-fcheck-data-deps/-fno-tree-vectorize/-fno-tree-pta//-fno-omit-frame-pointer/-fno-ipa-pta/-fno-fat-lto-objects/-fno-strict-aliasing/

You also can reproduce the step C.2. using the latest stable version when the paper was submitted, that is gcc-4.9.1. For that, follow the steps: A.1.1, A.1.2, A.1., A.1.4, A.1.9, and finally C.2.