News

BugsJS Poster

There will be a poster show-casing BugsJS on the ICST19 Poster Track. If you are interested, check it out!
[Poster]

BugsJS at ICST19!

BugsJS is going to be presented at the ICST19 conference. Come and meet us!

About BugsJS

BugsJS is a JavaScript bug dataset which includes 453 reproducible bugs from 10 popular, mature and active Node.js server-side JavaScript programs from GitHub that adopt the Mocha testing framework.

The bugs have been validated and cleaned to ensure that the bugs and their fixes were relevant, isolated, and reproducible.

• relevance: the bug-fixing commits are actually fixing the bug described in the issue, rather than, for example, implementing a new feature.
• isolation: the bug-fixing code exclusively aim at fixing the bug, and no other changes (e.g., refactorings) are interleaved within it. Isolation is particularly important in domains where bug-fixing changes should be clearly identifiable, hence must be cleaned (e.g., training machine learning models to learn the fixes from source code to devise automated program repair techniques).
• reproducibility: we have manually extracted the test cases that demonstrate each bug. Each set of such tests was executed on the buggy revision (i.e., dynamic validation) to ensure their relevance to the bug, and can be leveraged, for instance, to devise novel automated test generation techniques.

Framework

BugsJS consists of two other components: a Docker image that provides a runtime environment, and a bug dataset. The dataset can be queried through an API that supports the following commands:

• info : printing out information about a given bug
• checkout : checking-out the source code
• test : running all tests and retrieving the overall test suite coverage
• per-test : running each test individually and retrieving the per-test coverage

These commands facilitate the use of BugsJS for other researchers to a large extent
GitHub’s fork functionality was use to make a full copy of the git history of the subject systems. The fork is a separate git repository; therefore, we can push commits to it. Taking advantage of this possibility, we have extended the repositories with additional commits, to separate the bug-fixing commits and their corresponding tests. To make such commits easily identifiable, we tagged them using the following notation ( X denotes a sequential bug identifier):

• Bug-X : The parent commit of the revision in which the bug was fixed (i.e., the buggy revision)
• Bug-X-original: A revision with the original bug-fixing changes (including the production code and the newly added tests)
• Bug-X-test: A revision containing only the tests introduced in the bug-fixing commit, applied to the buggy revision
• Bug-X-fix: A revision containing only the production code changes introduced to fix the bug, applied to the buggy revision
• Bug-X-full: A revision containing both the cleaned fix and the newly added tests, applied to the buggy revision

Subjects

To select appropriate subject systems to include in BugsJS, we focused on popular and trending JavaScript projects on GitHub. Popularity was measured using the projects’ Stargazers count (i.e., the number of stars owned by the subject’s GitHub repository). Server-side Node.js applications were selected which are popular (Stargazers count ≥ 100) and mature (number of commits > 200), and have been actively maintained (year of the latest commit ≥ 2017). Particularly, only projects have been selected in which bug reports had a dedicated issue label on GitHub’s Issues page, which allows filtering irrelevant issues (pertaining to, e.g., feature requests, build problems, or documentation tasks), so that only actual bugs are included.

Download

BugsJS

Paper

BugsJS: A Benchmark of JavaScript Bugs

Péter Gyimesi, Béla Vancsics, Andrea Stocco, Davood Mazinanian, Árpád Beszédes, Rudolf Ferenc and Ali Mesbah

JavaScript is a popular programming language that is also error-prone due to its asynchronous, dynamic, and loosely-typed nature. In recent years, numerous techniques have been proposed for analyzing and testing JavaScript applications. However, our survey of the literature in this area revealed that the proposed techniques are often evaluated on different datasets of programs and bugs. The lack of a commonly used benchmark limits the ability to perform fair and unbiased comparisons for assessing the efficacy of new techniques. To fill this gap, we propose BugsJS, a benchmark of 453 real, manually validated JavaScript bugs from 10 popular JavaScript server-side programs, comprising 444k LOC in total. Each bug is accompanied by its bug report, the test cases that detect it, as well as the patch that fixes it. BugsJS features a rich interface for accessing the faulty and fixed versions of the programs and executing the corresponding test cases, which facilitates conducting highly-reproducible empirical studies and comparisons of JavaScript analysis and testing tools.

Poster paper

Poster: Supporting JavaScript Experimentationwith BugsJS

Béla Vancsics, Péter Gyimesi, Andrea Stocco, Davood Mazinanian, Árpád Beszédes, Rudolf Ferenc and Ali Mesbah

In our recent work, we proposed BugsJS, a benchmark of several hundred bugs from popular JavaScript server-side programs. In this abstract paper, we report the results of our initial evaluation in adopting BugsJS to support an experiment in fault localization. First, we describe how BugsJS facilitated accessing the information required to perform the experiment, namely, test case code, their outcomes, their associated code coverage and related bug information. Second, we illustrate how BugsJS can be improved to further enable easier application to fault localization research, for instance, by filtering out failing test cases that do not directly contribute to a bug. We hope that our preliminary results will foster researchers in using BugsJS to enable highly-reproducible empirical studies and comparisons of JavaScript analysis and testing tools.

How to contribute?

If you have a feature request, open an Issue ticket on the [GitHub repository]

If you would like to add another project to the benchmark, submit a pull request with the necessary CSV files and contact us, so we can fork the repository under the BugsJS Organization.

If you would like to add new bugs to the benchmark, submit a pull request with the modified CSV files and contact us, so we can add the extra commits and tags to the fork.

How to cite?

Péter Gyimesi, Béla Vancsics, Andrea Stocco, Davood Mazinanian, Árpád Beszédes, Rudolf Ferenc and Ali Mesbah. 2019. BugsJS: A Benchmark of JavaScript Bugs. In Proceedings of 12th IEEE International Conference on Software Testing, Verification and Validation (ICST).

@inproceedings{2019-Gyimesi-ICST,
author = {P\'{e}ter Gyimesi and B\'{e}la Vancsics and Andrea Stocco and Davood Mazinanian and \'{A}rp\'{a}d Besz\'{e}des and Rudolf Ferenc and Ali Mesbah},
title = {{BugJS}: A Benchmark of JavaScript Bugs},
booktitle = {Proceedings of 12th IEEE International Conference on Software Testing, Verification and Validation (ICST)},
year = {2019},
}

Contact