Post

Qstress Guide

Posted
By Brian Law 7 min read

qstress is a simple tool for stress testing in competitive programming.

Installation

Install qstress using a Python package manager (pipx is recommended for Python executables).

1

$ pipx install qstress

or

1

$ pip install qstress

Usage

Compare

Generates test cases and compares outputs from two programs. Compares the outputs of main_file and slow_file using tests generated by gen_file.

Example

We will use main.cpp, slow.cpp, and gen.cpp. The task is just outputting a + b.

We will purposely provide an incorrect solution in main.cpp.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

#include <cstdint>
#include <ios>
#include <iostream>

void solve() {

    std::int32_t a;
    std::int32_t b;

    std::cin >> a >> b;

    if (a == 0) {
        std::cout << 0 << '\n';
        return;
    }

    std::cout << a + b << '\n';

}

int main() {

    std::cin.tie(nullptr);

    std::ios_base::sync_with_stdio(false);

    solve();

    return 0;

}

We add a slower but correct solution in slow.cpp.

Keep whitespace consistent between the two programs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

#include <cstdint>
#include <ios>
#include <iostream>

void solve() {

    std::int32_t a;
    std::int32_t b;

    std::cin >> a >> b;

    while (b) {
        if (b < 0) {
            --a;
            ++b;
        } else {
            ++a;
            --b;
        }
    }

    std::cout << a << '\n';

}

int main() {

    std::cin.tie(nullptr);

    std::ios_base::sync_with_stdio(false);

    solve();

    return 0;

}

We also provide a generator to generate random test cases.

When writing a generator, make sure to check if the problem uses multitest!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

#include <chrono>
#include <cstdint>
#include <ios>
#include <iostream>
#include <random>

namespace generate {

    std::mt19937_64 rng(std::chrono::steady_clock::now().time_since_epoch().count());

    template <typename T>
    T next_int(T a, T b) {

        return std::uniform_int_distribution<T>(a, b - 1)(rng);

    }

}

void gen() {

    const std::int32_t a = generate::next_int(-10, 11);
    const std::int32_t b = generate::next_int(-10, 11);

    std::cout << a << ' ' << b << '\n';

}

int main() {

    std::ios_base::sync_with_stdio(false);

    gen();

    return 0;

}

We can check our generator by generating a test.

1

$ qstress gen gen.cpp

To start stress testing, we use the cmp command.

1

$ qstress cmp main.cpp slow.cpp gen.cpp

You should probably get output indicating that you found a failing test case. If not, you probably got very unlucky and none of the generated tests caused an incorrect answer. In that case, rerun the previous command and consider increasing tests.

The failing test cases are saved as <folder>/input_<n>.txt, where <folder> is test_cases by default.

To view the failing test cases, you can either open the files or use the view command.

1

$ qstress view

Arguments

OptionRequiredTypeDescription
main_fileYesstringFile to stress test
slow_fileYesstringFile to compare against
gen_fileYesstringFile to generate tests
testsNointegerMax number of tests to run
findNointegerMax number of failing tests to find
folderNostringFolder to save failing tests

Check

Generates test cases and checks whether output is valid. Checks the output of main_file with check_file using tests generated by gen_file.

Example

We will use main.cpp, check.cpp, and gen.cpp. The task is finding n integers between a and b that sum to s. It is guaranteed that there is a valid construction.

We will purposely provide an incorrect solution in main.cpp.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

#include <cstdint>
#include <ios>
#include <iostream>

void solve() {

    std::int32_t a;
    std::int32_t b;
    std::int32_t n;
    std::int32_t s;

    std::cin >> n >> a >> b >> s;

    for (std::int32_t i = 0; i < n - 1; ++i) {
        std::cout << a << ' ';
    }

    std::cout << s - a * (n - 1) << '\n';

}

int main() {

    std::cin.tie(nullptr);

    std::ios_base::sync_with_stdio(false);

    solve();

    return 0;

}

We add a checker in check.cpp. The checker accesses the generated input from input.txt and reads the output from standard input. The checker then outputs 1 if the output is valid or 0 if the output is invalid.

This is different from typical Unix conventions, where 0 indicates success.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

#include <cstdint>
#include <fstream>
#include <ios>
#include <iostream>

std::ifstream fin("input.txt");

void check() {

    std::int32_t a;
    std::int32_t b;
    std::int32_t n;
    std::int32_t s;

    fin >> n >> a >> b >> s;

    std::int32_t sum = 0;
    bool valid = true;

    for (std::int32_t i = 0; i < n; ++i) {
        std::int32_t val;
        std::cin >> val;
        valid &= val >= a && val <= b;
        sum += val;
    }

    valid &= sum == s;

    std::cout << valid << '\n';

}

int main() {

    std::cin.tie(nullptr);

    std::ios_base::sync_with_stdio(false);

    check();

    return 0;

}

We also provide a generator to generate random test cases.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

#include <chrono>
#include <cstdint>
#include <ios>
#include <iostream>
#include <random>

namespace generate {

    std::mt19937_64 rng(std::chrono::steady_clock::now().time_since_epoch().count());

    template <typename T>
    T next_int(T a, T b) {

        return std::uniform_int_distribution<T>(a, b - 1)(rng);

    }

}

void gen() {

    const std::int32_t a = generate::next_int(1, 11);
    const std::int32_t n = generate::next_int(2, 7);

    std::cout << n << ' ' << a << ' ';

    const std::int32_t b = generate::next_int(a, 11);

    std::cout << b << ' ';

    const std::int32_t s = generate::next_int(a * n, b * n + 1);

    std::cout << s << '\n';

}

int main() {

    std::ios_base::sync_with_stdio(false);

    gen();

    return 0;

}

To start stress testing, we use the check command.

1

$ qstress check main.cpp check.cpp gen.cpp

You should probably get output indicating that you found a failing test case. If not, you probably got very unlucky and none of the generated tests caused an incorrect answer. In that case, rerun the previous command and consider increasing tests.

The failing test cases are saved as <folder>/input_<n>.txt, where <folder> is test_cases by default.

To view the failing test cases, you can either open the files or use the view command with the checker flag.

1

$ qstress view --checker

Arguments

OptionRequiredTypeDescription
main_fileYesstringFile to stress test
check_fileYesstringFile to check outputs
gen_fileYesstringFile to generate tests
testsNointegerMax number of tests to run
findNointegerMax number of failing tests to find
folderNostringFolder to save failing tests

Configuration

The configuration is defined in ~/.qstress/config.json.

Default Configuration

1
2
3
4
5
6
7
8
9
10
11
12
13

{

    "compilerBin": "g++",

    "compileArgs": ["-O2", "-std=c++17"],

    "tests": 200,

    "find": 1,

    "folder": "test_cases"

}

Configuration Values

OptionTypeDescription
compilerBinstringPath to the compiler
compileArgsarray<string>Flags for the compiler
testsintegerMax number of tests to run
findintegerMax number of failing tests to find
folderstringFolder to save failing tests
Programming, Guide
This post is licensed under CC BY 4.0 by the author.