Basic Example¶
The code in this directory contains a very simple working example of the
features of pytest-experiments
.
To run this example:
Open terminal and change directory into the
demo
directory.Ensure that
pytest-experiments
has been installed.Run
pytest
at the command line.
After running pytest
, you should see some output to terminal that ends
with:
test_algorithm.py ........ [100%]
============================== 8 passed in 0.25s ==============================
Even though we have only defined one experiment, the experiment is run once for each
value of the input, x
. Since we gave it 8 values, the experiment is run 8 times.
You should now see a file in this directory called experiments.db
this is a sqlite
database containing the results of our experiments. It is automatically created and will
be automatically updated with new results. You can specify a different database with the
--experiments-database
option to pytest like so:
pytest --experiments-database sqlite::///results.db
Let’s take a look at the database. You may use any tool that interfaces with
sqlite. In the database you will find a table called experiments
with 8 rows:
id |
start_time |
end_time |
name |
outcome |
parameters |
data |
---|---|---|---|---|---|---|
1 |
2021-12-13 14:11:36.632226 |
2021-12-13 14:11:36.634156 |
demo/test_algorithm.py::test_my_algorithm[-4] |
passed |
{“x”: -4} |
{“y”: 16} |
2 |
2021-12-13 14:11:36.652584 |
2021-12-13 14:11:36.654346 |
demo/test_algorithm.py::test_my_algorithm[-3] |
passed |
{“x”: -3} |
{“y”: 9} |
3 |
2021-12-13 14:11:36.671212 |
2021-12-13 14:11:36.672924 |
demo/test_algorithm.py::test_my_algorithm[-2] |
passed |
{“x”: -2} |
{“y”: 4} |
4 |
2021-12-13 14:11:36.690071 |
2021-12-13 14:11:36.692138 |
demo/test_algorithm.py::test_my_algorithm[-1] |
passed |
{“x”: -1} |
{“y”: 1} |
5 |
2021-12-13 14:11:36.709403 |
2021-12-13 14:11:36.711488 |
demo/test_algorithm.py::test_my_algorithm[0] |
passed |
{“x”: 0} |
{“y”: 0} |
6 |
2021-12-13 14:11:36.727766 |
2021-12-13 14:11:36.729193 |
demo/test_algorithm.py::test_my_algorithm[1] |
passed |
{“x”: 1} |
{“y”: 1} |
7 |
2021-12-13 14:11:36.745785 |
2021-12-13 14:11:36.747479 |
demo/test_algorithm.py::test_my_algorithm[2] |
passed |
{“x”: 2} |
{“y”: 4} |
8 |
2021-12-13 14:11:36.764671 |
2021-12-13 14:11:36.766846 |
demo/test_algorithm.py::test_my_algorithm[3] |
passed |
{“x”: 3} |
{“y”: 9} |
9 |
2021-12-13 14:13:27.304881 |
2021-12-13 14:13:27.306669 |
demo/test_algorithm.py::test_my_algorithm[-4] |
passed |
{“x”: -4} |
{“y”: 16} |
In order, the fields are:
- id
A unique id for the experiment run.
- start_time, end_time
The start and end time of the experiment as a UTC timestamp.
- name
The fully qualified name of the experiment including the file, the name of the test and the value of any parameters
- outcome
The outcome of the test, one of: passed, failed, skipped
- parameters
The value of the inputs to the experiment.
- data
The data recorded (via
notebook.record
) during the experiment.