This adds a small script that parses two run_tests logs for session
times and compares them.
Signed-off-by: Gerwin Klein <gerwin.klein@proofcraft.systems>
On GitHub, the output of external processes such as isabelle overtake
the stdout/stderr output of the test driver. Flushing stdout/stderr
in the right spots avoids that.
Signed-off-by: Gerwin Klein <gerwin.klein@proofcraft.systems>
Add a folding group for verbose log output if running in a GitHub
context. GITHUB_REPOSITORY will be set for all GitHub contexts we're
interested in.
Signed-off-by: Gerwin Klein <gerwin.klein@proofcraft.systems>
PEP 394 expects that Python 3 installations provide a `python3` command,
but does not require a `python` command. Some distributions (including
Debian) are no longer providing a `python` command, but do provide
`python3`.
In this change, the `python3` interpreter is invoked via the existing
`#!` line in the `testspec.py` script.
Signed-off-by: Matthew Brecknell <Matthew.Brecknell@data61.csiro.au>
Python 2 has passed its sunset date, and many distributions are
withdrawing support for Python 2.
PEP 394 recommends distributions always install versioned interpreter
commands (e.g. `python3`), but does not make a recommendation about
whether or not an unversioned command (`python`) should exist, or what
version it should run.
It therefore seems advisable to explicitly run scripts using the
`python3` command, for scripts that are compatible with Python 3.
Here, we do this for Python scripts used by `run_tests`. For this to
work, some scripts have been updated in ways that will break Python 2
compatibility. But for some other scripts which were already compatible
with both Python 2 and 3, we have not yet removed Python 2
compatibility. There are also miscellaneous scripts that are not used by
`run_tests`, and these have not yet been updated to Python 3.
Signed-off-by: Matthew Brecknell <Matthew.Brecknell@data61.csiro.au>
These are being added to support builds that turn on extra debugging
or measurement options in Isabelle, which slow the builds down.
Might also be useful for running tests on very slow machines.
The run_tests script already has a -l/--list option to list all tests.
This adds a -L/--dry-run option to list the requested tests, which makes it
easier to figure out what tests -r/-x etc would select.
Also clarify in --help that -v adds more info to -l.
- timeout_output looks for isabelle relative to its file path
- run_tests looks for timeout_output relative to its file path
- output from timeout_output is utf8-decoded before attempting to
concatenate with other strings
Meta-regression should now pass.
The run_tests wrapper explicitly excludes any tests which are known to
fail on certain L4V_ARCH settings. The list of excluded tests includes
some tests which are only defined in private/internal repositories.
Recent changes to run_tests.py and testspec.py assumed that tests
excluded using `run_tests.py -r ...` would be defined in tests.xml
files. This caused regression to fail when performed with only
publicly-available repositories.
This change allows use of the `run_tests.py -r ...` option with test
names that are not defined in any tests.xml files.
tags: [NO_PROOF]
The watch_kill_switch loop was pretty busy, adding a simple
timeout reduces CPU consumption.
The CPU consumption of run_tests.py is still higher than I'd expect
to just update a terminal, but I don't know where to investigate
further.
At version 4.1.0, the Python psutil package changed the way it reports
CPU times for processes. This commit ensures that regression tests are
compatible with both old and new psutil APIs.
The run_tests.py script already searches the directory for all
test.xml files. It will now also note any files named extra_tests,
and also search any directories which appear as lines of those
files.
(Following symlinks would been more obvious but create other issues.)
It is sometimes desirable to overwrite the default set of tests that are
being run in a per execution basis (ex: to allow wrapper scripts to have
a custom default) RUN_TESTS_DEFAULT is an space separated list of tests
that will be run if no specific tests are given.
tags: [NO_PROOF]
Gerwin Klein
Gerwin Klein
Matthew Brecknell
Amirreza Zarrabi
Japheth Lim
Thomas Sewell
Rafal Kolanski
Alejandro Gomez-Londono
Alejandro Gomez-Londono
Thomas Sewell
Gerwin Klein
Japheth Lim