These are being added to support builds that turn on extra debugging
or measurement options in Isabelle, which slow the builds down.
Might also be useful for running tests on very slow machines.
When updating to Isabelle 2017 the non-error case was neglected and gave
a stack dump rather than a sane message. This has now been addressed.
Apparently the Beanshell scripting environment in jEdit prior to 5.5.0
accepted '==' as string comparison. In 5.5.0 it is no longer accepted,
and requires .equals() as in normal Java.
The run_tests script already has a -l/--list option to list all tests.
This adds a -L/--dry-run option to list the requested tests, which makes it
easier to figure out what tests -r/-x etc would select.
Also clarify in --help that -v adds more info to -l.
- timeout_output looks for isabelle relative to its file path
- run_tests looks for timeout_output relative to its file path
- output from timeout_output is utf8-decoded before attempting to
concatenate with other strings
Meta-regression should now pass.
The theory_imports regression test requires bitfield-generated theory
files. Previously, the theory_imports regression test depended on
CParser, and explicitly invoked "make" to ensure bitfield-generated
theories were present. However, these theories can also be generated by
the CKernel regression test. This meant that it was non-deterministic
whether bitfield-generated theories were generated during the
theory_imports regression test or the CKernel regression test.
This change adds a c-kernel regression test which generates the relevant
theories for the current L4V_ARCH, and makes both theory_imports and
CKernel depend on c-kernel. This ensures that those theories are always
generated during the c-kernel test, and should therefore make run_tests
timing results for the CKernel image more consistent.
Unfortunately, the check_theory_imports script does not have an easy way
to restrict itself to theories for the current L4V_ARCH, so the script
still needs to invoke "make c-kernel" for architectures other than the
current L4V_ARCH.
The run_tests wrapper explicitly excludes any tests which are known to
fail on certain L4V_ARCH settings. The list of excluded tests includes
some tests which are only defined in private/internal repositories.
Recent changes to run_tests.py and testspec.py assumed that tests
excluded using `run_tests.py -r ...` would be defined in tests.xml
files. This caused regression to fail when performed with only
publicly-available repositories.
This change allows use of the `run_tests.py -r ...` option with test
names that are not defined in any tests.xml files.
tags: [NO_PROOF]
The watch_kill_switch loop was pretty busy, adding a simple
timeout reduces CPU consumption.
The CPU consumption of run_tests.py is still higher than I'd expect
to just update a terminal, but I don't know where to investigate
further.
At version 4.1.0, the Python psutil package changed the way it reports
CPU times for processes. This commit ensures that regression tests are
compatible with both old and new psutil APIs.
The run_tests.py script already searches the directory for all
test.xml files. It will now also note any files named extra_tests,
and also search any directories which appear as lines of those
files.
(Following symlinks would been more obvious but create other issues.)
It is sometimes desirable to overwrite the default set of tests that are
being run in a per execution basis (ex: to allow wrapper scripts to have
a custom default) RUN_TESTS_DEFAULT is an space separated list of tests
that will be run if no specific tests are given.
tags: [NO_PROOF]
Gerwin Klein
Japheth Lim
Rafal Kolanski
Thomas Sewell
Matthew Brecknell
Adrian Danis
Joel Beeren
Alejandro Gomez-Londono
Alejandro Gomez-Londono
Joel Beeren
Daniel Matichuk
Thomas Sewell