This adds a small script that parses two run_tests logs for session
times and compares them.
Signed-off-by: Gerwin Klein <gerwin.klein@proofcraft.systems>
The AWS test runner does not have a GitHub ssh key, so can only
access repositories via https.
Signed-off-by: Gerwin Klein <gerwin.klein@proofcraft.systems>
Useful for copy/paste into PR comments to link to the testboard
results, or just to see them manually.
Signed-off-by: Gerwin Klein <gerwin.klein@proofcraft.systems>
Mostly for testing, to inspect what manifest is being constructed
without triggering tests.
Signed-off-by: Gerwin Klein <gerwin.klein@proofcraft.systems>
Use '> ' instead of '[log] ', since this is not a log file but
usually read interactively.
Signed-off-by: Gerwin Klein <gerwin.klein@proofcraft.systems>
The test runner does not have ssh access to GitHub, so can't use
ssh:// URLs. This means the tested revisions need to be public, but
since this is meant for foundation use, that should be fine.
(The testboard itself is public anway)
Signed-off-by: Gerwin Klein <gerwin.klein@proofcraft.systems>
On GitHub, the output of external processes such as isabelle overtake
the stdout/stderr output of the test driver. Flushing stdout/stderr
in the right spots avoids that.
Signed-off-by: Gerwin Klein <gerwin.klein@proofcraft.systems>
Add a folding group for verbose log output if running in a GitHub
context. GITHUB_REPOSITORY will be set for all GitHub contexts we're
interested in.
Signed-off-by: Gerwin Klein <gerwin.klein@proofcraft.systems>
search-replace.sh is a very simple script which takes a list of text
replacements and applies those replacements in all files in the current
directory. The README file contains more detailed information.
Signed-off-by: Mitchell Buckley <mitchell.alan.buckley@gmail.com>
PEP 394 expects that Python 3 installations provide a `python3` command,
but does not require a `python` command. Some distributions (including
Debian) are no longer providing a `python` command, but do provide
`python3`.
In this change, the `python3` interpreter is invoked via the existing
`#!` line in the `testspec.py` script.
Signed-off-by: Matthew Brecknell <Matthew.Brecknell@data61.csiro.au>
Python 2 has passed its sunset date, and many distributions are
withdrawing support for Python 2.
PEP 394 recommends distributions always install versioned interpreter
commands (e.g. `python3`), but does not make a recommendation about
whether or not an unversioned command (`python`) should exist, or what
version it should run.
It therefore seems advisable to explicitly run scripts using the
`python3` command, for scripts that are compatible with Python 3.
Here, we do this for Python scripts used by `run_tests`. For this to
work, some scripts have been updated in ways that will break Python 2
compatibility. But for some other scripts which were already compatible
with both Python 2 and 3, we have not yet removed Python 2
compatibility. There are also miscellaneous scripts that are not used by
`run_tests`, and these have not yet been updated to Python 3.
Signed-off-by: Matthew Brecknell <Matthew.Brecknell@data61.csiro.au>
Changes:
- Isabelle2017 compat for output heaps is now removed.
- We no longer select x86_64 as the ML_PLATFORM; the new default
x86_64_32 platform is generally superior.
- We still select ML_PLATFORM for Isabelle2018 compat.
ML_HOME is not always set already when this script is sourced (e.g. when
the polyml component is somehow faulty).
Isabelle heap output settings are now part of the TS isabelle patch queue;
explicit override is still possible as before.
These are being added to support builds that turn on extra debugging
or measurement options in Isabelle, which slow the builds down.
Might also be useful for running tests on very slow machines.
When updating to Isabelle 2017 the non-error case was neglected and gave
a stack dump rather than a sane message. This has now been addressed.
Apparently the Beanshell scripting environment in jEdit prior to 5.5.0
accepted '==' as string comparison. In 5.5.0 it is no longer accepted,
and requires .equals() as in normal Java.
The run_tests script already has a -l/--list option to list all tests.
This adds a -L/--dry-run option to list the requested tests, which makes it
easier to figure out what tests -r/-x etc would select.
Also clarify in --help that -v adds more info to -l.
- timeout_output looks for isabelle relative to its file path
- run_tests looks for timeout_output relative to its file path
- output from timeout_output is utf8-decoded before attempting to
concatenate with other strings
Meta-regression should now pass.