Rafal Kolanski
623f216d94
regression: meta-regression improvements for timeout_output
...
- timeout_output looks for isabelle relative to its file path
- run_tests looks for timeout_output relative to its file path
- output from timeout_output is utf8-decoded before attempting to
concatenate with other strings
Meta-regression should now pass.
2018-01-23 13:27:48 +11:00
Thomas Sewell
77837d16ac
Fix for python3 in run_tests.py.
...
We require run_tests.py to work with both python2 and python3, which requires
particular syntax for exception handling.
2018-01-23 11:48:11 +11:00
Thomas Sewell
18cf880573
run_tests: Create and fetch timing/timeout info.
2018-01-22 12:04:08 +11:00
Matthew Brecknell
379fafe084
Isabelle2017: add session-qualified imports to gen_isabelle_root
2017-10-30 12:23:26 +11:00
Matthew Brecknell
a5b544fd5b
Isabelle2017: ignore certain session-qualified imports in check_theory_imports
2017-10-30 12:23:26 +11:00
Rafal Kolanski
cc0f50790b
Isabelle2017: update Isabelle/jEdit goto-error macro for RC0
2017-10-30 12:23:26 +11:00
Matthew Brecknell
184d6b70b7
remove most tab characters
2017-10-20 14:22:36 +11:00
Adrian Danis
49d6cf721b
Munge test: Delete munge tests that have been moved to a new repository
...
tags: [NO_PROOF]
2017-09-28 15:21:11 +10:00
Matthew Brecknell
85a20c08a5
theory_imports: depend on c-kernel instead of CParser
...
The theory_imports regression test requires bitfield-generated theory
files. Previously, the theory_imports regression test depended on
CParser, and explicitly invoked "make" to ensure bitfield-generated
theories were present. However, these theories can also be generated by
the CKernel regression test. This meant that it was non-deterministic
whether bitfield-generated theories were generated during the
theory_imports regression test or the CKernel regression test.
This change adds a c-kernel regression test which generates the relevant
theories for the current L4V_ARCH, and makes both theory_imports and
CKernel depend on c-kernel. This ensures that those theories are always
generated during the c-kernel test, and should therefore make run_tests
timing results for the CKernel image more consistent.
Unfortunately, the check_theory_imports script does not have an easy way
to restrict itself to theories for the current L4V_ARCH, so the script
still needs to invoke "make c-kernel" for architectures other than the
current L4V_ARCH.
2017-09-12 14:47:24 +10:00
Matthew Brecknell
ae91c097d4
regression: ignore undefined tests in `run_tests.py -r ...`
...
The run_tests wrapper explicitly excludes any tests which are known to
fail on certain L4V_ARCH settings. The list of excluded tests includes
some tests which are only defined in private/internal repositories.
Recent changes to run_tests.py and testspec.py assumed that tests
excluded using `run_tests.py -r ...` would be defined in tests.xml
files. This caused regression to fail when performed with only
publicly-available repositories.
This change allows use of the `run_tests.py -r ...` option with test
names that are not defined in any tests.xml files.
tags: [NO_PROOF]
2017-09-07 15:01:34 +10:00
Matthew Brecknell
43f95c8a09
regression: collect more dependency information
2017-08-21 17:49:45 +10:00
Matthew Brecknell
e0e8a0a20b
regression: refactor tests.xml parser
2017-08-21 17:49:45 +10:00
Matthew Brecknell
c03548d9a8
regression: always parse tests.xml strictly
2017-08-21 17:15:34 +10:00
Matthew Brecknell
064c8b0718
regression: remove legacy run_tests options
2017-08-21 17:15:34 +10:00
Matthew Brecknell
2a2b3b9cbe
regression: clean up testspec and run_tests.py
2017-08-21 17:15:34 +10:00
Joel Beeren
f05bc45d59
misc: clean up before merging x64
2017-08-11 11:49:18 +10:00
Matthew Brecknell
752dd0794f
x64 regression: add x64 to theory_imports check
2017-08-09 17:00:56 +10:00
Matthew Brecknell
740a74704a
regression: ensure haskell-translator runs before theory_imports
2017-08-09 16:57:42 +10:00
Matthew Brecknell
238e8b307e
x64: merge master
2017-07-21 11:27:12 +10:00
Alejandro Gomez-Londono
796887d9b1
Removes all trailing whitespaces
2017-07-12 15:13:51 +10:00
Alejandro Gomez-Londono
4671d2615a
test: run_tests.py now prints a warning when excluding unknown tests (with [-x] [-r])
...
* This allows mentioning internal tests in exclusion lists without
breaking external (github) builds
tags: [NO_PROOF]
2017-06-26 15:21:06 +10:00
Alejandro Gomez-Londono
74500bd829
arm-hyp test: Adding exclusion test set for ARM_HYP + cleanup
2017-06-19 14:32:43 +10:00
Alejandro Gomez-Londono
a9d7c199c7
test: check_theory_imports multi arch support
2017-06-19 14:32:32 +10:00
Alejandro Gomez-Londono
ee71ad77a0
arm-hyp test: Fix race condition between isabelle and isabelle_ARM
2017-06-19 14:32:24 +10:00
Alejandro Gomez-Londono
4a76aec383
arm-hyp test: Fix race condition between ARM tests
2017-06-19 14:32:24 +10:00
Alejandro Gomez-Londono
0a31fa7fe4
Remove spec-check test and scripts
2017-05-12 12:50:55 +10:00
Joel Beeren
ce2220286b
misc: add whitespace removal tool
...
Also-by: Alejandro Gomez <Alejandro.Gomez@data61.csiro.au>
2017-03-29 18:07:21 +11:00
Daniel Matichuk
b1f9defffa
fix thydeps parser to handle % in keyword syntax
2017-02-17 14:29:39 +11:00
Matthew Brecknell
f6d8575bdf
Isabelle2016-1: update check_theory_imports with new Word_Lib dependency
2017-01-05 14:26:35 +11:00
Thomas Sewell
68e22b2d14
Slightly reduce run_tests.py CPU usage.
...
The watch_kill_switch loop was pretty busy, adding a simple
timeout reduces CPU consumption.
The CPU consumption of run_tests.py is still higher than I'd expect
to just update a terminal, but I don't know where to investigate
further.
2016-11-02 11:19:10 +11:00
Matthew Brecknell
0212696e60
VER-643: make regression compatible with psutil 4.1.0+
...
At version 4.1.0, the Python psutil package changed the way it reports
CPU times for processes. This commit ensures that regression tests are
compatible with both old and new psutil APIs.
2016-10-07 09:23:20 +11:00
Matthew Brecknell
569cd2822e
thydeps: support Perl version 5.24
...
Perl version 5.24 no longer allows use of $_ as a lexical variable.
2016-10-04 12:04:36 +11:00
Alejandro Gomez-Londono
0c29567bb2
Regression: re-applying [ 094fb48623
] to fix run_tests.py
...
The changes on [094fb48623
] where (for some weird reason) removed from
run_tests.py in the last update, this commit merely re-apply those
changes.
2016-08-29 17:08:34 +10:00
Thomas Sewell
8b2818299a
Read extra_tests as relative to dir it is in.
2016-08-26 16:59:27 +10:00
Thomas Sewell
37efb6326a
Have run_tests see an extra_tests special file.
...
The run_tests.py script already searches the directory for all
test.xml files. It will now also note any files named extra_tests,
and also search any directories which appear as lines of those
files.
(Following symlinks would been more obvious but create other issues.)
2016-08-26 16:59:27 +10:00
Alejandro Gomez-Londono
ef99749ee1
Regression: Added RUN_TESTS_DEFAULT for overwriting the default test set
...
It is sometimes desirable to overwrite the default set of tests that are
being run in a per execution basis (ex: to allow wrapper scripts to have
a custom default) RUN_TESTS_DEFAULT is an space separated list of tests
that will be run if no specific tests are given.
tags: [NO_PROOF]
2016-08-24 13:23:33 +10:00
Alejandro Gomez-Londono
e110f421d1
Munge test: updates test_munge.sh to support sorted output from c-parser
...
tags: [VER-624][NO_PROOF]
2016-08-17 13:59:28 +10:00
Alejandro Gomez-Londono
fb9122f2ed
Munge Test: Adding sorted munge output
...
tags: [VER-624][NO_PROOF]
2016-08-10 15:30:00 +10:00
Alejandro Gomez-Londono
7724dff916
test_munge.sh updated to print diff of symbols and kernel [VER-584]
2016-07-14 16:34:55 +10:00
Alejandro Gomez-Londono
93adccc141
license-tool: missing license headers + .licenseignore [VER-551]
2016-07-14 16:34:31 +10:00
Alejandro Gomez-Londono
89b64646ec
test_munge.sh and make_munge.sh updated to support
...
the new --ast flag in c-parser [VER-584][VER-512][NO_PROOF]
2016-07-06 16:59:37 +10:00
Alejandro Gomez-Londono
495665e3c9
Munge make/test scripts
2016-06-21 14:50:04 +10:00
Japheth Lim
d8c4ba1dec
regression: move theory_imports after CSpec
...
While CKernel already builds structures_proofs.thy, CSpec's build system
unconditionally builds it again, so this patch removes a race condition
between theory_imports and CSpec.
2016-06-03 13:10:55 +10:00
Japheth Lim
3b54eb8f70
regression: ensure that theory_imports runs after theory generation
2016-05-27 18:02:21 +10:00
Japheth Lim
40570bc4fe
regression: add test to check theory import paths
...
This commit also fixes all bad imports reported by the test script.
Jira issue VER-560
2016-05-27 16:17:13 +10:00
Japheth Lim
77c64e18af
thydeps: silence perl warnings; handle nested comments
2016-05-27 16:17:13 +10:00
Japheth Lim
9199c9234a
thydeps: support expanding variables in theory imports
...
See the updated documentation.
2016-05-27 16:17:13 +10:00
Japheth Lim
4fe15eac00
regression: use local Haskell kernel for spec-check
2016-05-24 14:52:51 +10:00
Gerwin Klein
445efb7c29
lib: closure for Word_Lib and own session
2016-05-16 21:11:40 +10:00
Alejandro Gomez-Londono
094fb48623
VER-525: run_tests handles dependencies incorrect (solved)
2016-05-10 14:15:32 +10:00
Japheth Lim
10d3a0d6fb
regression: add Word to "isabelle" test so other tests don't try to build it in parallel.
2016-04-14 21:20:23 +10:00
Japheth Lim
874e1998e5
Fix up c-parser and autocorres for AutoCorres 1.2 release.
2016-03-30 17:48:27 +11:00
Japheth Lim
ff270121c9
Isabelle zsh completion: handle no options (yxml); handle '=' in arg (jedit -D).
2016-03-16 14:53:12 +11:00
Japheth Lim
2060976f51
Isabelle zsh completion: allow multiple -d's; use proper path expansion.
2016-03-16 14:32:33 +11:00
Japheth Lim
4c89da8b0c
regression: restore Python 3 compat.
2016-02-25 11:44:49 +11:00
Japheth Lim
66e94dcf76
regression: decode test output as UTF-8, not ASCII.
2016-02-19 12:00:03 +11:00
Japheth Lim
b1ba5037c4
regression: compatibility fix for very old versions of bash (Mac OS).
2016-02-17 14:59:42 +11:00
Rafal Kolanski
4c18f6606b
Merge pull request #17 in SEL4/l4v from ~JALIM/l4v:master to master
...
* commit '63f1a0854a58a9c39d64d7ca8482bf2dd65b68c5':
regression: add --grace-period option to allow tests to clean up.
2016-02-17 03:56:25 +00:00
Japheth Lim
63f1a0854a
regression: add --grace-period option to allow tests to clean up.
2016-02-17 11:19:13 +11:00
Gao Xin
bee4ba0052
l4v-sabre: fix refine
2016-02-17 11:18:02 +11:00
Thomas Sewell
9c563f01ca
L4V support for using skip cache.
...
Ignore the cache files. Also, add a flag which has run_tests.sh
build the skip cache.
2016-02-10 15:39:47 +11:00
Japheth Lim
ec49042ec8
regression: fix bug in -x (from 5c776152b
).
2016-02-05 12:16:21 +11:00
Rafal Kolanski
557418e8da
jedit goto-error macro doc: installation path can be subdir
...
Since writing this, we have discovered that macros can live in arbitrary
hierarchies inside the macros folder, e.g. jedit/macros/Isabelle/, which
then show up in the macros menu. Very nice. Installation instructions
updated.
2016-02-05 09:49:42 +11:00
Japheth Lim
5c776152b3
regression: add -x option to exclude tests from running.
2016-02-04 15:22:39 +11:00
Japheth Lim
404b779e21
regression: add --fail-fast option; refactor stringly typed code.
2016-02-03 17:44:12 +11:00
Japheth Lim
e409f357e9
trivial: regression: fix comment about floating-point discrepancy.
2016-02-03 14:22:01 +11:00
Japheth Lim
2f94189c7b
regression: detect builds that get stuck without using CPU.
...
Such builds would have bypassed the cpu-timeout test settings
but can now be detected and killed.
2016-02-02 17:58:19 +11:00
Japheth Lim
253b04f6d9
regression: use CPU instead of real-time timeouts for all tests.
...
Also update and clarify test spec documentation.
2016-02-01 19:51:13 +11:00
Japheth Lim
0001e839bc
regression: fix floating-point rounding that would fail the CPU monotonicity check.
2016-02-01 16:47:28 +11:00
Japheth Lim
25fc04eae0
regression: only print start-of-test messages if stdout is a tty.
...
This avoids clutter in non-tty outputs, such as regression emails.
2016-02-01 16:26:41 +11:00
Japheth Lim
625c15626d
Merge pull request #9 in SEL4/l4v from regression-cputime to master
...
* commit '84e5f7d81403549f9d88fab3bf0da02b657f47bc':
regression: change CPU time non-mono from an error to a warning, so that it doesn't kill an otherwise working build.
style: regression: refactor use of print_test_line.
regression: report CPU usage; support timeouts based on CPU usage.
2016-02-01 05:23:47 +00:00
Japheth Lim
84e5f7d814
regression: change CPU time non-mono from an error to a warning, so that it doesn't kill an otherwise working build.
2016-02-01 16:15:14 +11:00
Japheth Lim
ac29ffc539
Merge pull request #7 in SEL4/l4v from ~CRICHARDSON/l4v:python3 to master
...
* commit 'af60787317365e05fe03df3d98a41fe9b8740b04':
various scripts: use print_function uniformly for python3 compat
regression: decode test output immediately
regression: python3 compat, use byte literals where bytes are expected
regression: python3 compat, Queue -> queue
2016-01-29 06:27:58 +00:00
Daniel Matichuk
8884a09a13
update isabelle settings to set L4V_ARCH to ARM by default
2016-01-29 13:46:26 +11:00
Japheth Lim
19b4b76ea8
style: regression: refactor use of print_test_line.
2016-01-29 11:53:52 +11:00
Corey Richardson
af60787317
various scripts: use print_function uniformly for python3 compat
2016-01-28 16:11:07 +11:00
Daniel Matichuk
62e44db4df
arch_split: fixed test_spec
2016-01-28 15:13:11 +11:00
Japheth Lim
197f48a25f
regression: report CPU usage; support timeouts based on CPU usage.
2016-01-27 19:24:42 +11:00
Corey Richardson
5370e01b67
regression: decode test output immediately
2016-01-22 15:16:58 +11:00
Corey Richardson
8133f8a292
regression: python3 compat, use byte literals where bytes are expected
2016-01-21 19:00:06 +11:00
Corey Richardson
e6d3bd31f8
regression: python3 compat, Queue -> queue
2016-01-21 18:59:30 +11:00
Daniel Matichuk
c282969c54
Merge remote-tracking branch 'verification/master' into arch_split
2016-01-21 10:22:48 +11:00
Japheth Lim
dcce50ce79
regression: make --junit-report output more Bamboo-friendly.
2016-01-14 15:55:43 +11:00
Japheth Lim
ea38a3303d
regression: get dependency tracking to work again.
2016-01-14 11:59:20 +11:00
Japheth Lim
2c7a36e4f1
regression: fix a bug that prevented the printing of failure logs.
2016-01-14 11:27:08 +11:00
Japheth Lim
5892043915
regression: add option to produce JUnit-style report. For Bamboo usage.
2016-01-13 18:01:23 +11:00
Japheth Lim
0afc807807
regression: remove assertion because it may go off spuriously for skipped tests.
2016-01-13 17:10:31 +11:00
Japheth Lim
aab9c38901
regression: prefer explicit isabelle path over using PATH.
2016-01-13 16:59:25 +11:00
Japheth Lim
65e98199e1
regression: adjust unnecessarily large test timeouts.
...
Some tests had timeouts of up to 4 hours. Note that timeouts are
applied on a per-test basis, not per-testsuite. This is now clarified
in the tests.xml documentation.
2016-01-13 16:59:25 +11:00
Japheth Lim
e66cba5466
regression: handle dependencies correctly for manually specified tests.
2016-01-13 16:59:25 +11:00
Daniel Matichuk
fad2c6aae9
paramatrised abstract and haskell specs over L4V_ARCH
...
Haskell translator was modified to support multiple translations
of the haskell, with different build parameters.
2016-01-13 12:01:40 +11:00
Japheth Lim
cf1dc6852f
regression: don't try to kill pids that have already finished.
...
Reduces errors due to pid reuse, such as in this regression build
Regression SUCCESS (2015-12-11 04:55:36)
2016-01-11 15:34:36 +11:00
Japheth Lim
46afb6d0fd
regression: fix status line behaviour to match docs.
2016-01-08 17:06:54 +11:00
Matthew Fernandez
81fe5cbd67
misc/pysymbols: Merge some CAmkES functionality for generating apply-style.
...
This commit resurrects some bitrotted support for generating apply-style proofs
from Python. It is somewhat incomplete and untested, but can serve to make
template proofs much more readable.
2016-01-08 15:47:14 +11:00
Matthew Fernandez
de77df45aa
misc/pysymbols: Move IsaSymbolsException to a separate file.
2016-01-08 15:47:04 +11:00
Japheth Lim
71143907be
regression: Python style fixes (credit: Matt F).
2016-01-07 20:41:45 +11:00
Japheth Lim
26eb9eba44
trivial: regression: revert unintended change in timeout handling.
2016-01-07 17:56:18 +11:00
Japheth Lim
2cdbadca87
regression: add status line for ttys; fix timeout and Python var scoping.
2016-01-07 17:49:22 +11:00
Japheth Lim
7d24c4a5b5
regression: parallelise run_tests.py. New options: -j, --legacy-status.
2016-01-07 17:22:00 +11:00
Gerwin Klein
da5f50aa94
Merge pull request #3 in SEL4/l4v from ~MFERNANDEZ/l4v:23e130d5-6f13-4fa3-ae00-e1c8174f2cb9 to master
...
* commit '56dfa8cca7981055b50969cdf787f2557e827233':
misc: Add vim syntax files.
2016-01-06 04:03:13 +00:00