Commit Graph

201 Commits

Author SHA1 Message Date
Rafal Kolanski 623f216d94 regression: meta-regression improvements for timeout_output
- timeout_output looks for isabelle relative to its file path
- run_tests looks for timeout_output relative to its file path
- output from timeout_output is utf8-decoded before attempting to
  concatenate with other strings

Meta-regression should now pass.
2018-01-23 13:27:48 +11:00
Thomas Sewell 77837d16ac Fix for python3 in run_tests.py.
We require run_tests.py to work with both python2 and python3, which requires
particular syntax for exception handling.
2018-01-23 11:48:11 +11:00
Thomas Sewell 18cf880573 run_tests: Create and fetch timing/timeout info. 2018-01-22 12:04:08 +11:00
Matthew Brecknell 379fafe084 Isabelle2017: add session-qualified imports to gen_isabelle_root 2017-10-30 12:23:26 +11:00
Matthew Brecknell a5b544fd5b Isabelle2017: ignore certain session-qualified imports in check_theory_imports 2017-10-30 12:23:26 +11:00
Rafal Kolanski cc0f50790b Isabelle2017: update Isabelle/jEdit goto-error macro for RC0 2017-10-30 12:23:26 +11:00
Matthew Brecknell 184d6b70b7 remove most tab characters 2017-10-20 14:22:36 +11:00
Adrian Danis 49d6cf721b Munge test: Delete munge tests that have been moved to a new repository
tags: [NO_PROOF]
2017-09-28 15:21:11 +10:00
Matthew Brecknell 85a20c08a5 theory_imports: depend on c-kernel instead of CParser
The theory_imports regression test requires bitfield-generated theory
files. Previously, the theory_imports regression test depended on
CParser, and explicitly invoked "make" to ensure bitfield-generated
theories were present. However, these theories can also be generated by
the CKernel regression test. This meant that it was non-deterministic
whether bitfield-generated theories were generated during the
theory_imports regression test or the CKernel regression test.

This change adds a c-kernel regression test which generates the relevant
theories for the current L4V_ARCH, and makes both theory_imports and
CKernel depend on c-kernel. This ensures that those theories are always
generated during the c-kernel test, and should therefore make run_tests
timing results for the CKernel image more consistent.

Unfortunately, the check_theory_imports script does not have an easy way
to restrict itself to theories for the current L4V_ARCH, so the script
still needs to invoke "make c-kernel" for architectures other than the
current L4V_ARCH.
2017-09-12 14:47:24 +10:00
Matthew Brecknell ae91c097d4 regression: ignore undefined tests in `run_tests.py -r ...`
The run_tests wrapper explicitly excludes any tests which are known to
fail on certain L4V_ARCH settings. The list of excluded tests includes
some tests which are only defined in private/internal repositories.

Recent changes to run_tests.py and testspec.py assumed that tests
excluded using `run_tests.py -r ...` would be defined in tests.xml
files. This caused regression to fail when performed with only
publicly-available repositories.

This change allows use of the `run_tests.py -r ...` option with test
names that are not defined in any tests.xml files.

tags: [NO_PROOF]
2017-09-07 15:01:34 +10:00
Matthew Brecknell 43f95c8a09 regression: collect more dependency information 2017-08-21 17:49:45 +10:00
Matthew Brecknell e0e8a0a20b regression: refactor tests.xml parser 2017-08-21 17:49:45 +10:00
Matthew Brecknell c03548d9a8 regression: always parse tests.xml strictly 2017-08-21 17:15:34 +10:00
Matthew Brecknell 064c8b0718 regression: remove legacy run_tests options 2017-08-21 17:15:34 +10:00
Matthew Brecknell 2a2b3b9cbe regression: clean up testspec and run_tests.py 2017-08-21 17:15:34 +10:00
Joel Beeren f05bc45d59 misc: clean up before merging x64 2017-08-11 11:49:18 +10:00
Matthew Brecknell 752dd0794f x64 regression: add x64 to theory_imports check 2017-08-09 17:00:56 +10:00
Matthew Brecknell 740a74704a regression: ensure haskell-translator runs before theory_imports 2017-08-09 16:57:42 +10:00
Matthew Brecknell 238e8b307e x64: merge master 2017-07-21 11:27:12 +10:00
Alejandro Gomez-Londono 796887d9b1 Removes all trailing whitespaces 2017-07-12 15:13:51 +10:00
Alejandro Gomez-Londono 4671d2615a test: run_tests.py now prints a warning when excluding unknown tests (with [-x] [-r])
* This allows mentioning internal tests in exclusion lists without
    breaking external (github) builds

    tags: [NO_PROOF]
2017-06-26 15:21:06 +10:00
Alejandro Gomez-Londono 74500bd829 arm-hyp test: Adding exclusion test set for ARM_HYP + cleanup 2017-06-19 14:32:43 +10:00
Alejandro Gomez-Londono a9d7c199c7 test: check_theory_imports multi arch support 2017-06-19 14:32:32 +10:00
Alejandro Gomez-Londono ee71ad77a0 arm-hyp test: Fix race condition between isabelle and isabelle_ARM 2017-06-19 14:32:24 +10:00
Alejandro Gomez-Londono 4a76aec383 arm-hyp test: Fix race condition between ARM tests 2017-06-19 14:32:24 +10:00
Alejandro Gomez-Londono 0a31fa7fe4 Remove spec-check test and scripts 2017-05-12 12:50:55 +10:00
Joel Beeren ce2220286b misc: add whitespace removal tool
Also-by: Alejandro Gomez <Alejandro.Gomez@data61.csiro.au>
2017-03-29 18:07:21 +11:00
Daniel Matichuk b1f9defffa fix thydeps parser to handle % in keyword syntax 2017-02-17 14:29:39 +11:00
Matthew Brecknell f6d8575bdf Isabelle2016-1: update check_theory_imports with new Word_Lib dependency 2017-01-05 14:26:35 +11:00
Thomas Sewell 68e22b2d14 Slightly reduce run_tests.py CPU usage.
The watch_kill_switch loop was pretty busy, adding a simple
timeout reduces CPU consumption.

The CPU consumption of run_tests.py is still higher than I'd expect
to just update a terminal, but I don't know where to investigate
further.
2016-11-02 11:19:10 +11:00
Matthew Brecknell 0212696e60 VER-643: make regression compatible with psutil 4.1.0+
At version 4.1.0, the Python psutil package changed the way it reports
CPU times for processes. This commit ensures that regression tests are
compatible with both old and new psutil APIs.
2016-10-07 09:23:20 +11:00
Matthew Brecknell 569cd2822e thydeps: support Perl version 5.24
Perl version 5.24 no longer allows use of $_ as a lexical variable.
2016-10-04 12:04:36 +11:00
Alejandro Gomez-Londono 0c29567bb2 Regression: re-applying [094fb48623] to fix run_tests.py
The changes on [094fb48623] where (for some weird reason) removed from
run_tests.py in the last update, this commit merely re-apply those
changes.
2016-08-29 17:08:34 +10:00
Thomas Sewell 8b2818299a Read extra_tests as relative to dir it is in. 2016-08-26 16:59:27 +10:00
Thomas Sewell 37efb6326a Have run_tests see an extra_tests special file.
The run_tests.py script already searches the directory for all
test.xml files. It will now also note any files named extra_tests,
and also search any directories which appear as lines of those
files.

(Following symlinks would been more obvious but create other issues.)
2016-08-26 16:59:27 +10:00
Alejandro Gomez-Londono ef99749ee1 Regression: Added RUN_TESTS_DEFAULT for overwriting the default test set
It is sometimes desirable to overwrite the default set of tests that are
being run in a per execution basis (ex: to allow wrapper scripts to have
a custom default) RUN_TESTS_DEFAULT is an space separated list of tests
that will be run if no specific tests are given.

  tags: [NO_PROOF]
2016-08-24 13:23:33 +10:00
Alejandro Gomez-Londono e110f421d1 Munge test: updates test_munge.sh to support sorted output from c-parser
tags: [VER-624][NO_PROOF]
2016-08-17 13:59:28 +10:00
Alejandro Gomez-Londono fb9122f2ed Munge Test: Adding sorted munge output
tags: [VER-624][NO_PROOF]
2016-08-10 15:30:00 +10:00
Alejandro Gomez-Londono 7724dff916 test_munge.sh updated to print diff of symbols and kernel [VER-584] 2016-07-14 16:34:55 +10:00
Alejandro Gomez-Londono 93adccc141 license-tool: missing license headers + .licenseignore [VER-551] 2016-07-14 16:34:31 +10:00
Alejandro Gomez-Londono 89b64646ec test_munge.sh and make_munge.sh updated to support
the new --ast flag in c-parser [VER-584][VER-512][NO_PROOF]
2016-07-06 16:59:37 +10:00
Alejandro Gomez-Londono 495665e3c9 Munge make/test scripts 2016-06-21 14:50:04 +10:00
Japheth Lim d8c4ba1dec regression: move theory_imports after CSpec
While CKernel already builds structures_proofs.thy, CSpec's build system
unconditionally builds it again, so this patch removes a race condition
between theory_imports and CSpec.
2016-06-03 13:10:55 +10:00
Japheth Lim 3b54eb8f70 regression: ensure that theory_imports runs after theory generation 2016-05-27 18:02:21 +10:00
Japheth Lim 40570bc4fe regression: add test to check theory import paths
This commit also fixes all bad imports reported by the test script.
Jira issue VER-560
2016-05-27 16:17:13 +10:00
Japheth Lim 77c64e18af thydeps: silence perl warnings; handle nested comments 2016-05-27 16:17:13 +10:00
Japheth Lim 9199c9234a thydeps: support expanding variables in theory imports
See the updated documentation.
2016-05-27 16:17:13 +10:00
Japheth Lim 4fe15eac00 regression: use local Haskell kernel for spec-check 2016-05-24 14:52:51 +10:00
Gerwin Klein 445efb7c29 lib: closure for Word_Lib and own session 2016-05-16 21:11:40 +10:00
Alejandro Gomez-Londono 094fb48623 VER-525: run_tests handles dependencies incorrect (solved) 2016-05-10 14:15:32 +10:00
Japheth Lim 10d3a0d6fb regression: add Word to "isabelle" test so other tests don't try to build it in parallel. 2016-04-14 21:20:23 +10:00
Japheth Lim 874e1998e5 Fix up c-parser and autocorres for AutoCorres 1.2 release. 2016-03-30 17:48:27 +11:00
Japheth Lim ff270121c9 Isabelle zsh completion: handle no options (yxml); handle '=' in arg (jedit -D). 2016-03-16 14:53:12 +11:00
Japheth Lim 2060976f51 Isabelle zsh completion: allow multiple -d's; use proper path expansion. 2016-03-16 14:32:33 +11:00
Japheth Lim 4c89da8b0c regression: restore Python 3 compat. 2016-02-25 11:44:49 +11:00
Japheth Lim 66e94dcf76 regression: decode test output as UTF-8, not ASCII. 2016-02-19 12:00:03 +11:00
Japheth Lim b1ba5037c4 regression: compatibility fix for very old versions of bash (Mac OS). 2016-02-17 14:59:42 +11:00
Rafal Kolanski 4c18f6606b Merge pull request #17 in SEL4/l4v from ~JALIM/l4v:master to master
* commit '63f1a0854a58a9c39d64d7ca8482bf2dd65b68c5':
  regression: add --grace-period option to allow tests to clean up.
2016-02-17 03:56:25 +00:00
Japheth Lim 63f1a0854a regression: add --grace-period option to allow tests to clean up. 2016-02-17 11:19:13 +11:00
Gao Xin bee4ba0052 l4v-sabre: fix refine 2016-02-17 11:18:02 +11:00
Thomas Sewell 9c563f01ca L4V support for using skip cache.
Ignore the cache files. Also, add a flag which has run_tests.sh
build the skip cache.
2016-02-10 15:39:47 +11:00
Japheth Lim ec49042ec8 regression: fix bug in -x (from 5c776152b). 2016-02-05 12:16:21 +11:00
Rafal Kolanski 557418e8da jedit goto-error macro doc: installation path can be subdir
Since writing this, we have discovered that macros can live in arbitrary
hierarchies inside the macros folder, e.g. jedit/macros/Isabelle/, which
then show up in the macros menu. Very nice. Installation instructions
updated.
2016-02-05 09:49:42 +11:00
Japheth Lim 5c776152b3 regression: add -x option to exclude tests from running. 2016-02-04 15:22:39 +11:00
Japheth Lim 404b779e21 regression: add --fail-fast option; refactor stringly typed code. 2016-02-03 17:44:12 +11:00
Japheth Lim e409f357e9 trivial: regression: fix comment about floating-point discrepancy. 2016-02-03 14:22:01 +11:00
Japheth Lim 2f94189c7b regression: detect builds that get stuck without using CPU.
Such builds would have bypassed the cpu-timeout test settings
but can now be detected and killed.
2016-02-02 17:58:19 +11:00
Japheth Lim 253b04f6d9 regression: use CPU instead of real-time timeouts for all tests.
Also update and clarify test spec documentation.
2016-02-01 19:51:13 +11:00
Japheth Lim 0001e839bc regression: fix floating-point rounding that would fail the CPU monotonicity check. 2016-02-01 16:47:28 +11:00
Japheth Lim 25fc04eae0 regression: only print start-of-test messages if stdout is a tty.
This avoids clutter in non-tty outputs, such as regression emails.
2016-02-01 16:26:41 +11:00
Japheth Lim 625c15626d Merge pull request #9 in SEL4/l4v from regression-cputime to master
* commit '84e5f7d81403549f9d88fab3bf0da02b657f47bc':
  regression: change CPU time non-mono from an error to a warning, so that it doesn't kill an otherwise working build.
  style: regression: refactor use of print_test_line.
  regression: report CPU usage; support timeouts based on CPU usage.
2016-02-01 05:23:47 +00:00
Japheth Lim 84e5f7d814 regression: change CPU time non-mono from an error to a warning, so that it doesn't kill an otherwise working build. 2016-02-01 16:15:14 +11:00
Japheth Lim ac29ffc539 Merge pull request #7 in SEL4/l4v from ~CRICHARDSON/l4v:python3 to master
* commit 'af60787317365e05fe03df3d98a41fe9b8740b04':
  various scripts: use print_function uniformly for python3 compat
  regression: decode test output immediately
  regression: python3 compat, use byte literals where bytes are expected
  regression: python3 compat, Queue -> queue
2016-01-29 06:27:58 +00:00
Daniel Matichuk 8884a09a13 update isabelle settings to set L4V_ARCH to ARM by default 2016-01-29 13:46:26 +11:00
Japheth Lim 19b4b76ea8 style: regression: refactor use of print_test_line. 2016-01-29 11:53:52 +11:00
Corey Richardson af60787317
various scripts: use print_function uniformly for python3 compat 2016-01-28 16:11:07 +11:00
Daniel Matichuk 62e44db4df arch_split: fixed test_spec 2016-01-28 15:13:11 +11:00
Japheth Lim 197f48a25f regression: report CPU usage; support timeouts based on CPU usage. 2016-01-27 19:24:42 +11:00
Corey Richardson 5370e01b67
regression: decode test output immediately 2016-01-22 15:16:58 +11:00
Corey Richardson 8133f8a292
regression: python3 compat, use byte literals where bytes are expected 2016-01-21 19:00:06 +11:00
Corey Richardson e6d3bd31f8
regression: python3 compat, Queue -> queue 2016-01-21 18:59:30 +11:00
Daniel Matichuk c282969c54 Merge remote-tracking branch 'verification/master' into arch_split 2016-01-21 10:22:48 +11:00
Japheth Lim dcce50ce79 regression: make --junit-report output more Bamboo-friendly. 2016-01-14 15:55:43 +11:00
Japheth Lim ea38a3303d regression: get dependency tracking to work again. 2016-01-14 11:59:20 +11:00
Japheth Lim 2c7a36e4f1 regression: fix a bug that prevented the printing of failure logs. 2016-01-14 11:27:08 +11:00
Japheth Lim 5892043915 regression: add option to produce JUnit-style report. For Bamboo usage. 2016-01-13 18:01:23 +11:00
Japheth Lim 0afc807807 regression: remove assertion because it may go off spuriously for skipped tests. 2016-01-13 17:10:31 +11:00
Japheth Lim aab9c38901 regression: prefer explicit isabelle path over using PATH. 2016-01-13 16:59:25 +11:00
Japheth Lim 65e98199e1 regression: adjust unnecessarily large test timeouts.
Some tests had timeouts of up to 4 hours. Note that timeouts are
applied on a per-test basis, not per-testsuite. This is now clarified
in the tests.xml documentation.
2016-01-13 16:59:25 +11:00
Japheth Lim e66cba5466 regression: handle dependencies correctly for manually specified tests. 2016-01-13 16:59:25 +11:00
Daniel Matichuk fad2c6aae9 paramatrised abstract and haskell specs over L4V_ARCH
Haskell translator was modified to support multiple translations
of the haskell, with different build parameters.
2016-01-13 12:01:40 +11:00
Japheth Lim cf1dc6852f regression: don't try to kill pids that have already finished.
Reduces errors due to pid reuse, such as in this regression build
  Regression SUCCESS (2015-12-11 04:55:36)
2016-01-11 15:34:36 +11:00
Japheth Lim 46afb6d0fd regression: fix status line behaviour to match docs. 2016-01-08 17:06:54 +11:00
Matthew Fernandez 81fe5cbd67 misc/pysymbols: Merge some CAmkES functionality for generating apply-style.
This commit resurrects some bitrotted support for generating apply-style proofs
from Python. It is somewhat incomplete and untested, but can serve to make
template proofs much more readable.
2016-01-08 15:47:14 +11:00
Matthew Fernandez de77df45aa misc/pysymbols: Move IsaSymbolsException to a separate file. 2016-01-08 15:47:04 +11:00
Japheth Lim 71143907be regression: Python style fixes (credit: Matt F). 2016-01-07 20:41:45 +11:00
Japheth Lim 26eb9eba44 trivial: regression: revert unintended change in timeout handling. 2016-01-07 17:56:18 +11:00
Japheth Lim 2cdbadca87 regression: add status line for ttys; fix timeout and Python var scoping. 2016-01-07 17:49:22 +11:00
Japheth Lim 7d24c4a5b5 regression: parallelise run_tests.py. New options: -j, --legacy-status. 2016-01-07 17:22:00 +11:00
Gerwin Klein da5f50aa94 Merge pull request #3 in SEL4/l4v from ~MFERNANDEZ/l4v:23e130d5-6f13-4fa3-ae00-e1c8174f2cb9 to master
* commit '56dfa8cca7981055b50969cdf787f2557e827233':
  misc: Add vim syntax files.
2016-01-06 04:03:13 +00:00