Michael Herzberg
|
ac3c1c7f20
|
Removed plain multiprocessing option.
|
2018-04-21 17:25:22 +01:00 |
|
Michael Herzberg
|
87b2847c6e
|
Make ProcessPool and pystuck the default (for now).
|
2018-04-11 15:39:23 +01:00 |
 Achim D. Brucker
|
0c70b2e20b
|
Increase number of parallel downloads.
|
2018-04-08 22:45:56 +01:00 |
|
Achim D. Brucker
|
7f71a40ff4
|
Configured number of parallel processes.
|
2018-04-07 21:14:36 +01:00 |
|
Achim D. Brucker
|
2fc154d643
|
Use UTC-based time/dates for logging.
|
2018-04-07 15:54:39 +01:00 |
|
Achim D. Brucker
|
6a86b37e7c
|
Increase number of parallel downloads.
|
2018-04-06 21:36:11 +01:00 |
|
Achim D. Brucker
|
69f1618db2
|
Reduced number of parallel downloads, as pebble seems to be much more memory hungry ...
|
2018-04-06 13:34:36 +01:00 |
|
Achim D. Brucker
|
7c688afee8
|
Configured 32 parallel downloads.
|
2018-04-03 22:14:02 +01:00 |
|
Achim D. Brucker
|
c05ad75bca
|
Minor increasement of parallel downloads (testing).
|
2018-04-02 11:36:22 +01:00 |
|
Achim D. Brucker
|
7fdadf5885
|
Reduced number of concurrent downloads.
|
2018-03-27 13:55:50 +01:00 |
|
Achim D. Brucker
|
fd194d0103
|
Increased number of parallel downloads.
|
2018-02-12 08:48:28 +00:00 |
|
Achim D. Brucker
|
1c8694dcb8
|
Bind /srv/.
|
2017-11-02 19:18:53 +00:00 |
|
Achim D. Brucker
|
9bc9889767
|
Fixed location of singularity image.
|
2017-11-02 19:09:29 +00:00 |
|
Achim D. Brucker
|
be7f9919ad
|
Use singularity for extension update.
|
2017-11-02 18:50:24 +00:00 |
|
Achim D. Brucker
|
b3240c48ef
|
Bug fix: error reporting.
|
2017-10-12 00:05:21 +01:00 |
|
Achim D. Brucker
|
c910e68575
|
Bug fix: counting of ERRORs.
|
2017-10-10 23:07:48 +01:00 |
|
Achim D. Brucker
|
2f7fcd005a
|
Increased number of parallel runs.
|
2017-10-10 17:19:27 +01:00 |
|
Achim D. Brucker
|
aa143b2724
|
Sort ERRORs by time stamp.
|
2017-10-09 11:50:11 +01:00 |
|
Achim D. Brucker
|
88b8a1791c
|
Added summary reporting.
|
2017-10-09 08:45:06 +01:00 |
|
Achim D. Brucker
|
cd3eef31d8
|
Limit parallel download to 24 and report ERRORs after download.
|
2017-10-07 19:31:43 +01:00 |
|
Achim D. Brucker
|
08b70ed63a
|
Updated archive dir to reflect new file hierarchy by default.
|
2017-09-08 21:10:40 +01:00 |
 Michael Herzberg
|
cbd2dea820
|
Removed everything related to sqlite and updated README.
|
2017-08-30 15:38:04 +01:00 |
|
Achim D. Brucker
|
5af2cff2d5
|
Updated cron example.
|
2017-08-10 09:05:40 +01:00 |
|
Achim D. Brucker
|
d8bd7af2dd
|
Refactoring: Generate databases from backup.
|
2017-08-09 20:27:49 +01:00 |
|
Michael Herzberg
|
44756e44d6
|
Added python-based sqlite merger.
|
2017-07-28 16:47:25 +01:00 |
|
Achim D. Brucker
|
9b291d02ae
|
Added local bin/lib-directory to PATH.
|
2017-07-24 10:47:42 +01:00 |
|
Achim D. Brucker
|
d6d1d21d05
|
Bug fix: Append log to log instead of overwritting it.
|
2017-07-24 10:26:03 +01:00 |
|
Achim D. Brucker
|
052748af07
|
Create log file for profiling.
|
2017-07-20 07:51:30 +01:00 |
|
Achim D. Brucker
|
c73f4cda3e
|
Fixed log output.
|
2017-07-19 07:01:38 +01:00 |
|
Achim D. Brucker
|
3346b1b61a
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-07-16 14:02:40 +01:00 |
|
Achim D. Brucker
|
049db03021
|
Add user local to binary and library search paths.
|
2017-07-16 14:02:22 +01:00 |
|
Michael Herzberg
|
5e60715d98
|
Removed schema creation from scripts as it is now being done by the merge script.
|
2017-07-13 14:47:42 +01:00 |
|
Achim D. Brucker
|
02ce0cfd2c
|
Print info: sqlite3 binary used.
|
2017-07-12 20:36:29 +01:00 |
|
Achim D. Brucker
|
817cf11ca7
|
Update script via cron.
|
2017-07-02 17:35:51 +01:00 |
|
Achim D. Brucker
|
c7a06dcf8e
|
Remove old compressed sqlite file prior to compressing updated sqlite file.
|
2017-07-02 17:31:48 +01:00 |
|
Achim D. Brucker
|
cc80684630
|
Disabled compressing of log files due to raise condition problems on network mounted drive.
|
2017-07-02 17:30:09 +01:00 |
|
Achim D. Brucker
|
b73ed18b67
|
Ignore errors while compressing log files.
|
2017-07-01 20:09:00 +01:00 |
|
Achim D. Brucker
|
797328b855
|
Overwrite compressed files.
|
2017-07-01 19:57:29 +01:00 |
|
Achim D. Brucker
|
5702a43368
|
Use parallel bzip2 (pbzip2) instead of regular bzip2.
|
2017-06-30 20:54:08 +01:00 |
|
Achim D. Brucker
|
062f2dea26
|
Report size before/after compressing of data base files.
|
2017-06-30 20:02:49 +01:00 |
|
Achim D. Brucker
|
70f69f652f
|
Compress aa-ac.sqlite.
|
2017-06-29 22:46:59 +01:00 |
|
Achim D. Brucker
|
ef04c71314
|
Delete compressed database before re-creating it.
|
2017-06-29 20:38:31 +01:00 |
|
Achim D. Brucker
|
77c1fae66d
|
Compress log files.
|
2017-06-29 20:22:13 +01:00 |
|
Achim D. Brucker
|
69c20edfe8
|
Use same date/time for log files.
|
2017-06-29 19:45:21 +01:00 |
|
Achim D. Brucker
|
1f963ff4e4
|
Improved log output.
|
2017-06-28 20:34:47 +01:00 |
|
Achim D. Brucker
|
0a3d46f6da
|
Simple update script to simplify crontab entries.
|
2017-06-27 14:19:13 +01:00 |