 Achim D. Brucker
|
3491214aad
|
Store log in montly directory and replace : by _ in names of log files.
|
2019-01-27 11:15:15 +00:00 |
|
Achim D. Brucker
|
2f9c9e7c78
|
Testing python 3.7.
|
2019-01-13 00:10:28 +00:00 |
|
Achim D. Brucker
|
3b75b839f0
|
Added SPDX identifier.
|
2018-09-03 00:30:58 +01:00 |
Michael Herzberg
|
e71323644a
|
Don't use forkserver anymore since it seems to lead to many sem_unlink/file not found exceptions.
|
2018-09-01 11:34:54 +01:00 |
|
Michael Herzberg
|
595f0f8759
|
Use 16 threads to discover new extensions.
|
2018-07-15 19:19:26 +01:00 |
|
Michael Herzberg
|
2e1769a853
|
Fixed no attribute 'id' error.
|
2018-04-23 15:50:31 +01:00 |
|
Michael Herzberg
|
9eb164bb81
|
Fixed refactor bug.
|
2018-04-22 21:47:30 +01:00 |
|
Michael Herzberg
|
d8d49b1b80
|
Moved ext_id into logger formatter to make logger output more uniform.
|
2018-04-21 19:59:02 +01:00 |
|
Michael Herzberg
|
dd011aaad1
|
Removed -P option.
|
2018-04-21 19:28:47 +01:00 |
|
Michael Herzberg
|
ecb00f6009
|
Merge branch 'master' into mixed_forums
|
2018-04-21 19:19:07 +01:00 |
|
Michael Herzberg
|
a789fe505f
|
Fixed style errors and warnings.
|
2018-04-21 19:00:07 +01:00 |
|
Michael Herzberg
|
ac3c1c7f20
|
Removed plain multiprocessing option.
|
2018-04-21 17:25:22 +01:00 |
|
Michael Herzberg
|
aee916a629
|
Moved setting of forkserver further outwards...
|
2018-04-15 16:26:26 +01:00 |
|
Michael Herzberg
|
ff78f8e7d8
|
Fixed missing parameter.
|
2018-04-12 23:25:31 +01:00 |
|
Michael Herzberg
|
cd09e2509d
|
Removed retry of worker exceptions; instead, properly log them similary to tar and sql exceptions.
|
2018-04-11 15:38:32 +01:00 |
|
Michael Herzberg
|
22dc8f8263
|
Added --pystuck option to start pystuck servers for all processes.
|
2018-04-11 15:15:52 +01:00 |
|
Michael Herzberg
|
46494ec18b
|
Re-setup logging in new processes.
|
2018-04-10 18:19:12 +01:00 |
|
Michael Herzberg
|
3d136daae3
|
Various small bug fixes.
|
2018-04-08 17:44:59 +01:00 |
|
Michael Herzberg
|
faa2214af4
|
Timeout must be an integer.
|
2018-04-08 13:10:26 +01:00 |
|
Achim D. Brucker
|
33898a4cf3
|
Updated help text.
|
2018-04-08 10:10:30 +01:00 |
|
Achim D. Brucker
|
e1ef0758f7
|
Made the choice of Pool vs. ProcessPool a configuration option.
|
2018-04-08 10:06:26 +01:00 |
|
Achim D. Brucker
|
d9fc65a089
|
Reformatting.
|
2018-04-06 07:27:57 +01:00 |
|
Achim D. Brucker
|
8c9aab8216
|
Converted timeout into a proper configuration parameter.
|
2018-04-06 07:25:21 +01:00 |
|
Achim D. Brucker
|
fd9cc1855a
|
Improved command line interface for selecting which type of extensiosn should be crawled.
|
2018-04-06 07:17:20 +01:00 |
|
Achim D. Brucker
|
fee88ed0fe
|
Implemented sequential download mode.
|
2018-04-05 17:32:11 +01:00 |
|
Achim D. Brucker
|
7d1f41589f
|
Reformatting.
|
2017-11-04 19:27:15 +00:00 |
|
Achim D. Brucker
|
7ba829c90f
|
Made python 3.6 the default.
|
2017-11-02 18:46:20 +00:00 |
|
Achim D. Brucker
|
b7bed2a341
|
Bug fix: log of corupted tar archives.
|
2017-10-12 00:01:41 +01:00 |
 Michael Herzberg
|
98a2d69ebb
|
This time actually disable annoying HTTPS log messages.
|
2017-09-18 13:41:54 +01:00 |
|
Michael Herzberg
|
cee90ececc
|
Moved urllib logger config.
|
2017-09-18 12:59:52 +01:00 |
|
Michael Herzberg
|
1ddb9c1c10
|
Surpress HTTPS connection log messages.
|
2017-09-17 12:26:51 +01:00 |
|
Michael Herzberg
|
abd9605ebc
|
Use python3.5 for all files.
|
2017-09-01 14:12:05 +01:00 |
|
Michael Herzberg
|
5c24608c4d
|
Added --max-discover <N> option to limit the number of new extensions.
|
2017-09-01 13:30:42 +01:00 |
|
Michael Herzberg
|
e06d3f4ac4
|
Reduced timeout and fixed logging.
|
2017-08-31 23:01:05 +01:00 |
|
Michael Herzberg
|
cbd2dea820
|
Removed everything related to sqlite and updated README.
|
2017-08-30 15:38:04 +01:00 |
|
Michael Herzberg
|
5f234d8539
|
Improved logging.
|
2017-08-30 15:12:54 +01:00 |
|
Michael Herzberg
|
3e24d1f08c
|
Changed logging to use logging library.
|
2017-08-29 22:29:38 +01:00 |
|
Michael Herzberg
|
9521240d90
|
Make stuff configurable.
|
2017-08-27 18:28:19 +01:00 |
|
Achim D. Brucker
|
eb0054b47d
|
Refactoring: Moved default configuration to config module.
|
2017-07-29 12:36:20 +01:00 |
|
Achim D. Brucker
|
659f37c90c
|
Refactoring.
|
2017-07-28 21:18:10 +01:00 |
|
Achim D. Brucker
|
d5d2251de9
|
Refactoring.
|
2017-07-28 20:21:38 +01:00 |
|
Achim D. Brucker
|
22f18ec3c8
|
Refactoring.
|
2017-07-28 19:44:51 +01:00 |
|
Achim D. Brucker
|
1833dbad90
|
Refactoring.
|
2017-07-28 15:57:17 +01:00 |
|
Achim D. Brucker
|
5658facebd
|
Set unused variables to None to help garbage collector.
|
2017-07-28 15:32:19 +01:00 |
|
Achim D. Brucker
|
04d242ce7e
|
Added sqlite3 version to config log.
|
2017-07-24 11:10:16 +01:00 |
|
Achim D. Brucker
|
675bd301cc
|
Added missing parallel parameter to second call of update_extensions.
|
2017-06-21 15:26:48 +01:00 |
|
Achim D. Brucker
|
c5a7f82e13
|
Fixed getopt setup.
|
2017-06-19 06:03:36 +01:00 |
|
Achim D. Brucker
|
d9195c8174
|
Max. number of concurrent download can now be configured via command line.
|
2017-06-18 15:36:21 +01:00 |
|
Achim D. Brucker
|
2e6323c8c5
|
Report number of extensions for which the SQL database was updated.
|
2017-06-17 18:15:08 +01:00 |
|
Achim D. Brucker
|
760ac171f1
|
Releaxed supported version to 3.4 or 3.5.
|
2017-06-17 15:43:18 +01:00 |