 Achim D. Brucker
|
58f524e31a
|
Fixed typo.
|
2019-05-14 06:53:48 +01:00 |
|
Achim D. Brucker
|
a6bf9090fb
|
No DB update (for testing).
|
2019-05-13 07:16:41 +01:00 |
Michael Herzberg
|
6afc1429ed
|
Small changes to extgrep.
|
2019-02-14 11:50:49 +00:00 |
|
Achim D. Brucker
|
1ecf8f26ed
|
Update to Python 3.7.
|
2019-02-02 18:17:20 +00:00 |
|
Michael Herzberg
|
253c89fd7f
|
Started extgrep.
|
2019-01-24 13:59:31 +00:00 |
|
Michael Herzberg
|
d4d7e3c8ca
|
Fix missing host error in mariadb 10.3 config files.
|
2019-01-23 16:48:31 +00:00 |
|
Achim D. Brucker
|
2a946570b5
|
Moved to Python 3.7.
|
2019-01-18 07:23:18 +00:00 |
|
Achim D. Brucker
|
af2419a473
|
Using python 3.7.
|
2019-01-17 22:58:39 +00:00 |
|
Achim D. Brucker
|
43715c6ed1
|
Using python 3.7.
|
2019-01-16 07:23:18 +00:00 |
|
Achim D. Brucker
|
b2e18aa26d
|
Using python 3.7.
|
2019-01-15 18:41:12 +00:00 |
|
Michael Herzberg
|
63fe0806ee
|
Fixed double-logging when not using forkserver.
|
2018-09-02 11:46:27 +01:00 |
|
Michael Herzberg
|
873c249504
|
Build list for simhash lazily to save memory.
|
2018-08-17 15:20:00 +01:00 |
|
Michael Herzberg
|
66db569d5f
|
Only open DB connection when needed.
|
2018-08-02 12:37:57 +01:00 |
|
Michael Herzberg
|
05c1cbdea5
|
Give MySQL server up to 1 hour to recover.
|
2018-08-02 11:47:38 +01:00 |
|
Michael Herzberg
|
947ecf50d4
|
Removed queue length and reduced mysql insert batch size.
|
2018-07-28 19:02:38 +02:00 |
|
Michael Herzberg
|
4592cba9b2
|
Actually return n new ids when discovering.
|
2018-07-28 10:32:16 +02:00 |
|
Michael Herzberg
|
45a8486f69
|
Fixed small bug.
|
2018-07-27 16:39:42 +02:00 |
|
Michael Herzberg
|
eb616b0ac3
|
Fix some encoding issues.
|
2018-07-21 01:50:59 +01:00 |
|
Michael Herzberg
|
250bdd2c6b
|
Bundle mysql inserts.
|
2018-07-19 23:26:25 +01:00 |
|
Michael Herzberg
|
a1d866d0ff
|
Overwrite last_updated on duplicate.
|
2018-07-17 14:06:43 +01:00 |
|
Michael Herzberg
|
a6173fe23e
|
Don't look for etags in the DB anymore.
|
2018-07-16 19:19:26 +01:00 |
|
Michael Herzberg
|
4b5cc276ee
|
Added option to use INSERT DELAYED with create-db.
|
2018-07-16 19:14:24 +01:00 |
|
Michael Herzberg
|
1857ec7b75
|
Don't restart processes (hopefully mitigates semlock rebuilding error).
|
2018-07-16 11:21:42 +01:00 |
|
Michael Herzberg
|
8bc4e8fa37
|
Cache etags in applications.
|
2018-07-16 01:04:27 +01:00 |
|
Michael Herzberg
|
595f0f8759
|
Use 16 threads to discover new extensions.
|
2018-07-15 19:19:26 +01:00 |
|
Michael Herzberg
|
c9e66186ef
|
Log duration of tar append.
|
2018-07-15 19:15:20 +01:00 |
|
Michael Herzberg
|
3bef0afe7a
|
Group mysql inserts and don't compress them.
|
2018-07-15 00:08:11 +01:00 |
|
Achim D. Brucker
|
29bfc6fd38
|
Switched from pycrypto (2.6.1) to pycryptodomex (3.4.6). This should
mitigate CVE-2018-6594.
|
2018-07-12 23:02:51 +01:00 |
|
Michael Herzberg
|
651506bd0c
|
Sort db inserts to prevent deadlocks.
|
2018-06-13 09:33:55 +01:00 |
|
Michael Herzberg
|
630fcba1df
|
Actually write simhash into db...
|
2018-06-10 01:22:12 +01:00 |
|
Achim D. Brucker
|
fd4ed697a7
|
Added default value for ext_id in const_log_format() to ensure backwards compatibility.
|
2018-04-22 22:50:27 +01:00 |
|
Michael Herzberg
|
49ea3bb496
|
Make sure semaphore is released if an exception occurs during http request.
|
2018-04-22 13:59:15 +01:00 |
|
Michael Herzberg
|
756dcb3ed1
|
Increased wait time again...
|
2018-04-21 21:33:36 +01:00 |
|
Michael Herzberg
|
1dab51d3f5
|
Reduced bot detection timeout.
|
2018-04-21 20:50:08 +01:00 |
|
Michael Herzberg
|
5b0f49b35a
|
Deleted annoying Creating DB Connection message.
|
2018-04-21 20:35:23 +01:00 |
|
Michael Herzberg
|
d8d49b1b80
|
Moved ext_id into logger formatter to make logger output more uniform.
|
2018-04-21 19:59:02 +01:00 |
|
Michael Herzberg
|
ecb00f6009
|
Merge branch 'master' into mixed_forums
|
2018-04-21 19:19:07 +01:00 |
|
Michael Herzberg
|
a789fe505f
|
Fixed style errors and warnings.
|
2018-04-21 19:00:07 +01:00 |
|
Michael Herzberg
|
ac3c1c7f20
|
Removed plain multiprocessing option.
|
2018-04-21 17:25:22 +01:00 |
|
Michael Herzberg
|
0613ac1ac1
|
Removed explicitly calling the garbage collector.
|
2018-04-21 16:52:58 +01:00 |
|
Michael Herzberg
|
2715e95665
|
Only try to add review and support pages if HTTP return code is 200.
|
2018-04-21 16:50:33 +01:00 |
|
Michael Herzberg
|
dbeba9e9bf
|
Use a lock to mix forum downloads into the parallel mode.
|
2018-04-21 13:59:33 +01:00 |
|
Michael Herzberg
|
aee916a629
|
Moved setting of forkserver further outwards...
|
2018-04-15 16:26:26 +01:00 |
|
Michael Herzberg
|
a758134c97
|
Readded mimetype from mimetypes. TODO: add mysql columns
|
2018-04-11 16:52:22 +01:00 |
|
Michael Herzberg
|
cd09e2509d
|
Removed retry of worker exceptions; instead, properly log them similary to tar and sql exceptions.
|
2018-04-11 15:38:32 +01:00 |
|
Michael Herzberg
|
22dc8f8263
|
Added --pystuck option to start pystuck servers for all processes.
|
2018-04-11 15:15:52 +01:00 |
|
Michael Herzberg
|
46494ec18b
|
Re-setup logging in new processes.
|
2018-04-10 18:19:12 +01:00 |
|
Michael Herzberg
|
410fa3cf1c
|
Moved setting of forkserver to prevent multiple invocations.
|
2018-04-10 17:24:10 +01:00 |
|
Michael Herzberg
|
12bdc1b00f
|
Don't crash if something is wrong with the etag file.
|
2018-04-10 16:32:12 +01:00 |
|
Michael Herzberg
|
385003771a
|
Set chunksize, maxtasksperchild, and max_tasks to 100.
|
2018-04-10 16:23:22 +01:00 |