Commit Graph

500 Commits

Author SHA1 Message Date
Achim D. Brucker 58f524e31a Fixed typo. 2019-05-14 06:53:48 +01:00
Achim D. Brucker a6bf9090fb No DB update (for testing). 2019-05-13 07:16:41 +01:00
Michael Herzberg 6afc1429ed Small changes to extgrep. 2019-02-14 11:50:49 +00:00
Achim D. Brucker 1ecf8f26ed Update to Python 3.7. 2019-02-02 18:17:20 +00:00
Michael Herzberg 253c89fd7f Started extgrep. 2019-01-24 13:59:31 +00:00
Michael Herzberg d4d7e3c8ca Fix missing host error in mariadb 10.3 config files. 2019-01-23 16:48:31 +00:00
Achim D. Brucker 2a946570b5 Moved to Python 3.7. 2019-01-18 07:23:18 +00:00
Achim D. Brucker af2419a473 Using python 3.7. 2019-01-17 22:58:39 +00:00
Achim D. Brucker 43715c6ed1 Using python 3.7. 2019-01-16 07:23:18 +00:00
Achim D. Brucker b2e18aa26d Using python 3.7. 2019-01-15 18:41:12 +00:00
Michael Herzberg 63fe0806ee Fixed double-logging when not using forkserver. 2018-09-02 11:46:27 +01:00
Michael Herzberg 873c249504 Build list for simhash lazily to save memory. 2018-08-17 15:20:00 +01:00
Michael Herzberg 66db569d5f Only open DB connection when needed. 2018-08-02 12:37:57 +01:00
Michael Herzberg 05c1cbdea5 Give MySQL server up to 1 hour to recover. 2018-08-02 11:47:38 +01:00
Michael Herzberg 947ecf50d4 Removed queue length and reduced mysql insert batch size. 2018-07-28 19:02:38 +02:00
Michael Herzberg 4592cba9b2 Actually return n new ids when discovering. 2018-07-28 10:32:16 +02:00
Michael Herzberg 45a8486f69 Fixed small bug. 2018-07-27 16:39:42 +02:00
Michael Herzberg eb616b0ac3 Fix some encoding issues. 2018-07-21 01:50:59 +01:00
Michael Herzberg 250bdd2c6b Bundle mysql inserts. 2018-07-19 23:26:25 +01:00
Michael Herzberg a1d866d0ff Overwrite last_updated on duplicate. 2018-07-17 14:06:43 +01:00
Michael Herzberg a6173fe23e Don't look for etags in the DB anymore. 2018-07-16 19:19:26 +01:00
Michael Herzberg 4b5cc276ee Added option to use INSERT DELAYED with create-db. 2018-07-16 19:14:24 +01:00
Michael Herzberg 1857ec7b75 Don't restart processes (hopefully mitigates semlock rebuilding error). 2018-07-16 11:21:42 +01:00
Michael Herzberg 8bc4e8fa37 Cache etags in applications. 2018-07-16 01:04:27 +01:00
Michael Herzberg 595f0f8759 Use 16 threads to discover new extensions. 2018-07-15 19:19:26 +01:00
Michael Herzberg c9e66186ef Log duration of tar append. 2018-07-15 19:15:20 +01:00
Michael Herzberg 3bef0afe7a Group mysql inserts and don't compress them. 2018-07-15 00:08:11 +01:00
Achim D. Brucker 29bfc6fd38 Switched from pycrypto (2.6.1) to pycryptodomex (3.4.6). This should
mitigate CVE-2018-6594.
2018-07-12 23:02:51 +01:00
Michael Herzberg 651506bd0c Sort db inserts to prevent deadlocks. 2018-06-13 09:33:55 +01:00
Michael Herzberg 630fcba1df Actually write simhash into db... 2018-06-10 01:22:12 +01:00
Achim D. Brucker fd4ed697a7 Added default value for ext_id in const_log_format() to ensure backwards compatibility. 2018-04-22 22:50:27 +01:00
Michael Herzberg 49ea3bb496 Make sure semaphore is released if an exception occurs during http request. 2018-04-22 13:59:15 +01:00
Michael Herzberg 756dcb3ed1 Increased wait time again... 2018-04-21 21:33:36 +01:00
Michael Herzberg 1dab51d3f5 Reduced bot detection timeout. 2018-04-21 20:50:08 +01:00
Michael Herzberg 5b0f49b35a Deleted annoying Creating DB Connection message. 2018-04-21 20:35:23 +01:00
Michael Herzberg d8d49b1b80 Moved ext_id into logger formatter to make logger output more uniform. 2018-04-21 19:59:02 +01:00
Michael Herzberg ecb00f6009 Merge branch 'master' into mixed_forums 2018-04-21 19:19:07 +01:00
Michael Herzberg a789fe505f Fixed style errors and warnings. 2018-04-21 19:00:07 +01:00
Michael Herzberg ac3c1c7f20 Removed plain multiprocessing option. 2018-04-21 17:25:22 +01:00
Michael Herzberg 0613ac1ac1 Removed explicitly calling the garbage collector. 2018-04-21 16:52:58 +01:00
Michael Herzberg 2715e95665 Only try to add review and support pages if HTTP return code is 200. 2018-04-21 16:50:33 +01:00
Michael Herzberg dbeba9e9bf Use a lock to mix forum downloads into the parallel mode. 2018-04-21 13:59:33 +01:00
Michael Herzberg aee916a629 Moved setting of forkserver further outwards... 2018-04-15 16:26:26 +01:00
Michael Herzberg a758134c97 Readded mimetype from mimetypes. TODO: add mysql columns 2018-04-11 16:52:22 +01:00
Michael Herzberg cd09e2509d Removed retry of worker exceptions; instead, properly log them similary to tar and sql exceptions. 2018-04-11 15:38:32 +01:00
Michael Herzberg 22dc8f8263 Added --pystuck option to start pystuck servers for all processes. 2018-04-11 15:15:52 +01:00
Michael Herzberg 46494ec18b Re-setup logging in new processes. 2018-04-10 18:19:12 +01:00
Michael Herzberg 410fa3cf1c Moved setting of forkserver to prevent multiple invocations. 2018-04-10 17:24:10 +01:00
Michael Herzberg 12bdc1b00f Don't crash if something is wrong with the etag file. 2018-04-10 16:32:12 +01:00
Michael Herzberg 385003771a Set chunksize, maxtasksperchild, and max_tasks to 100. 2018-04-10 16:23:22 +01:00