 Michael Herzberg
|
abd9605ebc
|
Use python3.5 for all files.
|
2017-09-01 14:12:05 +01:00 |
 Achim D. Brucker
|
8289d50d38
|
Download all extensions in parallel and later do a second download for a subset including forums/reviews.
|
2017-09-01 12:14:39 +01:00 |
|
Achim D. Brucker
|
22264fb9e0
|
Changed download order: first download parallel all extensions without forum/review download, then download extensions with forums.
|
2017-09-01 12:02:12 +01:00 |
|
Michael Herzberg
|
e06d3f4ac4
|
Reduced timeout and fixed logging.
|
2017-08-31 23:01:05 +01:00 |
|
Michael Herzberg
|
cbd2dea820
|
Removed everything related to sqlite and updated README.
|
2017-08-30 15:38:04 +01:00 |
|
Michael Herzberg
|
5f234d8539
|
Improved logging.
|
2017-08-30 15:12:54 +01:00 |
|
Michael Herzberg
|
9b8a693a5f
|
Changed logging a little bit.
|
2017-08-30 12:12:57 +01:00 |
|
Michael Herzberg
|
3e24d1f08c
|
Changed logging to use logging library.
|
2017-08-29 22:29:38 +01:00 |
|
Achim D. Brucker
|
5268f2a732
|
Refactoring: clean-up of imports and a few other minor improvements.
|
2017-07-29 16:13:39 +01:00 |
|
Achim D. Brucker
|
0b24fb15fe
|
Refactoring.
|
2017-07-29 11:32:06 +01:00 |
|
Achim D. Brucker
|
73eedab07d
|
Log time delta for each extension upate.
|
2017-07-26 07:29:48 +01:00 |
|
Michael Herzberg
|
9b1e5db96f
|
Check for attributes key first and use traceback module instead of printing str(e).
|
2017-07-17 14:00:39 +01:00 |
|
Michael Herzberg
|
eded1ca893
|
Only attempt to search for replies when we actually have search parameters.
|
2017-07-16 20:14:50 +01:00 |
|
Michael Herzberg
|
51bdcb4f16
|
Also download replies for support forum.
|
2017-07-12 16:57:16 +01:00 |
|
Michael Herzberg
|
11b0ccee4a
|
Added download of review replies.
|
2017-07-12 16:10:47 +01:00 |
|
Michael Herzberg
|
d6ae9d28b8
|
Fixed bug that lead to downloading the first review page twice instead of the first and second review page.
|
2017-07-12 14:09:01 +01:00 |
|
Achim D. Brucker
|
d5d0a44b69
|
Reformatting.
|
2017-07-05 08:21:40 +01:00 |
|
Achim D. Brucker
|
600ec933f4
|
Introduced optional argument to last_crx - return latest crx that is not newer than the passed date/time.
|
2017-07-05 08:21:00 +01:00 |
|
Achim D. Brucker
|
30c0b92979
|
Ignore empty crx files in calculating last crx file date.
|
2017-07-04 09:30:33 +01:00 |
|
Achim D. Brucker
|
939b29f55a
|
Use getmembers instead of getnames in last_crx().
|
2017-07-03 07:04:03 +01:00 |
|
Michael Herzberg
|
6d5221c5d7
|
Make db path configurable.
|
2017-06-22 17:46:18 +01:00 |
|
Achim D. Brucker
|
8dbd535e3e
|
Merge branch 'master' into production
|
2017-06-20 20:03:10 +01:00 |
|
Achim D. Brucker
|
d9ebe265ae
|
Re-formatting
|
2017-06-20 18:17:44 +01:00 |
|
Michael Herzberg
|
dae5d4caa9
|
Fixed creation of empty .crx files.
|
2017-06-20 18:05:22 +01:00 |
|
Michael Herzberg
|
c1cd41c2e1
|
Improved logging.
|
2017-06-19 18:41:29 +01:00 |
|
Michael Herzberg
|
282e2c4e8c
|
Worked on sqlite stuff.
|
2017-06-19 16:42:35 +01:00 |
|
Achim D. Brucker
|
f95619670c
|
Merge branch 'master' into production
|
2017-06-18 15:38:13 +01:00 |
|
Achim D. Brucker
|
d9195c8174
|
Max. number of concurrent download can now be configured via command line.
|
2017-06-18 15:36:21 +01:00 |
|
Achim D. Brucker
|
85c8f6a546
|
Pass is_new flag to sqlite update.
|
2017-06-17 18:19:44 +01:00 |
|
Achim D. Brucker
|
2e6323c8c5
|
Report number of extensions for which the SQL database was updated.
|
2017-06-17 18:15:08 +01:00 |
|
Achim D. Brucker
|
ea71c5b6e3
|
Removed debugging code raising execptions.
|
2017-06-17 15:38:17 +01:00 |
|
Achim D. Brucker
|
8fcc7ab99f
|
Fixed logging.
|
2017-06-17 00:48:34 +01:00 |
|
Achim D. Brucker
|
97460c498f
|
Basic support for logging of errors related to SQL import/update.
|
2017-06-17 00:43:40 +01:00 |
|
Achim D. Brucker
|
c4a5c5a231
|
Ignore non extensions ids in forums.conf.
|
2017-06-16 23:32:52 +01:00 |
|
Achim D. Brucker
|
86a608c6a1
|
Re-formatting.
|
2017-06-16 23:19:13 +01:00 |
|
Achim D. Brucker
|
1c8d68d495
|
Moved path utility functions into config module.
|
2017-06-16 23:09:23 +01:00 |
|
Michael Herzberg
|
6e2772711f
|
Next version of sqlite generator.
|
2017-06-16 20:40:48 +01:00 |
|
Achim D. Brucker
|
ab4c0ad002
|
Fixed logging.
|
2017-06-16 12:07:51 +01:00 |
|
Achim D. Brucker
|
73baef61b2
|
Re-formatting.
|
2017-06-16 10:29:07 +01:00 |
|
Achim D. Brucker
|
64778c783e
|
Check etags in addition to modified-since (basic implementation).
|
2017-06-16 10:28:47 +01:00 |
|
Achim D. Brucker
|
763ac137b2
|
Re-introduced parallel download (they are not causing the If-Modified-Since problem).
|
2017-06-15 23:14:07 +01:00 |
|
Achim D. Brucker
|
3ff67bddc8
|
Disabled parallel download (IF-Modified Bug Hunting).
|
2017-06-14 16:56:05 +01:00 |
|
Achim D. Brucker
|
e164b7fe72
|
Reformatting.
|
2017-05-27 20:38:56 +01:00 |
|
Achim D. Brucker
|
b4e680ccef
|
Disabled local backup.
|
2017-05-20 21:09:03 +01:00 |
|
Achim D. Brucker
|
02eed607b4
|
Re-added wait before downloading first forum/support page - old setup was too aggressive.
|
2017-04-14 19:56:39 +01:00 |
|
Achim D. Brucker
|
a547e840cb
|
Normalized formatting.
|
2017-04-13 09:34:33 +01:00 |
|
Achim D. Brucker
|
0ee655f018
|
Interlaving forum and support download with crx download and removed inital wait before downloading first forum/support page.
|
2017-04-12 08:59:08 +01:00 |
|
Achim D. Brucker
|
0f88b5d8c5
|
Reduced pool size to reduce load.
|
2017-04-11 07:44:19 +01:00 |
|
Achim D. Brucker
|
63bf936e1e
|
Avoid use of ReadTarFS (seems to be very slow compared to the tarfile module).
|
2017-04-10 07:08:52 +01:00 |
|
Achim D. Brucker
|
e65c21a3b1
|
Increased pool size to 48.
|
2017-04-08 00:12:40 +01:00 |