acb14jd
/
ExtensionCrawler
forked from BrowserSecurity/ExtensionCrawler
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
356 Commits 2 Branches 0 Tags 398 KiB
Commit Graph

356 Commits

Author SHA1 Message Date
Michael Herzberg b6bf280d1e Fixed error. 2017-06-20 08:49:01 +01:00
Michael Herzberg 3496e89460 Fixed error. 2017-06-20 08:43:43 +01:00
Michael Herzberg aa259807e2 Catch exceptions due to empty crx header file. 2017-06-20 08:42:30 +01:00
Michael Herzberg c47ba57c97 Changed handing of manifest parsing exceptions. 2017-06-20 08:28:50 +01:00
Michael Herzberg d2dd2aaf81 Moved db path into config file. 2017-06-20 08:10:28 +01:00
Michael Herzberg 3b6662d05a Added .swp file to gitignore. 2017-06-20 08:09:40 +01:00
Michael Herzberg 39d7bf0330 Deal with missing annotation block in reviews. 2017-06-20 08:03:15 +01:00
Michael Herzberg fa82129c2b Deal with a possibly missing overview.html.status file. 2017-06-19 21:34:42 +01:00
Michael Herzberg c1cd41c2e1 Improved logging. 2017-06-19 18:41:29 +01:00
Michael Herzberg 282e2c4e8c Worked on sqlite stuff. 2017-06-19 16:42:35 +01:00
Achim D. Brucker 1f403ff2b5 Merge branch 'master' into production 2017-06-19 06:03:52 +01:00
Achim D. Brucker c5a7f82e13 Fixed getopt setup. 2017-06-19 06:03:36 +01:00
Achim D. Brucker 456bf292c8 Merge branch 'master' into production 2017-06-19 05:57:29 +01:00
Achim D. Brucker be293b1ba8 Fixed import. 2017-06-19 05:57:17 +01:00
Achim D. Brucker f95619670c Merge branch 'master' into production 2017-06-18 15:38:13 +01:00
Achim D. Brucker d9195c8174 Max. number of concurrent download can now be configured via command line. 2017-06-18 15:36:21 +01:00
Achim D. Brucker 66eff6780d Fixed passign is_new. 2017-06-17 18:26:04 +01:00
Achim D. Brucker 85c8f6a546 Pass is_new flag to sqlite update. 2017-06-17 18:19:44 +01:00
Achim D. Brucker 2e6323c8c5 Report number of extensions for which the SQL database was updated. 2017-06-17 18:15:08 +01:00
Michael Herzberg c4ff67a129 Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-06-17 17:10:36 +01:00
Michael Herzberg 7f24a9da7a Split db creation into incremental part and separate full regeneration script. 2017-06-17 17:10:18 +01:00
Achim D. Brucker 760ac171f1 Releaxed supported version to 3.4 or 3.5. 2017-06-17 15:43:18 +01:00
Achim D. Brucker ea71c5b6e3 Removed debugging code raising execptions. 2017-06-17 15:38:17 +01:00
Achim D. Brucker 8fcc7ab99f Fixed logging. 2017-06-17 00:48:34 +01:00
Achim D. Brucker 97460c498f Basic support for logging of errors related to SQL import/update. 2017-06-17 00:43:40 +01:00
Achim D. Brucker c4a5c5a231 Ignore non extensions ids in forums.conf. 2017-06-16 23:32:52 +01:00
Achim D. Brucker 86a608c6a1 Re-formatting. 2017-06-16 23:19:13 +01:00
Achim D. Brucker 1c8d68d495 Moved path utility functions into config module. 2017-06-16 23:09:23 +01:00
Achim D. Brucker 5dd424c06d Removed obsolete permstats.py. 2017-06-16 22:48:37 +01:00
Achim D. Brucker 012c285c84 Added assert to ensure Python 3.5. 2017-06-16 22:46:04 +01:00
Achim D. Brucker 9f174f6785 Downport to python 3.5. 2017-06-16 22:38:48 +01:00
Michael Herzberg 6e2772711f Next version of sqlite generator. 2017-06-16 20:40:48 +01:00
Michael Herzberg c08124fa17 First version of sqlite generator. 2017-06-16 14:56:23 +01:00
Achim D. Brucker ab4c0ad002 Fixed logging. 2017-06-16 12:07:51 +01:00
Achim D. Brucker b9e5ca6f82 Stub for updating sqlite. 2017-06-16 11:06:04 +01:00
Achim D. Brucker 73baef61b2 Re-formatting. 2017-06-16 10:29:07 +01:00
Achim D. Brucker 64778c783e Check etags in addition to modified-since (basic implementation). 2017-06-16 10:28:47 +01:00
Achim D. Brucker 763ac137b2 Re-introduced parallel download (they are not causing the If-Modified-Since problem). 2017-06-15 23:14:07 +01:00
Achim D. Brucker 3ff67bddc8 Disabled parallel download (IF-Modified Bug Hunting). 2017-06-14 16:56:05 +01:00
Achim D. Brucker fd717d1516 Reformatting. 2017-05-27 20:39:00 +01:00
Achim D. Brucker e164b7fe72 Reformatting. 2017-05-27 20:38:56 +01:00
Achim D. Brucker b4e680ccef Disabled local backup. 2017-05-20 21:09:03 +01:00
Achim D. Brucker 02eed607b4 Re-added wait before downloading first forum/support page - old setup was too aggressive. 2017-04-14 19:56:39 +01:00
Achim D. Brucker a547e840cb Normalized formatting. 2017-04-13 09:34:33 +01:00
Achim D. Brucker 0ee655f018 Interlaving forum and support download with crx download and removed inital wait before downloading first forum/support page. 2017-04-12 08:59:08 +01:00
Achim D. Brucker 0f88b5d8c5 Reduced pool size to reduce load. 2017-04-11 07:44:19 +01:00
Achim D. Brucker 63bf936e1e Avoid use of ReadTarFS (seems to be very slow compared to the tarfile module). 2017-04-10 07:08:52 +01:00
Achim D. Brucker 23fd7731bc Reduced waiting time for DOS protection. 2017-04-09 08:20:52 +01:00
Achim D. Brucker e65c21a3b1 Increased pool size to 48. 2017-04-08 00:12:40 +01:00
Achim D. Brucker c03fa789bd Fixed reporting of new extensions. 2017-04-07 07:07:49 +01:00