Commit Graph

141 Commits

Author SHA1 Message Date
Michael Herzberg 68e7e72e93 Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-08-09 13:06:42 +01:00
Michael Herzberg 40f800b4de Check if 'annotations' exists in search results. 2017-08-09 13:06:22 +01:00
Achim D. Brucker d3da686e16 Changed formular for computing download delay. 2017-08-05 10:57:45 +01:00
Michael Herzberg c61f19e860 Use INSERT OR IGNORE. 2017-07-31 23:55:21 +01:00
Michael Herzberg b8f57196c7 Changed fts table structure. 2017-07-31 23:23:57 +01:00
Michael Herzberg b34d45c4dc Added md5sum to sqlite. 2017-07-31 20:38:21 +01:00
Achim D. Brucker 35c133e395 Slightly more aggressive implementation of google_dos_protection. 2017-07-30 14:41:20 +01:00
Achim D. Brucker 5268f2a732 Refactoring: clean-up of imports and a few other minor improvements. 2017-07-29 16:13:39 +01:00
Achim D. Brucker eb0054b47d Refactoring: Moved default configuration to config module. 2017-07-29 12:36:20 +01:00
Achim D. Brucker 0b24fb15fe Refactoring. 2017-07-29 11:32:06 +01:00
Achim D. Brucker 0ca3476b09 Slightly more aggressive implementation of google_dos_protection. 2017-07-29 11:21:15 +01:00
Achim D. Brucker d05ca9678e Refactoring. 2017-07-29 10:57:35 +01:00
Achim D. Brucker ac663299b3 Refactoring. 2017-07-29 10:17:16 +01:00
Achim D. Brucker 10cce2859d Renamed variable/attribute pk to public_key. 2017-07-29 09:15:22 +01:00
Achim D. Brucker 333bcaa62d Strip path from crx file. 2017-07-29 09:12:01 +01:00
Achim D. Brucker e5d671c7c4 Refactoring. 2017-07-29 09:05:16 +01:00
Michael Herzberg 11604c0fa5 Collect jsfilesize instead of jsloc. 2017-07-26 12:05:54 +01:00
Achim D. Brucker 73eedab07d Log time delta for each extension upate. 2017-07-26 07:29:48 +01:00
Michael Herzberg b3d1ab912e Wait a maximum of 10min before stopping jsbeautifier. 2017-07-25 22:57:55 +01:00
Michael Herzberg 072e008fe2 Run the garbage collector manually after using jsbeautify. 2017-07-19 17:25:21 +01:00
Michael Herzberg 186f6162af Fixed NoneType str conversion exception. 2017-07-17 15:29:00 +01:00
Michael Herzberg 9b1e5db96f Check for attributes key first and use traceback module instead of printing str(e). 2017-07-17 14:00:39 +01:00
Michael Herzberg eded1ca893 Only attempt to search for replies when we actually have search parameters. 2017-07-16 20:14:50 +01:00
Michael Herzberg 26bddde328 Removed primary keys from fts tables as that had no effect. 2017-07-12 18:30:37 +01:00
Michael Herzberg 6a6a12c88a Added parsing of support to sqlite. 2017-07-12 18:11:31 +01:00
Michael Herzberg 16a44cf499 Added parsing of review replies to sqlite. 2017-07-12 17:56:40 +01:00
Michael Herzberg 0ed8c15a2d Made review a fts table. 2017-07-12 17:04:56 +01:00
Michael Herzberg 51bdcb4f16 Also download replies for support forum. 2017-07-12 16:57:16 +01:00
Michael Herzberg 11b0ccee4a Added download of review replies. 2017-07-12 16:10:47 +01:00
Michael Herzberg d6ae9d28b8 Fixed bug that lead to downloading the first review page twice instead of the first and second review page. 2017-07-12 14:09:01 +01:00
Michael Herzberg 60dd98e60e Fixed parsing of developer from overview page. 2017-07-12 13:54:44 +01:00
Michael Herzberg e60265975f Renamed etag to crx_etag. 2017-07-10 12:46:41 +01:00
Achim D. Brucker c4e13daae5 Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-07-09 18:55:24 +01:00
Michael Herzberg 77023e001e Do not treat js file decoding strictly. 2017-07-07 22:17:56 +01:00
Michael Herzberg 38c88d7461 Added parsing of content_script_urls. 2017-07-07 20:09:22 +01:00
Michael Herzberg fbc0a7c87c Added crx size and jsloc. 2017-07-07 19:47:14 +01:00
Michael Herzberg 62dc61826a Added parsing of itemcategory. 2017-07-07 19:29:51 +01:00
Michael Herzberg dbe8a26a6b Fixed download parsing. 2017-07-05 16:20:52 +01:00
Michael Herzberg cface0128c Changed download number extraction to also work with Google Docs extensions (and potentially others). 2017-07-05 16:08:15 +01:00
Michael Herzberg 4c01b95f69 Added ratingValue and ratingCount to db. 2017-07-05 14:23:45 +01:00
Achim D. Brucker d5d0a44b69 Reformatting. 2017-07-05 08:21:40 +01:00
Achim D. Brucker 600ec933f4 Introduced optional argument to last_crx - return latest crx that is not newer than the passed date/time. 2017-07-05 08:21:00 +01:00
Achim D. Brucker 30c0b92979 Ignore empty crx files in calculating last crx file date. 2017-07-04 09:30:33 +01:00
Achim D. Brucker 939b29f55a Use getmembers instead of getnames in last_crx(). 2017-07-03 07:04:03 +01:00
Michael Herzberg 6d5221c5d7 Make db path configurable. 2017-06-22 17:46:18 +01:00
Michael Herzberg 6833ba6683 Fixed sqlite creation, added missing commit 2017-06-20 23:47:31 +01:00
Michael Herzberg 4220d48d34 Close db when an exception is thrown. 2017-06-20 23:15:15 +01:00
Achim D. Brucker 8dbd535e3e Merge branch 'master' into production 2017-06-20 20:03:10 +01:00
Achim D. Brucker d9ebe265ae Re-formatting 2017-06-20 18:17:44 +01:00
Achim D. Brucker 05227494d6 Re-formatting 2017-06-20 18:17:36 +01:00