Commit Graph

356 Commits

Author SHA1 Message Date
Achim D. Brucker 22f18ec3c8 Refactoring. 2017-07-28 19:44:51 +01:00
Michael Herzberg fb64499c8f Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-07-28 16:47:33 +01:00
Michael Herzberg 44756e44d6 Added python-based sqlite merger. 2017-07-28 16:47:25 +01:00
Achim D. Brucker 1833dbad90 Refactoring. 2017-07-28 15:57:17 +01:00
Achim D. Brucker 5658facebd Set unused variables to None to help garbage collector. 2017-07-28 15:32:19 +01:00
Michael Herzberg 11604c0fa5 Collect jsfilesize instead of jsloc. 2017-07-26 12:05:54 +01:00
Achim D. Brucker 349adb4b39 Updated README.md to reflect new requirements.txt. 2017-07-26 07:41:37 +01:00
Achim D. Brucker 9c8cedad74 Add hoc generated requirements.txt. 2017-07-26 07:38:17 +01:00
Achim D. Brucker 73eedab07d Log time delta for each extension upate. 2017-07-26 07:29:48 +01:00
Michael Herzberg b3d1ab912e Wait a maximum of 10min before stopping jsbeautifier. 2017-07-25 22:57:55 +01:00
Achim D. Brucker 04d242ce7e Added sqlite3 version to config log. 2017-07-24 11:10:16 +01:00
Achim D. Brucker 9b291d02ae Added local bin/lib-directory to PATH. 2017-07-24 10:47:42 +01:00
Achim D. Brucker d6d1d21d05 Bug fix: Append log to log instead of overwritting it. 2017-07-24 10:26:03 +01:00
Achim D. Brucker 2bf32b7a8f Merge branch 'production' of logicalhacking.com:BrowserSecurity/ExtensionCrawler into production 2017-07-22 21:14:07 +01:00
Achim D. Brucker 052748af07 Create log file for profiling. 2017-07-20 07:51:30 +01:00
Michael Herzberg 072e008fe2 Run the garbage collector manually after using jsbeautify. 2017-07-19 17:25:21 +01:00
Achim D. Brucker c73f4cda3e Fixed log output. 2017-07-19 07:01:38 +01:00
Michael Herzberg cf0478d740 Added sge file for merging the individual dbs into the individual dbs of our archive. 2017-07-18 23:51:11 +01:00
Michael Herzberg 186f6162af Fixed NoneType str conversion exception. 2017-07-17 15:29:00 +01:00
Michael Herzberg 9b1e5db96f Check for attributes key first and use traceback module instead of printing str(e). 2017-07-17 14:00:39 +01:00
Michael Herzberg 50123f14b8 Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-07-16 20:14:59 +01:00
Michael Herzberg eded1ca893 Only attempt to search for replies when we actually have search parameters. 2017-07-16 20:14:50 +01:00
Achim D. Brucker 3346b1b61a Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-07-16 14:02:40 +01:00
Achim D. Brucker 049db03021 Add user local to binary and library search paths. 2017-07-16 14:02:22 +01:00
Michael Herzberg 5e60715d98 Removed schema creation from scripts as it is now being done by the merge script. 2017-07-13 14:47:42 +01:00
Michael Herzberg 34e62d4e13 Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-07-13 14:43:50 +01:00
Michael Herzberg a901fa30bb Adjusted merge scripts for fts tables. 2017-07-13 14:43:37 +01:00
Achim D. Brucker 02ce0cfd2c Print info: sqlite3 binary used. 2017-07-12 20:36:29 +01:00
Michael Herzberg 26bddde328 Removed primary keys from fts tables as that had no effect. 2017-07-12 18:30:37 +01:00
Michael Herzberg 2cb868b3d0 Removed removal of review row ids as we now use author ids as primary keys. 2017-07-12 18:13:04 +01:00
Michael Herzberg 6a6a12c88a Added parsing of support to sqlite. 2017-07-12 18:11:31 +01:00
Michael Herzberg 16a44cf499 Added parsing of review replies to sqlite. 2017-07-12 17:56:40 +01:00
Michael Herzberg 0ed8c15a2d Made review a fts table. 2017-07-12 17:04:56 +01:00
Michael Herzberg 51bdcb4f16 Also download replies for support forum. 2017-07-12 16:57:16 +01:00
Michael Herzberg 11b0ccee4a Added download of review replies. 2017-07-12 16:10:47 +01:00
Michael Herzberg d6ae9d28b8 Fixed bug that lead to downloading the first review page twice instead of the first and second review page. 2017-07-12 14:09:01 +01:00
Michael Herzberg 60dd98e60e Fixed parsing of developer from overview page. 2017-07-12 13:54:44 +01:00
Michael Herzberg 885f580f2e Fixed set -u. 2017-07-11 11:15:43 +01:00
Michael Herzberg 6d03e8da71 Refactored sge scripts. 2017-07-11 11:11:32 +01:00
Michael Herzberg 8ca76f373e Made extension crawler repo path mandatory for sge scripts. 2017-07-10 15:25:56 +01:00
Michael Herzberg 00ad40e3a0 Added more sge scripts. 2017-07-10 14:42:03 +01:00
Michael Herzberg 3b3bbfbb6d Improved grepper script. 2017-07-10 14:09:45 +01:00
Michael Herzberg dac1f3e998 Improved grepper script. 2017-07-10 13:45:25 +01:00
Michael Herzberg 5f360e40e4 Improved grepper script. 2017-07-10 13:17:48 +01:00
Michael Herzberg e60265975f Renamed etag to crx_etag. 2017-07-10 12:46:41 +01:00
Michael Herzberg fe6b563f31 Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-07-10 12:29:45 +01:00
Michael Herzberg af68a95139 Fixed sed string. 2017-07-10 12:28:55 +01:00
Achim D. Brucker c4e13daae5 Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-07-09 18:55:24 +01:00
Michael Herzberg 77023e001e Do not treat js file decoding strictly. 2017-07-07 22:17:56 +01:00
Michael Herzberg 38c88d7461 Added parsing of content_script_urls. 2017-07-07 20:09:22 +01:00