Commit Graph

444 Commits

Author SHA1 Message Date
Michael Herzberg 080f00f17c Added new columns for jsfile table. 2017-08-29 22:40:01 +01:00
Michael Herzberg 95d71a9edc Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-08-29 22:29:49 +01:00
Michael Herzberg 3e24d1f08c Changed logging to use logging library. 2017-08-29 22:29:38 +01:00
Achim D. Brucker 39cd03dccc Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-08-29 18:01:42 +01:00
Achim D. Brucker 97f5b14158 Compute sha1 for JavaScript files. 2017-08-29 18:01:28 +01:00
Michael Herzberg bddd80c138 Made removal of manifest.json comments stricter. 2017-08-29 15:43:04 +01:00
Michael Herzberg 7ffdf30545 Push manifest into table crx column manifest. 2017-08-29 15:41:13 +01:00
Michael Herzberg 2b11117b6f Always process crx, regardless whether or not crx_etag is already in db. 2017-08-29 15:24:59 +01:00
Michael Herzberg 8b91957372 Reduced default MySQL timeout. 2017-08-29 15:20:58 +01:00
Michael Herzberg 6a99d41471 Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-08-29 15:11:37 +01:00
Achim D. Brucker d4ad5f96f8 Report empty files as own category/type. 2017-08-28 22:38:06 +01:00
Michael Herzberg f81aac7c61 Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-08-28 22:38:05 +01:00
Achim D. Brucker 422112e537 Fixed JSON syntax. 2017-08-28 22:20:38 +01:00
Achim D. Brucker d853b6e2ab Added md5 hashes for several underscore versions. 2017-08-28 22:16:29 +01:00
Achim D. Brucker 9c871e0560 Added new library identifiers. 2017-08-28 22:09:27 +01:00
Achim D. Brucker c3fb2cd598 Added md5 hashes for bootstrap 3.x. 2017-08-28 21:47:30 +01:00
Achim D. Brucker 2ace19f453 Compute js_info (including md5 hash and character set detection) only once per file. 2017-08-28 21:05:15 +01:00
Achim D. Brucker 91dfe67513 Auto-detect character encoding of JavaScript files using cchardet. 2017-08-28 20:53:55 +01:00
Michael Herzberg c30f0c4147 Removed database and host setting. To be set in ~/.my.cnf file now. 2017-08-28 20:17:11 +01:00
Achim D. Brucker 5cff2bc1b7 New check based on file hash (md5). 2017-08-28 20:09:34 +01:00
Achim D. Brucker 030adb6adc Minor refactoring and cleanup. 2017-08-28 19:20:50 +01:00
Michael Herzberg 5175d28edc Convert some stuff to string for db insert. 2017-08-28 17:12:32 +01:00
Achim D. Brucker 1a09277e70 Added mysqlclient to dependencies. 2017-08-28 14:42:16 +01:00
Michael Herzberg 35ec26e001 Flush guarded output immediatly and remove parallelism for sharc jobs :(. 2017-08-28 11:56:34 +01:00
Michael Herzberg 0a4e8839a1 Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-08-28 11:50:49 +01:00
Michael Herzberg 81077b807c Added mysql retry logic and use time.time() now. 2017-08-28 11:50:41 +01:00
Achim D. Brucker 9bf0b47f98 Minor improvement of string conversion for JsBlock. 2017-08-28 10:50:52 +01:00
Achim D. Brucker c721e6fdbf Merge with upstream. 2017-08-28 10:49:01 +01:00
Achim D. Brucker f10923af03 Integreated js_mincer into decomposition analysis to allow, in the future, to check comments, code, and string literals explicitely. 2017-08-28 10:40:37 +01:00
Achim D. Brucker 9ef27f9ac9 Added missing return statements. 2017-08-28 10:28:21 +01:00
Achim D. Brucker 90b1db4a25 Added additional comment checks. 2017-08-28 01:26:13 +01:00
Achim D. Brucker 9b272c9302 Added option to merge subsequent single line comments into a single line comment block. 2017-08-28 01:17:00 +01:00
Achim D. Brucker 111777c821 Improved position counting. 2017-08-28 00:57:58 +01:00
Achim D. Brucker d4de20efc1 Bug fix: start position of blocks and omit empty code blocks. 2017-08-28 00:19:28 +01:00
Achim D. Brucker e2e92594ce Bug fix: catch also last block of file. 2017-08-27 23:34:33 +01:00
Michael Herzberg e04ec98eca Added logging for create-db. 2017-08-27 23:04:38 +01:00
Michael Herzberg 257afe92f0 Use selective insert. 2017-08-27 23:00:28 +01:00
Achim D. Brucker 629f492fa7 Added tests for code blocks and comments. 2017-08-27 22:58:09 +01:00
Achim D. Brucker 7ff1623bc6 Introduced JavaScript mincer working on file objects. 2017-08-27 22:51:55 +01:00
Michael Herzberg b98b7bc0f7 Fixed column typo. 2017-08-27 22:49:07 +01:00
Achim D. Brucker e324ab9483 Re-formatted and added documentation. 2017-08-27 22:41:04 +01:00
Achim D. Brucker 9376b4056f Collect string literals in code blocks. 2017-08-27 22:27:35 +01:00
Achim D. Brucker 41ca506b9f Return iterator that iterates over JavaScript blocks. 2017-08-27 22:17:04 +01:00
Achim D. Brucker 5add586da3 Initial commit. 2017-08-27 20:47:24 +01:00
Achim D. Brucker f6f0bc0394 Renamed jsdecompose.py to js_decomposer.py. 2017-08-27 20:45:56 +01:00
Michael Herzberg 9521240d90 Make stuff configurable. 2017-08-27 18:28:19 +01:00
Michael Herzberg 0cff600861 Fixed etag keys. 2017-08-27 17:35:58 +01:00
Michael Herzberg d4b0a6535b Fixed some things. 2017-08-27 16:57:23 +01:00
Michael Herzberg f075192b44 made sqlite default again. 2017-08-27 03:26:29 +01:00
Michael Herzberg 22c90dcb4f Truncate timezone from timestamps for mysql, make mysql default. 2017-08-27 03:14:43 +01:00