Commit Graph

452 Commits

Author SHA1 Message Date
Achim D. Brucker d7120fad45 Bug fix: update char if loops reads another char (next_char). This avoids missing an escape character or a newline. 2017-08-30 01:43:18 +01:00
Achim D. Brucker 9be998efd1 Ignore vscode workspace configuration. 2017-08-30 00:52:47 +01:00
Achim D. Brucker b4dbfc1f19 Added new tool: crx-jsdecompose. 2017-08-30 00:48:28 +01:00
Achim D. Brucker 66818b2fa6 Renamed hash to md5 in JSON file and added support for sha1 hashes. 2017-08-30 00:38:30 +01:00
Achim D. Brucker e947e69f37 Define type and detection method for all generated entries. 2017-08-30 00:24:19 +01:00
Achim D. Brucker ae3bbd7339 Using values of enumeration to obtain nice and short human readable representations. 2017-08-30 00:12:57 +01:00
Achim D. Brucker fb13925be9 Added new tool: crx-jsdecompose. 2017-08-30 00:04:32 +01:00
Achim D. Brucker 4f59c56e73 Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-08-29 22:54:54 +01:00
Michael Herzberg 95d71a9edc Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-08-29 22:29:49 +01:00
Michael Herzberg 3e24d1f08c Changed logging to use logging library. 2017-08-29 22:29:38 +01:00
Achim D. Brucker b4e00ea4fe Renamed extract-crx to crx-extract. 2017-08-29 22:06:18 +01:00
Achim D. Brucker 39cd03dccc Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-08-29 18:01:42 +01:00
Achim D. Brucker 97f5b14158 Compute sha1 for JavaScript files. 2017-08-29 18:01:28 +01:00
Michael Herzberg bddd80c138 Made removal of manifest.json comments stricter. 2017-08-29 15:43:04 +01:00
Michael Herzberg 7ffdf30545 Push manifest into table crx column manifest. 2017-08-29 15:41:13 +01:00
Michael Herzberg 2b11117b6f Always process crx, regardless whether or not crx_etag is already in db. 2017-08-29 15:24:59 +01:00
Michael Herzberg 8b91957372 Reduced default MySQL timeout. 2017-08-29 15:20:58 +01:00
Michael Herzberg 6a99d41471 Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-08-29 15:11:37 +01:00
Achim D. Brucker d4ad5f96f8 Report empty files as own category/type. 2017-08-28 22:38:06 +01:00
Michael Herzberg f81aac7c61 Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-08-28 22:38:05 +01:00
Achim D. Brucker 422112e537 Fixed JSON syntax. 2017-08-28 22:20:38 +01:00
Achim D. Brucker d853b6e2ab Added md5 hashes for several underscore versions. 2017-08-28 22:16:29 +01:00
Achim D. Brucker 9c871e0560 Added new library identifiers. 2017-08-28 22:09:27 +01:00
Achim D. Brucker c3fb2cd598 Added md5 hashes for bootstrap 3.x. 2017-08-28 21:47:30 +01:00
Achim D. Brucker 2ace19f453 Compute js_info (including md5 hash and character set detection) only once per file. 2017-08-28 21:05:15 +01:00
Achim D. Brucker 91dfe67513 Auto-detect character encoding of JavaScript files using cchardet. 2017-08-28 20:53:55 +01:00
Michael Herzberg c30f0c4147 Removed database and host setting. To be set in ~/.my.cnf file now. 2017-08-28 20:17:11 +01:00
Achim D. Brucker 5cff2bc1b7 New check based on file hash (md5). 2017-08-28 20:09:34 +01:00
Achim D. Brucker 030adb6adc Minor refactoring and cleanup. 2017-08-28 19:20:50 +01:00
Michael Herzberg 5175d28edc Convert some stuff to string for db insert. 2017-08-28 17:12:32 +01:00
Achim D. Brucker 1a09277e70 Added mysqlclient to dependencies. 2017-08-28 14:42:16 +01:00
Michael Herzberg 35ec26e001 Flush guarded output immediatly and remove parallelism for sharc jobs :(. 2017-08-28 11:56:34 +01:00
Michael Herzberg 0a4e8839a1 Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler 2017-08-28 11:50:49 +01:00
Michael Herzberg 81077b807c Added mysql retry logic and use time.time() now. 2017-08-28 11:50:41 +01:00
Achim D. Brucker 9bf0b47f98 Minor improvement of string conversion for JsBlock. 2017-08-28 10:50:52 +01:00
Achim D. Brucker c721e6fdbf Merge with upstream. 2017-08-28 10:49:01 +01:00
Achim D. Brucker f10923af03 Integreated js_mincer into decomposition analysis to allow, in the future, to check comments, code, and string literals explicitely. 2017-08-28 10:40:37 +01:00
Achim D. Brucker 9ef27f9ac9 Added missing return statements. 2017-08-28 10:28:21 +01:00
Achim D. Brucker 90b1db4a25 Added additional comment checks. 2017-08-28 01:26:13 +01:00
Achim D. Brucker 9b272c9302 Added option to merge subsequent single line comments into a single line comment block. 2017-08-28 01:17:00 +01:00
Achim D. Brucker 111777c821 Improved position counting. 2017-08-28 00:57:58 +01:00
Achim D. Brucker d4de20efc1 Bug fix: start position of blocks and omit empty code blocks. 2017-08-28 00:19:28 +01:00
Achim D. Brucker e2e92594ce Bug fix: catch also last block of file. 2017-08-27 23:34:33 +01:00
Michael Herzberg e04ec98eca Added logging for create-db. 2017-08-27 23:04:38 +01:00
Michael Herzberg 257afe92f0 Use selective insert. 2017-08-27 23:00:28 +01:00
Achim D. Brucker 629f492fa7 Added tests for code blocks and comments. 2017-08-27 22:58:09 +01:00
Achim D. Brucker 7ff1623bc6 Introduced JavaScript mincer working on file objects. 2017-08-27 22:51:55 +01:00
Michael Herzberg b98b7bc0f7 Fixed column typo. 2017-08-27 22:49:07 +01:00
Achim D. Brucker e324ab9483 Re-formatted and added documentation. 2017-08-27 22:41:04 +01:00
Achim D. Brucker 9376b4056f Collect string literals in code blocks. 2017-08-27 22:27:35 +01:00