 Michael Herzberg
|
300a8c905a
|
Only log last mysql exception as error, rest as warning.
|
2017-10-08 20:57:25 +01:00 |
 Achim D. Brucker
|
25c37d83c1
|
Silently correct 'name use count' exception from libmagic (caused by a but in the magic Python module).
|
2017-10-08 15:18:58 +01:00 |
|
Achim D. Brucker
|
1963a20b69
|
Report starting positions of string literals.
|
2017-10-08 12:03:50 +01:00 |
|
Michael Herzberg
|
615b8f46a3
|
Fixed mysql caching.
|
2017-10-07 21:01:14 +01:00 |
|
Michael Herzberg
|
2abc386f48
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-10-06 20:13:16 +01:00 |
|
Michael Herzberg
|
6372c62336
|
Removed sorting again.
|
2017-10-06 20:13:08 +01:00 |
|
Achim D. Brucker
|
1ee76d9817
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-10-06 19:36:12 +01:00 |
|
Achim D. Brucker
|
c1750838f1
|
Added support for tar files.
|
2017-10-06 18:33:35 +01:00 |
|
Michael Herzberg
|
d6869455a8
|
Sort extension ids before processing.
|
2017-10-06 12:12:49 +01:00 |
|
Michael Herzberg
|
d05194b9bb
|
Group cached commits for efficiency.
|
2017-10-06 12:08:21 +01:00 |
|
Michael Herzberg
|
2cb56edd9b
|
Adjusted retries for create-db.
|
2017-10-05 11:14:59 +01:00 |
|
Michael Herzberg
|
6ba73c2ed9
|
Changed autocommit behaviour.
|
2017-10-04 20:56:47 +01:00 |
|
Achim D. Brucker
|
e63a13ae09
|
Bug fix: decompression.
|
2017-09-22 08:42:02 +01:00 |
|
Achim D. Brucker
|
e4245ed1dd
|
Reformatting.
|
2017-09-20 10:03:14 +01:00 |
|
Achim D. Brucker
|
a63dd53e45
|
Refactoring.
|
2017-09-20 10:02:02 +01:00 |
|
Achim D. Brucker
|
0cb0a4226d
|
Added option for passing a list with libs to update.
|
2017-09-20 07:57:14 +01:00 |
|
Michael Herzberg
|
4712e15249
|
Fixed autocommit bug.
|
2017-09-19 17:09:35 +01:00 |
|
Achim D. Brucker
|
50a7ba8a91
|
Minor refactoring.
|
2017-09-19 10:02:46 +01:00 |
|
Achim D. Brucker
|
4f84c5626d
|
Minor refactoring.
|
2017-09-19 09:16:32 +01:00 |
|
Achim D. Brucker
|
061622f588
|
Refactoring: stub of new main analysis method.
|
2017-09-18 09:09:00 +01:00 |
|
Achim D. Brucker
|
aadbc5aa0c
|
Refactoring: removed unused variables.
|
2017-09-18 00:35:35 +01:00 |
|
Achim D. Brucker
|
50b91d3a35
|
Renaming jsFilename -> filename.
|
2017-09-18 00:30:55 +01:00 |
|
Michael Herzberg
|
175ebd53b7
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-17 17:45:17 +01:00 |
|
Michael Herzberg
|
7277e6f76e
|
Fixed log msg bug.
|
2017-09-17 17:45:01 +01:00 |
|
Michael Herzberg
|
0cb7d6e792
|
Fixed error in exception handling.
|
2017-09-17 17:40:48 +01:00 |
|
Achim D. Brucker
|
3626b9fb76
|
Ordered and extended enumeration DetectionType. Order reflects reliability of checks.
|
2017-09-17 13:40:38 +01:00 |
|
Achim D. Brucker
|
a3346cb95e
|
Use file_identfiers module to compute file identifiers.
|
2017-09-17 13:18:49 +01:00 |
|
Achim D. Brucker
|
6d69377f28
|
Introduced optional parameter data to compute identifiers without opening a file handle.
|
2017-09-17 13:18:20 +01:00 |
|
Michael Herzberg
|
1fab393e56
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-16 17:23:16 +01:00 |
|
Michael Herzberg
|
c3e295267b
|
Log loglevel and only print stacktrace on first mysql exception.
|
2017-09-16 17:22:57 +01:00 |
|
Achim D. Brucker
|
205c8836e9
|
Bug fix: do not catch exceptions too aggresively and fix libvers computation for updates.
|
2017-09-16 17:20:23 +01:00 |
|
Achim D. Brucker
|
4cf41e2e4f
|
Refactoring: moved generic file identifiers into own module.
|
2017-09-16 17:19:36 +01:00 |
|
Achim D. Brucker
|
e98f58fff8
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-16 13:41:56 +01:00 |
|
Achim D. Brucker
|
24c65daecf
|
Bug fix: check for dirty missed actual function application.
|
2017-09-16 13:41:47 +01:00 |
|
Achim D. Brucker
|
c274b96f66
|
Added csv output for debugging.
|
2017-09-16 13:21:49 +01:00 |
|
Michael Herzberg
|
69e95fdf13
|
Catch json parse extensions for reviews etc. more nicely.
|
2017-09-16 12:53:35 +01:00 |
|
Michael Herzberg
|
58aacef3ff
|
Reopen connection after every exception.
|
2017-09-16 12:31:00 +01:00 |
|
Michael Herzberg
|
a514c0001e
|
Added check for empty crx files.
|
2017-09-16 12:14:41 +01:00 |
|
Michael Herzberg
|
b51de8577f
|
Added compression for mysql.
|
2017-09-16 12:04:35 +01:00 |
|
Achim D. Brucker
|
92e1c4c2e5
|
Skip deleted files.
|
2017-09-16 11:41:21 +01:00 |
|
Achim D. Brucker
|
082cd2fc65
|
Added hacking pull method that uses the regular git binary. While method will not work well with filenames containg spaces and there mit be other glitches, it allows to pull an update of the cdnjs git reposistory (> 100GB) within a couple of minutes compared to a couple of days that the non hackish solution needs.
|
2017-09-16 11:36:40 +01:00 |
|
Achim D. Brucker
|
5d3343acf1
|
Refactoring: moved git_repo creation into pull_get_list_changed_files(...).
|
2017-09-16 10:33:11 +01:00 |
|
Achim D. Brucker
|
7b0e63da10
|
Implemented n/N options for external parallelisation (only for fresh initialization).
|
2017-09-15 22:40:46 +01:00 |
|
Achim D. Brucker
|
400e74ae3f
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-15 20:21:45 +01:00 |
|
Achim D. Brucker
|
26678636eb
|
Ignore commits where blobs are None.
|
2017-09-15 20:21:05 +01:00 |
|
Michael Herzberg
|
85680d360b
|
Automatically reopen database connection on failure.
|
2017-09-15 18:23:25 +01:00 |
|
Michael Herzberg
|
ddbbc2672d
|
Try to insert also other data if some inserts fail. Use autocommit to prevent data loss on retries.
|
2017-09-15 18:15:03 +01:00 |
|
Achim D. Brucker
|
936f2d3189
|
Log git info before starting pull (update).
|
2017-09-14 22:54:37 +01:00 |
|
Achim D. Brucker
|
2ff30f7382
|
Parallel execution of git date queries.
|
2017-09-14 15:11:53 +01:00 |
|
Achim D. Brucker
|
12a1e282aa
|
The method pull_get_updated_lib_files(...) now also returns unique library/version information.
|
2017-09-14 10:44:30 +01:00 |
|
Achim D. Brucker
|
e3f1202e44
|
Use version dictionary.
|
2017-09-14 10:33:00 +01:00 |
|
Achim D. Brucker
|
f54f29c9ba
|
Added build_release_date_dic(...).
|
2017-09-14 09:50:09 +01:00 |
|
Achim D. Brucker
|
3b217922c5
|
Added line count.
|
2017-09-13 16:41:01 +01:00 |
|
Achim D. Brucker
|
420eec7462
|
Minor memory optimizations.
|
2017-09-13 11:12:33 +01:00 |
|
Achim D. Brucker
|
ec1c47625a
|
Added support for parallel update of database.
|
2017-09-13 09:13:35 +01:00 |
|
Achim D. Brucker
|
c386bd01dd
|
Added missing string conversion.
|
2017-09-13 08:29:23 +01:00 |
|
Achim D. Brucker
|
42e685ee32
|
Added missing string conversion.
|
2017-09-13 08:01:02 +01:00 |
|
Achim D. Brucker
|
18fb23d3dc
|
Use glob instead of os.walk() to avoid memory leak in the latter.
|
2017-09-13 04:04:38 +01:00 |
|
Achim D. Brucker
|
76d5993794
|
Added logging output.
|
2017-09-13 03:02:39 +01:00 |
|
Achim D. Brucker
|
c30f7fdd7c
|
Implemented skeleton of main routine.
|
2017-09-13 02:56:13 +01:00 |
|
Achim D. Brucker
|
a8a5534be1
|
Renamed module.
|
2017-09-13 01:13:17 +01:00 |
|
Achim D. Brucker
|
bdb84c2120
|
Renamed module.
|
2017-09-13 01:09:30 +01:00 |
|
Achim D. Brucker
|
4e5b52617f
|
Catch exception during decompression and increase max. allowed size of decompressed data to 100 times of compressed size.
|
2017-09-13 00:23:17 +01:00 |
|
Achim D. Brucker
|
88efe2b8a4
|
Reformatting.
|
2017-09-13 00:02:20 +01:00 |
|
Achim D. Brucker
|
ea9339bc53
|
Compute data identifiers for uncompressed content of gzip compressed files.
|
2017-09-13 00:01:15 +01:00 |
|
Achim D. Brucker
|
f9cf7bd35f
|
Refactoring: moved computation of data related identifiers into own method.
|
2017-09-12 23:52:52 +01:00 |
|
Achim D. Brucker
|
8243664974
|
Use StringIO representation for normalizing js/css files (avoid re-reading the file content from disk).
|
2017-09-12 23:43:09 +01:00 |
|
Achim D. Brucker
|
933c4d4d11
|
Determine file description from buffer instead from file (avoid reading file twice).
|
2017-09-12 23:23:22 +01:00 |
|
Achim D. Brucker
|
6353202ee8
|
Renaming: fileinfo -> filedb.
|
2017-09-10 22:59:07 +01:00 |
|
Achim D. Brucker
|
0426d7d3d1
|
Reformatting.
|
2017-09-10 22:39:47 +01:00 |
|
Achim D. Brucker
|
e5da9abaea
|
Added get_file_libinfo(...).
|
2017-09-10 22:38:49 +01:00 |
|
Achim D. Brucker
|
ad2af517a3
|
Agressively try to normalize as many filetypes as possible.
|
2017-09-10 17:40:30 +01:00 |
|
Achim D. Brucker
|
06ff5f3057
|
Method for computing basic file identifiers.
|
2017-09-10 15:57:07 +01:00 |
|
Achim D. Brucker
|
a6e90794bc
|
Extended const_basedir to check environment variable EXTENSION_ARCHIVE and modified main scripts to actually use const_basedir.
|
2017-09-10 15:55:22 +01:00 |
|
Achim D. Brucker
|
4b31097975
|
Added function for computing a list of normalized code blocks for a JavaScript file.
|
2017-09-10 15:02:57 +01:00 |
|
Achim D. Brucker
|
52b42dfaef
|
Changed pull method to return list of changed files.
|
2017-09-10 11:01:29 +01:00 |
|
Achim D. Brucker
|
c3053427c0
|
Added method for obtaining initial commit date and pulling git repos.
|
2017-09-09 23:13:26 +01:00 |
|
Achim D. Brucker
|
8c33558934
|
Reformatting.
|
2017-09-07 20:09:29 +01:00 |
|
Achim D. Brucker
|
3b2913616b
|
Skip first_seen if not defined.
|
2017-09-05 10:15:48 +01:00 |
|
Michael Herzberg
|
a9173345e8
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-04 15:54:38 +01:00 |
|
Michael Herzberg
|
36d36facfe
|
Relaxed mysql retries.
|
2017-09-04 15:54:28 +01:00 |
|
Achim D. Brucker
|
6395d98443
|
Releaxed handling of network errors.
|
2017-09-04 09:11:27 +01:00 |
|
Achim D. Brucker
|
cfeb29d95f
|
Clean-up of logging infrastructure.
|
2017-09-03 15:56:27 +01:00 |
|
Achim D. Brucker
|
f42f8e3d03
|
Improved error handling for request failures.
|
2017-09-03 15:43:33 +01:00 |
|
Achim D. Brucker
|
872346fa61
|
Add timout parameter to http get requests.
|
2017-09-03 12:03:51 +01:00 |
|
Achim D. Brucker
|
0b0268e320
|
Copy outphased date to hash map of files archive.
|
2017-09-03 11:13:27 +01:00 |
|
Achim D. Brucker
|
0f716e98da
|
Bug fix: only try to preserve outphased library information is there is any stored locally.
|
2017-09-03 11:09:39 +01:00 |
|
Achim D. Brucker
|
80c8e7caa0
|
Preserve outphased library versions.
|
2017-09-03 11:00:05 +01:00 |
|
Achim D. Brucker
|
03504ff81a
|
Improved error handling.
|
2017-09-03 10:45:56 +01:00 |
|
Achim D. Brucker
|
13191f1ce0
|
Renaming: date -> first_seen.
|
2017-09-03 10:32:45 +01:00 |
|
Achim D. Brucker
|
59f9b47a81
|
Switched to Logging framework.
|
2017-09-03 10:29:57 +01:00 |
|
Achim D. Brucker
|
074447064c
|
Enabled parallel download.
|
2017-09-03 10:06:55 +01:00 |
|
Achim D. Brucker
|
515a462938
|
Added methods for generating/updating index files based on the file hash.
|
2017-09-02 22:10:43 +01:00 |
|
Achim D. Brucker
|
9ae5905973
|
Generalized hash map builders.
|
2017-09-02 21:53:58 +01:00 |
|
Achim D. Brucker
|
22c3a7581d
|
Reformatting.
|
2017-09-02 21:44:20 +01:00 |
|
Achim D. Brucker
|
3097db3790
|
Added methods for generating sha1 indexed dictionary.
|
2017-09-02 21:40:44 +01:00 |
|
Achim D. Brucker
|
e5c2372222
|
Improved log output (verbose mode).
|
2017-09-02 20:57:01 +01:00 |
|
Achim D. Brucker
|
c32ab6bc94
|
print URL of downloaded library files in verbose mode.
|
2017-09-02 20:44:47 +01:00 |
|
Achim D. Brucker
|
ea8460f1b8
|
Updated local update.
|
2017-09-02 20:41:16 +01:00 |
|
Achim D. Brucker
|
030a4b36ca
|
Added functionality for deleting information of orphaned libraries.
|
2017-09-02 19:43:10 +01:00 |
|
Achim D. Brucker
|
247b96db6d
|
Refactoring: moved core functionality in own module.
|
2017-09-02 18:47:41 +01:00 |
|
Achim D. Brucker
|
7bcf9aca8e
|
Removed executable flag.
|
2017-09-02 18:08:20 +01:00 |
|
Achim D. Brucker
|
99028c3763
|
Removed executable flag.
|
2017-09-02 18:08:06 +01:00 |
|
Achim D. Brucker
|
9ed8f5f926
|
Improved reporting.
|
2017-09-02 00:05:07 +01:00 |
|
Achim D. Brucker
|
a69c173064
|
Activated preliminary check of regexps for specific libs.
|
2017-09-01 23:41:45 +01:00 |
|
Achim D. Brucker
|
28f6aa5f45
|
Bug fix: indentation
|
2017-09-01 23:24:55 +01:00 |
|
Achim D. Brucker
|
5c987833a4
|
Bug fix: NoneType object is not iterable.
|
2017-09-01 23:23:11 +01:00 |
|
Michael Herzberg
|
bb03a67a29
|
Deleted ropeproject stuff.
|
2017-09-01 17:04:25 +01:00 |
|
Achim D. Brucker
|
2693fb0fcd
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-01 16:28:18 +01:00 |
|
Achim D. Brucker
|
3fb0d740c0
|
Bug fix: exception due to reading from the wrong dictionary.
|
2017-09-01 16:27:44 +01:00 |
|
Michael Herzberg
|
ab943c87f0
|
Expand user directory for mysql config file.
|
2017-09-01 16:17:51 +01:00 |
|
Michael Herzberg
|
abd9605ebc
|
Use python3.5 for all files.
|
2017-09-01 14:12:05 +01:00 |
|
Michael Herzberg
|
cbcb3bc3b0
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-01 13:30:57 +01:00 |
|
Michael Herzberg
|
5c24608c4d
|
Added --max-discover <N> option to limit the number of new extensions.
|
2017-09-01 13:30:42 +01:00 |
|
Achim D. Brucker
|
258269abb6
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-01 12:14:53 +01:00 |
|
Achim D. Brucker
|
8289d50d38
|
Download all extensions in parallel and later do a second download for a subset including forums/reviews.
|
2017-09-01 12:14:39 +01:00 |
|
Michael Herzberg
|
b5fd382ab8
|
Use utf8mb4 for mysql connections.
|
2017-09-01 12:11:37 +01:00 |
|
Achim D. Brucker
|
53f080ba36
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-01 12:02:17 +01:00 |
|
Achim D. Brucker
|
22264fb9e0
|
Changed download order: first download parallel all extensions without forum/review download, then download extensions with forums.
|
2017-09-01 12:02:12 +01:00 |
|
Michael Herzberg
|
62c353f647
|
Removed crawling restriction to 10 extids.
|
2017-09-01 10:42:21 +01:00 |
|
Michael Herzberg
|
21a7741f0c
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-01 08:15:36 +01:00 |
|
Michael Herzberg
|
05ffdc6e24
|
Added explicit utf-8 request to mysql connector.
|
2017-09-01 08:15:22 +01:00 |
|
Achim D. Brucker
|
9446c20d01
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-08-31 23:45:00 +01:00 |
|
Achim D. Brucker
|
883e7ddcd8
|
Report details of matches.
|
2017-08-31 23:44:51 +01:00 |
|
Michael Herzberg
|
e06d3f4ac4
|
Reduced timeout and fixed logging.
|
2017-08-31 23:01:05 +01:00 |
|
Achim D. Brucker
|
e0db2a5f47
|
Added detection details.
|
2017-08-31 08:43:19 +01:00 |
|
Michael Herzberg
|
ccf43de3d0
|
Pad process id to 6 chars.
|
2017-08-30 20:05:17 +01:00 |
|
Michael Herzberg
|
906d81ab86
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-08-30 19:59:33 +01:00 |
|
Michael Herzberg
|
4145f88a94
|
MySQL wait +/- 20%.
|
2017-08-30 19:59:15 +01:00 |
|
Achim D. Brucker
|
e70cf5d28f
|
Bug fix: missing hex decoding of md5/sha1 hashes.
|
2017-08-30 19:25:58 +01:00 |
|
Michael Herzberg
|
b76eef12d5
|
Added randomized delay for MySQL.
|
2017-08-30 18:55:13 +01:00 |
|
Michael Herzberg
|
bde59c5040
|
Fixed crx_etag select bug and some logging.
|
2017-08-30 16:32:36 +01:00 |
|
Michael Herzberg
|
cbd2dea820
|
Removed everything related to sqlite and updated README.
|
2017-08-30 15:38:04 +01:00 |
|
Michael Herzberg
|
c7a808db3f
|
Don't process replies if there are none.
|
2017-08-30 15:15:12 +01:00 |
|
Michael Herzberg
|
5f234d8539
|
Improved logging.
|
2017-08-30 15:12:54 +01:00 |
|
Michael Herzberg
|
12c111ca11
|
Once more, make mysql the default. Also, increased timeout.
|
2017-08-30 12:13:25 +01:00 |
|
Michael Herzberg
|
9b8a693a5f
|
Changed logging a little bit.
|
2017-08-30 12:12:57 +01:00 |
|
Michael Herzberg
|
f8c8382919
|
Merge.
|
2017-08-30 11:42:12 +01:00 |
|
Michael Herzberg
|
6a9a1cda63
|
Moved crx logging to where crx will actually be parsed.
|
2017-08-30 11:29:30 +01:00 |
|
Michael Herzberg
|
d99142f8d0
|
Added and changed a few columns.
|
2017-08-30 10:07:06 +01:00 |
|
Achim D. Brucker
|
3269a4900c
|
Bug fix: printing of file name in Javascript mode.
|
2017-08-30 09:56:19 +01:00 |
|
Achim D. Brucker
|
b5b6a17ee5
|
Support analysis of crx files and plain JavaScript files.
|
2017-08-30 09:11:55 +01:00 |
|
Achim D. Brucker
|
cacdf1f727
|
Refactoring.
|
2017-08-30 08:28:39 +01:00 |
|
Achim D. Brucker
|
85d6ec084d
|
Bug fix: missing detection method for empty files.
|
2017-08-30 08:24:16 +01:00 |
|
Achim D. Brucker
|
d7120fad45
|
Bug fix: update char if loops reads another char (next_char). This avoids missing an escape character or a newline.
|
2017-08-30 01:43:18 +01:00 |
|
Achim D. Brucker
|
66818b2fa6
|
Renamed hash to md5 in JSON file and added support for sha1 hashes.
|
2017-08-30 00:38:30 +01:00 |
|
Achim D. Brucker
|
e947e69f37
|
Define type and detection method for all generated entries.
|
2017-08-30 00:24:19 +01:00 |
|
Achim D. Brucker
|
ae3bbd7339
|
Using values of enumeration to obtain nice and short human readable representations.
|
2017-08-30 00:12:57 +01:00 |
|
Michael Herzberg
|
47f424cf2f
|
Added more logging.
|
2017-08-29 23:10:46 +01:00 |
|
Michael Herzberg
|
080f00f17c
|
Added new columns for jsfile table.
|
2017-08-29 22:40:01 +01:00 |
|
Michael Herzberg
|
95d71a9edc
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-08-29 22:29:49 +01:00 |
|
Michael Herzberg
|
3e24d1f08c
|
Changed logging to use logging library.
|
2017-08-29 22:29:38 +01:00 |
|
Achim D. Brucker
|
39cd03dccc
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-08-29 18:01:42 +01:00 |
|
Achim D. Brucker
|
97f5b14158
|
Compute sha1 for JavaScript files.
|
2017-08-29 18:01:28 +01:00 |
|
Michael Herzberg
|
bddd80c138
|
Made removal of manifest.json comments stricter.
|
2017-08-29 15:43:04 +01:00 |
|
Michael Herzberg
|
7ffdf30545
|
Push manifest into table crx column manifest.
|
2017-08-29 15:41:13 +01:00 |
|
Michael Herzberg
|
2b11117b6f
|
Always process crx, regardless whether or not crx_etag is already in db.
|
2017-08-29 15:24:59 +01:00 |
|
Michael Herzberg
|
8b91957372
|
Reduced default MySQL timeout.
|
2017-08-29 15:20:58 +01:00 |
|
Michael Herzberg
|
6a99d41471
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-08-29 15:11:37 +01:00 |
|
Achim D. Brucker
|
d4ad5f96f8
|
Report empty files as own category/type.
|
2017-08-28 22:38:06 +01:00 |
|
Michael Herzberg
|
f81aac7c61
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-08-28 22:38:05 +01:00 |
|
Achim D. Brucker
|
2ace19f453
|
Compute js_info (including md5 hash and character set detection) only once per file.
|
2017-08-28 21:05:15 +01:00 |
|
Achim D. Brucker
|
91dfe67513
|
Auto-detect character encoding of JavaScript files using cchardet.
|
2017-08-28 20:53:55 +01:00 |
|
Michael Herzberg
|
c30f0c4147
|
Removed database and host setting. To be set in ~/.my.cnf file now.
|
2017-08-28 20:17:11 +01:00 |
|
Achim D. Brucker
|
5cff2bc1b7
|
New check based on file hash (md5).
|
2017-08-28 20:09:34 +01:00 |
|
Achim D. Brucker
|
030adb6adc
|
Minor refactoring and cleanup.
|
2017-08-28 19:20:50 +01:00 |
|
Michael Herzberg
|
5175d28edc
|
Convert some stuff to string for db insert.
|
2017-08-28 17:12:32 +01:00 |
|
Michael Herzberg
|
0a4e8839a1
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-08-28 11:50:49 +01:00 |
|
Michael Herzberg
|
81077b807c
|
Added mysql retry logic and use time.time() now.
|
2017-08-28 11:50:41 +01:00 |
|
Achim D. Brucker
|
9bf0b47f98
|
Minor improvement of string conversion for JsBlock.
|
2017-08-28 10:50:52 +01:00 |
|
Achim D. Brucker
|
c721e6fdbf
|
Merge with upstream.
|
2017-08-28 10:49:01 +01:00 |
|
Achim D. Brucker
|
f10923af03
|
Integreated js_mincer into decomposition analysis to allow, in the future, to check comments, code, and string literals explicitely.
|
2017-08-28 10:40:37 +01:00 |
|
Achim D. Brucker
|
9ef27f9ac9
|
Added missing return statements.
|
2017-08-28 10:28:21 +01:00 |
|
Achim D. Brucker
|
90b1db4a25
|
Added additional comment checks.
|
2017-08-28 01:26:13 +01:00 |
|
Achim D. Brucker
|
9b272c9302
|
Added option to merge subsequent single line comments into a single line comment block.
|
2017-08-28 01:17:00 +01:00 |
|
Achim D. Brucker
|
111777c821
|
Improved position counting.
|
2017-08-28 00:57:58 +01:00 |
|
Achim D. Brucker
|
d4de20efc1
|
Bug fix: start position of blocks and omit empty code blocks.
|
2017-08-28 00:19:28 +01:00 |
|
Achim D. Brucker
|
e2e92594ce
|
Bug fix: catch also last block of file.
|
2017-08-27 23:34:33 +01:00 |
|
Michael Herzberg
|
257afe92f0
|
Use selective insert.
|
2017-08-27 23:00:28 +01:00 |
|
Achim D. Brucker
|
629f492fa7
|
Added tests for code blocks and comments.
|
2017-08-27 22:58:09 +01:00 |
|
Achim D. Brucker
|
7ff1623bc6
|
Introduced JavaScript mincer working on file objects.
|
2017-08-27 22:51:55 +01:00 |
|
Michael Herzberg
|
b98b7bc0f7
|
Fixed column typo.
|
2017-08-27 22:49:07 +01:00 |
|
Achim D. Brucker
|
e324ab9483
|
Re-formatted and added documentation.
|
2017-08-27 22:41:04 +01:00 |
|
Achim D. Brucker
|
9376b4056f
|
Collect string literals in code blocks.
|
2017-08-27 22:27:35 +01:00 |
|
Achim D. Brucker
|
41ca506b9f
|
Return iterator that iterates over JavaScript blocks.
|
2017-08-27 22:17:04 +01:00 |
|
Achim D. Brucker
|
5add586da3
|
Initial commit.
|
2017-08-27 20:47:24 +01:00 |
|
Achim D. Brucker
|
f6f0bc0394
|
Renamed jsdecompose.py to js_decomposer.py.
|
2017-08-27 20:45:56 +01:00 |
|
Michael Herzberg
|
9521240d90
|
Make stuff configurable.
|
2017-08-27 18:28:19 +01:00 |
|
Michael Herzberg
|
0cff600861
|
Fixed etag keys.
|
2017-08-27 17:35:58 +01:00 |
|
Michael Herzberg
|
d4b0a6535b
|
Fixed some things.
|
2017-08-27 16:57:23 +01:00 |
|
Michael Herzberg
|
f075192b44
|
made sqlite default again.
|
2017-08-27 03:26:29 +01:00 |
|
Michael Herzberg
|
22c90dcb4f
|
Truncate timezone from timestamps for mysql, make mysql default.
|
2017-08-27 03:14:43 +01:00 |
|
Michael Herzberg
|
585c8faf0e
|
Added mysql, but still outcommented.
|
2017-08-27 02:53:15 +01:00 |
|
Michael Herzberg
|
c5c04cd1ed
|
Refactored sqlite-specifics into own class.
|
2017-08-27 00:22:19 +01:00 |
|
Achim D. Brucker
|
0bd6a55adb
|
Added documentation for analyse_filename.
|
2017-08-26 22:45:14 +01:00 |
|
Achim D. Brucker
|
df472fbbe8
|
Refactored filename check.
|
2017-08-26 22:43:57 +01:00 |
|
Achim D. Brucker
|
b2c862ede1
|
Added fields for storing evidence information for detected library/version information.
|
2017-08-25 07:07:34 +01:00 |
|
Achim D. Brucker
|
807af6f32d
|
Refactoring: proper use of enumerations.
|
2017-08-24 21:37:35 +01:00 |
|
Achim D. Brucker
|
45d2c7ad44
|
Fundamental refactoring.
|
2017-08-24 19:43:48 +01:00 |
|
Achim D. Brucker
|
676cc5ac9d
|
Renamed detectLibraries to decompose_js.
|
2017-08-24 00:47:35 +01:00 |