 Michael Herzberg
|
a9173345e8
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-04 15:54:38 +01:00 |
|
Michael Herzberg
|
36d36facfe
|
Relaxed mysql retries.
|
2017-09-04 15:54:28 +01:00 |
 Achim D. Brucker
|
6395d98443
|
Releaxed handling of network errors.
|
2017-09-04 09:11:27 +01:00 |
|
Achim D. Brucker
|
cfeb29d95f
|
Clean-up of logging infrastructure.
|
2017-09-03 15:56:27 +01:00 |
|
Achim D. Brucker
|
f42f8e3d03
|
Improved error handling for request failures.
|
2017-09-03 15:43:33 +01:00 |
|
Achim D. Brucker
|
872346fa61
|
Add timout parameter to http get requests.
|
2017-09-03 12:03:51 +01:00 |
|
Achim D. Brucker
|
0b0268e320
|
Copy outphased date to hash map of files archive.
|
2017-09-03 11:13:27 +01:00 |
|
Achim D. Brucker
|
0f716e98da
|
Bug fix: only try to preserve outphased library information is there is any stored locally.
|
2017-09-03 11:09:39 +01:00 |
|
Achim D. Brucker
|
80c8e7caa0
|
Preserve outphased library versions.
|
2017-09-03 11:00:05 +01:00 |
|
Achim D. Brucker
|
03504ff81a
|
Improved error handling.
|
2017-09-03 10:45:56 +01:00 |
|
Achim D. Brucker
|
13191f1ce0
|
Renaming: date -> first_seen.
|
2017-09-03 10:32:45 +01:00 |
|
Achim D. Brucker
|
59f9b47a81
|
Switched to Logging framework.
|
2017-09-03 10:29:57 +01:00 |
|
Achim D. Brucker
|
074447064c
|
Enabled parallel download.
|
2017-09-03 10:06:55 +01:00 |
|
Achim D. Brucker
|
e3aa92f1b8
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-02 22:15:36 +01:00 |
|
Achim D. Brucker
|
515a462938
|
Added methods for generating/updating index files based on the file hash.
|
2017-09-02 22:10:43 +01:00 |
|
Achim D. Brucker
|
9ae5905973
|
Generalized hash map builders.
|
2017-09-02 21:53:58 +01:00 |
|
Achim D. Brucker
|
22c3a7581d
|
Reformatting.
|
2017-09-02 21:44:20 +01:00 |
|
Achim D. Brucker
|
3097db3790
|
Added methods for generating sha1 indexed dictionary.
|
2017-09-02 21:40:44 +01:00 |
|
Achim D. Brucker
|
e5c2372222
|
Improved log output (verbose mode).
|
2017-09-02 20:57:01 +01:00 |
|
Achim D. Brucker
|
c32ab6bc94
|
print URL of downloaded library files in verbose mode.
|
2017-09-02 20:44:47 +01:00 |
|
Achim D. Brucker
|
ea8460f1b8
|
Updated local update.
|
2017-09-02 20:41:16 +01:00 |
|
Achim D. Brucker
|
030a4b36ca
|
Added functionality for deleting information of orphaned libraries.
|
2017-09-02 19:43:10 +01:00 |
|
Achim D. Brucker
|
247b96db6d
|
Refactoring: moved core functionality in own module.
|
2017-09-02 18:47:41 +01:00 |
|
Achim D. Brucker
|
7bcf9aca8e
|
Removed executable flag.
|
2017-09-02 18:08:20 +01:00 |
|
Achim D. Brucker
|
99028c3763
|
Removed executable flag.
|
2017-09-02 18:08:06 +01:00 |
|
Achim D. Brucker
|
6b3ef921ff
|
Reformatting and minor refactoring.
|
2017-09-02 18:00:24 +01:00 |
|
Michael Herzberg
|
45496a0d5d
|
Log parameters.
|
2017-09-02 17:52:51 +01:00 |
|
Michael Herzberg
|
d7dcfdbcbd
|
Use $* instead of $@.
|
2017-09-02 17:50:16 +01:00 |
|
Michael Herzberg
|
54475b97a8
|
Added arg option to sge script.
|
2017-09-02 17:45:12 +01:00 |
|
Achim D. Brucker
|
7e07c6d734
|
Initital commit: tool for crawling cdnjs.com.
|
2017-09-02 17:42:46 +01:00 |
|
Michael Herzberg
|
c94f23dcee
|
Added --from-date option for create-db.
|
2017-09-02 17:42:18 +01:00 |
|
Michael Herzberg
|
c33e8204ea
|
Cleaned up create-db sge script a bit.
|
2017-09-02 17:05:42 +01:00 |
|
Michael Herzberg
|
1647aac086
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-02 16:46:04 +01:00 |
|
Michael Herzberg
|
08fced735c
|
Fixed queries.
|
2017-09-02 16:45:38 +01:00 |
|
Michael Herzberg
|
f94f6140b7
|
Added same query for content script urls.
|
2017-09-02 14:27:01 +01:00 |
|
Michael Herzberg
|
1137421548
|
Added download numbers to query.
|
2017-09-02 14:16:11 +01:00 |
|
Achim D. Brucker
|
8af3c99d26
|
Changed pip to pip3.
|
2017-09-02 00:07:50 +01:00 |
|
Achim D. Brucker
|
9ed8f5f926
|
Improved reporting.
|
2017-09-02 00:05:07 +01:00 |
|
Achim D. Brucker
|
8faba1d00f
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-01 23:45:51 +01:00 |
|
Achim D. Brucker
|
a69c173064
|
Activated preliminary check of regexps for specific libs.
|
2017-09-01 23:41:45 +01:00 |
|
Achim D. Brucker
|
28f6aa5f45
|
Bug fix: indentation
|
2017-09-01 23:24:55 +01:00 |
|
Achim D. Brucker
|
5c987833a4
|
Bug fix: NoneType object is not iterable.
|
2017-09-01 23:23:11 +01:00 |
|
Achim D. Brucker
|
faaa458921
|
Updated regexps.
|
2017-09-01 22:26:30 +01:00 |
|
Michael Herzberg
|
f125f683ec
|
Fixed query.
|
2017-09-01 20:10:50 +01:00 |
|
Michael Herzberg
|
f20da9490f
|
Updated README and setup.py from my experiments.
|
2017-09-01 20:01:53 +01:00 |
|
Michael Herzberg
|
4980bdbe9e
|
Updated query for MySQL.
|
2017-09-01 20:00:28 +01:00 |
|
Michael Herzberg
|
bb03a67a29
|
Deleted ropeproject stuff.
|
2017-09-01 17:04:25 +01:00 |
|
Achim D. Brucker
|
2693fb0fcd
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-09-01 16:28:18 +01:00 |
|
Achim D. Brucker
|
3fb0d740c0
|
Bug fix: exception due to reading from the wrong dictionary.
|
2017-09-01 16:27:44 +01:00 |
|
Michael Herzberg
|
ab943c87f0
|
Expand user directory for mysql config file.
|
2017-09-01 16:17:51 +01:00 |