 Achim D. Brucker
|
1ecf8f26ed
|
Update to Python 3.7.
|
2019-02-02 18:17:20 +00:00 |
Michael Herzberg
|
a789fe505f
|
Fixed style errors and warnings.
|
2018-04-21 19:00:07 +01:00 |
|
Achim D. Brucker
|
41bfe0cbf0
|
Reformatting.
|
2017-11-22 23:08:12 +00:00 |
|
Achim D. Brucker
|
06caca81e9
|
Store simhash in data base.
|
2017-11-22 06:52:31 +00:00 |
|
Achim D. Brucker
|
91e6014c6c
|
Moved to single-threaded mode.
|
2017-11-12 14:07:25 +00:00 |
|
Achim D. Brucker
|
4cb49f2281
|
Merge branch 'production'
|
2017-11-11 21:56:33 +00:00 |
|
Achim D. Brucker
|
9bd283f35a
|
Fixed use of append.
|
2017-11-10 00:13:06 +00:00 |
|
Achim D. Brucker
|
7dfbdac670
|
Disabled parallel updates (for debugging a deadlock situation).
|
2017-11-09 23:38:05 +00:00 |
|
Achim D. Brucker
|
5cc7a92f90
|
Fixed typo.
|
2017-11-09 00:17:09 +00:00 |
|
Achim D. Brucker
|
ac910bf819
|
Updated python version to 3.6.
|
2017-11-07 20:58:24 +00:00 |
|
Achim D. Brucker
|
631f461d1f
|
Removed not supported connection_timeout parameter.
|
2017-11-06 06:11:14 +00:00 |
|
Achim D. Brucker
|
6279bd9909
|
Fixed syntax error.
|
2017-11-05 20:14:12 +00:00 |
|
Achim D. Brucker
|
15079496cc
|
Merge branch 'master' of logicalhacking.com:BrowserSecurity/ExtensionCrawler
|
2017-11-05 00:07:23 +00:00 |
|
Achim D. Brucker
|
fcab770233
|
Reformatting.
|
2017-11-05 00:07:04 +00:00 |
|
Achim D. Brucker
|
cfc26e62d7
|
Free git object as early as possible.
|
2017-10-22 21:20:49 +01:00 |
|
Achim D. Brucker
|
d88e73167d
|
Explicitely free git_obj.
|
2017-10-20 18:59:53 +01:00 |
|
Achim D. Brucker
|
e4a8075da9
|
Configure timeout and retries for data base connection.
|
2017-10-18 20:19:43 +01:00 |
|
Achim D. Brucker
|
14da483046
|
Even more logging.
|
2017-10-17 15:17:29 +01:00 |
|
Achim D. Brucker
|
37ebd510c9
|
Reformatting.
|
2017-10-16 09:47:14 +01:00 |
|
Achim D. Brucker
|
4ee9c51ef7
|
Reformatting.
|
2017-10-16 09:42:43 +01:00 |
|
Achim D. Brucker
|
fc33abb7a6
|
Fixed logging.
|
2017-10-16 09:35:53 +01:00 |
|
Achim D. Brucker
|
8780eb8f2f
|
Added further logging output (info).
|
2017-10-16 05:36:59 +01:00 |
|
Achim D. Brucker
|
a4926aed19
|
Only store relative path for library files.
|
2017-10-11 09:22:27 +01:00 |
|
Achim D. Brucker
|
518372c6f2
|
Fixed library/version computation for sub-tasks.
|
2017-10-10 23:02:21 +01:00 |
|
Achim D. Brucker
|
61010a6a01
|
Bug fix: library identification for multi-task jobs.
|
2017-10-10 22:16:46 +01:00 |
 Michael Herzberg
|
6632cd0ded
|
Added database update for cdnjs.
|
2017-10-10 15:35:02 +01:00 |
|
Achim D. Brucker
|
e4245ed1dd
|
Reformatting.
|
2017-09-20 10:03:14 +01:00 |
|
Achim D. Brucker
|
a63dd53e45
|
Refactoring.
|
2017-09-20 10:02:02 +01:00 |
|
Achim D. Brucker
|
0cb0a4226d
|
Added option for passing a list with libs to update.
|
2017-09-20 07:57:14 +01:00 |
|
Achim D. Brucker
|
205c8836e9
|
Bug fix: do not catch exceptions too aggresively and fix libvers computation for updates.
|
2017-09-16 17:20:23 +01:00 |
|
Achim D. Brucker
|
4cf41e2e4f
|
Refactoring: moved generic file identifiers into own module.
|
2017-09-16 17:19:36 +01:00 |
|
Achim D. Brucker
|
24c65daecf
|
Bug fix: check for dirty missed actual function application.
|
2017-09-16 13:41:47 +01:00 |
|
Achim D. Brucker
|
c274b96f66
|
Added csv output for debugging.
|
2017-09-16 13:21:49 +01:00 |
|
Achim D. Brucker
|
92e1c4c2e5
|
Skip deleted files.
|
2017-09-16 11:41:21 +01:00 |
|
Achim D. Brucker
|
082cd2fc65
|
Added hacking pull method that uses the regular git binary. While method will not work well with filenames containg spaces and there mit be other glitches, it allows to pull an update of the cdnjs git reposistory (> 100GB) within a couple of minutes compared to a couple of days that the non hackish solution needs.
|
2017-09-16 11:36:40 +01:00 |
|
Achim D. Brucker
|
5d3343acf1
|
Refactoring: moved git_repo creation into pull_get_list_changed_files(...).
|
2017-09-16 10:33:11 +01:00 |
|
Achim D. Brucker
|
7b0e63da10
|
Implemented n/N options for external parallelisation (only for fresh initialization).
|
2017-09-15 22:40:46 +01:00 |
|
Achim D. Brucker
|
26678636eb
|
Ignore commits where blobs are None.
|
2017-09-15 20:21:05 +01:00 |
|
Achim D. Brucker
|
936f2d3189
|
Log git info before starting pull (update).
|
2017-09-14 22:54:37 +01:00 |
|
Achim D. Brucker
|
2ff30f7382
|
Parallel execution of git date queries.
|
2017-09-14 15:11:53 +01:00 |
|
Achim D. Brucker
|
12a1e282aa
|
The method pull_get_updated_lib_files(...) now also returns unique library/version information.
|
2017-09-14 10:44:30 +01:00 |
|
Achim D. Brucker
|
e3f1202e44
|
Use version dictionary.
|
2017-09-14 10:33:00 +01:00 |
|
Achim D. Brucker
|
f54f29c9ba
|
Added build_release_date_dic(...).
|
2017-09-14 09:50:09 +01:00 |
|
Achim D. Brucker
|
3b217922c5
|
Added line count.
|
2017-09-13 16:41:01 +01:00 |
|
Achim D. Brucker
|
420eec7462
|
Minor memory optimizations.
|
2017-09-13 11:12:33 +01:00 |
|
Achim D. Brucker
|
ec1c47625a
|
Added support for parallel update of database.
|
2017-09-13 09:13:35 +01:00 |
|
Achim D. Brucker
|
c386bd01dd
|
Added missing string conversion.
|
2017-09-13 08:29:23 +01:00 |
|
Achim D. Brucker
|
42e685ee32
|
Added missing string conversion.
|
2017-09-13 08:01:02 +01:00 |
|
Achim D. Brucker
|
18fb23d3dc
|
Use glob instead of os.walk() to avoid memory leak in the latter.
|
2017-09-13 04:04:38 +01:00 |
|
Achim D. Brucker
|
76d5993794
|
Added logging output.
|
2017-09-13 03:02:39 +01:00 |