Achim D. Brucker
d491ff2950
|
3 years ago | |
|---|---|---|
| ExtensionCrawler | 3 years ago | |
| PermissionAnalysis | 4 years ago | |
| analysis/library-detector | 4 years ago | |
| database | 3 years ago | |
| resources | 6 years ago | |
| scripts | 3 years ago | |
| sge | 4 years ago | |
| .gitignore | 4 years ago | |
| LICENSE | 7 years ago | |
| README.md | 4 years ago | |
| cdnjs-git-miner | 4 years ago | |
| crawler | 4 years ago | |
| create-db | 4 years ago | |
| crx-extract | 4 years ago | |
| crx-jsinventory | 4 years ago | |
| crx-jsstrings | 4 years ago | |
| crx-tool | 4 years ago | |
| extgrep | 4 years ago | |
| requirements.txt | 5 years ago | |
| setup.py | 5 years ago | |
| simhashbucket | 4 years ago | |
README.md
ExtensionCrawler
A collection of utilities for downloading and analyzing browser extension from the Chrome Web store.
crawler: A crawler for extensions from the Chrome Web Store.crx-tool: A tool for analyzing and extracting*.crxfiles (i.e., Chrome extensions). Callingcrx-tool.py <extension>.crxwill check the integrity of the extension.crx-extract: A simple tool for extracting*.crxfiles from the tar-based archive hierarchy.crx-jsinventory: Build a JavaScript inventory of a*.crxfile using a JavaScript decomposition analysis.crx-jsstrings: A tool for extracting code blocks, comment blocks, and string literals from JavaScript.create-db: A tool for updating a remote MariaDB from already existing extension archives.
The utilities store the extensions in the following directory hierarchy:
archive
├── conf
│ └── forums.conf
├── data
│ └── ...
└── log
└── ...
The crawler downloads the most recent extension (i.e., the *.crx
file as well as the overview page. In addition, the conf directory
may contain one file, called forums.conf that lists the ids of
extensions for which the forums and support pages should be downloaded
as well. The data directory will contain the downloaded extensions.
The crawler and create-db scripts will access and update a MariaDB.
They will use the host, datebase, and credentials found in ~/.my.cnf.
Since they make use of various JSON features, it is recommended to use at
least version 10.2.8 of MariaDB.
All utilities are written in Python 3.7. The required modules are listed
in the file requirements.txt.
Installation
Clone and use pip3 to install as a package.
git clone git@logicalhacking.com:BrowserSecurity/ExtensionCrawler.git
pip3 install --user -e ExtensionCrawler
Team
Contributors
- Mehmet Balande
License
This project is licensed under the GPL 3.0 (or any later version).
SPDX-License-Identifier: GPL-3.0-or-later
Master Repository
The master git repository for this project is hosted by the Software Assurance & Security Research Team at https://git.logicalhacking.com/BrowserSecurity/ExtensionCrawler.