Achim D. Brucker
216fd64b90
|
||
|---|---|---|
| ExtensionCrawler | ||
| queries | ||
| scripts | ||
| sge | ||
| .gitignore | ||
| LICENSE | ||
| README.md | ||
| crawler | ||
| create-db | ||
| crx-tool | ||
| extract-crx | ||
| grepper | ||
| requirements.txt | ||
README.md
ExtensionCrawler
A collection of utilities for downloading and analyzing browser extension from the Chrome Web store.
crawler: A crawler for extensions from the Chrome Web Store.crx-tool: A tool for analyzing and extracting*.crxfiles (i.e., Chrome extensions). Callingcrx-tool.py <extension>.crxwill check the integrity of the extension.extract-crx: A simple tool for extracint*.crxfiles from the tar-based archive hierarchy.create-db: A tool for creating/initializing the database files from already existing extension archives.
The utilities store the extensions in the following directory hierarchy:
archive
├── conf
│ └── forums.conf
├── data
│ └── ...
└── log
└── ...
The crawler downloads the most recent extension (i.e., the *.crx
file as well as the overview page. In addition, the conf directory
may contain one file, called forums.conf that lists the ids of
extensions for which the forums and support pages should be downloaded
as well. The data directory will contain the downloaded extensions
as well as sqlite files containing the extracted meta data. The sqlite
files can easily be re-generated using the create-db tool.
All utilities are written in Python 3.x. The required modules are listed
in the file requirements.txt.
Team
License
This project is licensed under the GPL 3.0 (or any later version).