A Python crawler for extensions from the Chrome Web Store.
Go to file
Achim D. Brucker e5d671c7c4 Refactoring. 2017-07-29 09:05:16 +01:00
ExtensionCrawler Refactoring. 2017-07-29 09:05:16 +01:00
scripts Added python-based sqlite merger. 2017-07-28 16:47:25 +01:00
.gitignore Added .swp file to gitignore. 2017-06-20 08:09:40 +01:00
LICENSE initial commit 2016-09-08 20:43:35 +02:00
README.md Updated README.md to reflect new requirements.txt. 2017-07-26 07:41:37 +01:00
crawler Refactoring. 2017-07-28 21:18:10 +01:00
create-db Normalized script name. 2017-06-30 18:39:26 +01:00
crx-tool Refactoring. 2017-07-29 09:05:16 +01:00
extract-crx Added parameter for specifying output directory. 2017-07-07 09:18:53 +01:00
grepper Improved grepper script. 2017-07-10 14:09:45 +01:00
requirements.txt Add hoc generated requirements.txt. 2017-07-26 07:38:17 +01:00

README.md

ExtensionCrawler

A collection of utilities for downloading and analyzing browser extension from the Chrome Web store.

  • crawler: A crawler for extensions from the Chrome Web Store.
  • crx-tool: A tool for analyzing and extracting *.crx files (i.e., Chrome extensions). Calling crx-tool.py <extension>.crx will check the integrity of the extension.
  • create-db: A tool for creating/initializing the database files from already existing extension archives.

The utilities store the extensions in the following directory hierarchy:

   archive
   ├── conf
   │   └── forums.conf
   ├── data
   │   └── ...
   └── log
       └── ...

The crawler downloads the most recent extension (i.e., the *.crx file as well as the overview page. In addition, the conf directory may contain one file, called forums.conf that lists the ids of extensions for which the forums and support pages should be downloaded as well. The data directory will contain the downloaded extensions as well as sqlite files containing the extracted meta data. The sqlite files can easily be re-generated using the create-db tool.

All utilities are written in Python 3.x. The required modules are listed in the file requirements.txt.

Team

License

This project is licensed under the GPL 3.0 (or any later version).