BrowserSecurity
/
ExtensionCrawler
2
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Added database documentation.

This commit is contained in:
Michael Herzberg 2018-08-09 14:20:01 +01:00
parent e492f516ac
commit e7b7625453
32 changed files with 1242 additions and 105 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

134
database/README.md Normal file
View File

@ -0,0 +1,134 @@
# Introduction
The extension crawler downloads all metadata and extension files into tar files.
This is great for archival, but not so great for analyzing the data. The crawler
therefore also supports inserting all newly crawled information into a MariaDB
database. Additionally, there exists a script to regenerate the database from
old tar files.
# Setting up the database
## Hardware requirements
The database is meant to be setup on a (old) PC, although it should also work
with common cloud offerings.
The amount of data that the database needs to handle grows over time. Currently,
containing ~18 months worth of data, the database requires ~150GB of space.
It is recommended to have at least 16GB of RAM to keep the indices available;
less RAM might work, more RAM will certainly speed queries up. It is also good
to have at least 16GB of swap; while this detrimental to the performance of
MariaDB, it is often better than it being killed by the OS.
For storage, it is beneficial to have at least one HDD and one SSD, as the
database workload can be split into sequential and random IO.
## Configuration
A commented configuration file for MariaDB can be found in `config/my.cnf`.
Configuration options such as pool size and storage locations will need to be
adjusted.
## Table schemas
To set up the tables and schemas, make sure that you have the credentials for
root in your `~/.my.cnf` file, and execute the following:
```bash
mysql -e "create database extensions;"
for f in schemas/*.sql; do mysql extensions < $f; done
for f in views/*.sql; do mysql extensions < $f; done
```
# Maintaining the database
## Memory consumption
MariaDB will, at times, use much more memory than specified for the pool size --
100GB with a pool size of 4GB is certainly possible while regenerating the data.
In these cases, the database should be restarted. The crawler and regeneration
script will retry their database operations by default for around one hour.
## Backup
Regenerating the whole data set can take days, if not weeks, so even though all
data can be restored, having a backup speeds up recovery. For this purpose, the
MariaDB binary log is enabled to allow physical backups, which are much faster
than logical backups for our case. The folder `scripts/` contains scripts to do
full and incremental backups, as well as scripts to backup the schemas and users
(including permissions and hashed passwords).
# Regenerating extension data
When the crawler is changed to extract more or different data from the
extensions, one will probably want to regenerate all data, i.e., ask the crawler
to go through all existing tar files and re-extract the already downloaded data.
In order to do so, the `create-db` or `sge/create-db.sh` (for HPCs) can be used.
More information can be found when calling these scripts with `--help`.
# Using the data set
## Example queries
For more (commented) queries, see the `queries/` folder.
- ```sql
select extid,crx_etag,count(filename) from extension_most_recent_small join crxfile using (crx_etag) where filename like '%.js' group by extid,crx_etag limit 10;
```
This query will print the number of JavaScript files per extension.
## Table schemas
All schema files can be found in the `schemas/` folder.
| Table name | Description |
| --- | --- |
| extension | General extension metadata from the store pages. One row per \
extension and crawldate (!). If you are only interested in the most recent \
*view* of the Chrome Web Store, use the `extension_most_recent` view. For \
testing your queries, suffix either table/view with *\_small* to only get \
roughly 1/256th of all extensions. |
| status | The HTTP status codes for the store page and `crx` download. |
| crx | General metadata of the extension file (the `crx` archive itself). Also \
contains the manifest. |
| crxfile | General metadata of the extension files, e.g., the files contained \
in the `crx` archives (JavaScript files, etc.).|
| category | Categories of the extensions, e.g. *productivity*, *office*, \
or *game*. |
| permission | Permissions found in the manifests, e.g., *webRequest*, *tab*, but also \
host permissions such as *https://www.google.com*. |
| content_script_url | Content script URLs found in the manifest. These are the \
URLs where the extensions request to have a content script executed when the \
user visits the website. |
| libdet | Information about used libraries. For each file found in `crx` \
archives (identified by MD5 sums), this table stores classifications of the \
file, e.g., whether it is a certain library. |
| review{,\_comment} | Post-metadata and posts from the review forum of an extension. |
| support{,\_comment} | Post-metadata and posts from the support forum of an extension. |
| reply{,\_comment} | Reply-post-metadata and posts for both the review and support forums. |
## Views
All views can be found in the `views/` folder.
| View name | Description |
| --- | --- |
| extension_small | Contains only roughly 1/256th of all extensions. |
| extension_most_recent | Instead of one row for every combination of extension \
id and crawl date, this view only contains the rows from the most recent crawl \
date. |
| extension_most_recent_small | Same, but roughly only 1/256th of all extensions. |
| extension_second_most_recent | Similar to `extension_most_recent`, but \
contains the second-most recent entry for all extensions. This is useful for \
investigating how extensions change. |
| extension_{most,second_most}_recent_until_date | Parameterized query. Only \
considers extensions crawled before a given date. Usage: \
```sql
select * from (select @until_date:='2018-05-25') foo, extension_most_recent_until_date;
``` |
| extension_update | Selects all extension updates in the database. A row in the result represents \
one extension update, with the date and crx_etag when we have first seen the \
update, and the date and crx_etag when we have last seen the old version. As \
we crawl every night, the difference should be around 24 hours on average. |

61
database/config/my.cnf Normal file
View File

@ -0,0 +1,61 @@
[client]
port = 3306
socket = /run/mysqld/mysqld.sock
[mysqld]
port = 3306
socket = /run/mysqld/mysqld.sock
wait_timeout=1800
max_connections=1000
explicit_defaults_for_timestamp=1
default_time_zone='+00:00'
server-id = 1
expire_logs_days=8
log-basename=master1-bin
# Ideally, the MariaDB datadir resides on a HDD, as there will be a lot of sequential IO.
# After creating a database, it is best moved to a SSD, as there will be a lot of
# random IO. This can be done by simply moving the directory (do NOT move individual table
# files!), e.g.: cd /hdd/mysql; mv extensions /ssd/databases/; ln -s /ssd/databases/extensions
datadir=/hdd/mysql
# When adding indices, MariaDB uses a lot of space in /tmp. If that space is not enough, the
# used tmpdir can be moved:
innodb_tmpdir=/ssd/innodb_tmp
# The pool size is said to be around 75% of the available RAM on db-only hosts. However, current
# versions of MariaDB seem to have serious memory leaks when doing a lot of concurrent writes.
# Therefore, expect MariaDB to use a lot more memory, create sufficient swap to prevent killing,
# and restart MariaDB when the usage grows too high.
innodb_buffer_pool_size = 18G
# General performance tweaks
innodb_read_io_threads=8
innodb_write_io_threads=8
innodb_sort_buffer_size=67108864
innodb_log_file_size=256M
innodb_log_buffer_size=256M
# Performance tweaks for inserts
#innodb_flush_log_at_trx_commit=0
#innodb_change_buffer_max_size=50
#innodb_flush_method=O_DIRECT
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[myisamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout

16
database/queries/get_added_content_scripts.sql Normal file
View File

@ -0,0 +1,16 @@
select downloads, eu.extid, name, url, new_crx_etag
from extension_update eu join extension e on eu.extid=e.extid and eu.first_date_with_new_crx_etag=e.date
join content_script_url c on eu.new_crx_etag=c.crx_etag
where
url in (
"file://*/*",
"http://*/*",
"https://*/*",
"*://*/*",
"<all_urls>"
)
and
url not in (select url from content_script_url where crx_etag=previous_crx_etag)
and
first_date_with_new_crx_etag > NOW() - INTERVAL 2 DAY
order by downloads desc;

16
database/queries/get_added_permissions.sql Normal file
View File

@ -0,0 +1,16 @@
select downloads, eu.extid, name, permission, new_crx_etag
from extension_update eu join extension e on eu.extid=e.extid and eu.first_date_with_new_crx_etag=e.date
join permission p on eu.new_crx_etag=p.crx_etag
where
permission in (
"<all_url>",
"http://*/*",
"https://*/*",
"webRequest",
"webRequestBlocking"
)
and
permission not in (select permission from permission where crx_etag=previous_crx_etag)
and
first_date_with_new_crx_etag > NOW() - INTERVAL 2 DAY
order by downloads desc;

41
database/schemas/category.sql Normal file
View File

@ -0,0 +1,41 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `category`
--
DROP TABLE IF EXISTS `category`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `category` (
`extid` varchar(32) COLLATE utf8mb4_unicode_ci NOT NULL,
`date` timestamp(6) NOT NULL,
`category_md5` varbinary(16) NOT NULL,
`category` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`extid`,`date`,`category_md5`) KEY_BLOCK_SIZE=8
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

54
database/schemas/cdnjs.sql Normal file
View File

@ -0,0 +1,54 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `cdnjs`
--
DROP TABLE IF EXISTS `cdnjs`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `cdnjs` (
`path` varchar(512) COLLATE utf8mb4_unicode_ci NOT NULL,
`typ` enum('AS_IS','NORMALIZED','DECOMPRESSED','DECOMPRESSED_NORMALIZED') COLLATE utf8mb4_unicode_ci NOT NULL,
`md5` varbinary(16) NOT NULL,
`filename` varchar(253) /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`sha1` varbinary(20) DEFAULT NULL,
`sha256` varbinary(32) DEFAULT NULL,
`simhash` varbinary(64) DEFAULT NULL,
`size` bigint(20) DEFAULT NULL,
`loc` bigint(20) DEFAULT NULL,
`description` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`encoding` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`mimetype` varchar(126) /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`add_date` timestamp(6) NULL DEFAULT NULL,
`library` varchar(254) /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`version` varchar(30) /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`mimetype_detail` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`path`,`typ`),
KEY `cdnjs_md5_typ` (`md5`,`typ`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

40
database/schemas/content_script_url.sql Normal file
View File

@ -0,0 +1,40 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `content_script_url`
--
DROP TABLE IF EXISTS `content_script_url`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `content_script_url` (
`crx_etag` varchar(44) COLLATE utf8mb4_unicode_ci NOT NULL,
`url_md5` varbinary(16) NOT NULL,
`url` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`crx_etag`,`url_md5`) KEY_BLOCK_SIZE=8
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

42
database/schemas/crx.sql Normal file
View File

@ -0,0 +1,42 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `crx`
--
DROP TABLE IF EXISTS `crx`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `crx` (
`crx_etag` varchar(44) COLLATE utf8mb4_unicode_ci NOT NULL,
`filename` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
`size` int(11) NOT NULL,
`publickey` blob NOT NULL,
`manifest` longtext /*!100301 COMPRESSED*/ CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '',
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`crx_etag`) KEY_BLOCK_SIZE=8
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

48
database/schemas/crxfile.sql Normal file
View File

@ -0,0 +1,48 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `crxfile`
--
DROP TABLE IF EXISTS `crxfile`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `crxfile` (
`crx_etag` varchar(44) COLLATE utf8mb4_unicode_ci NOT NULL,
`path` varchar(512) COLLATE utf8mb4_unicode_ci NOT NULL,
`typ` enum('AS_IS','NORMALIZED','DECOMPRESSED','DECOMPRESSED_NORMALIZED') COLLATE utf8mb4_unicode_ci NOT NULL,
`md5` varbinary(16) DEFAULT NULL,
`filename` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`sha1` varbinary(20) DEFAULT NULL,
`sha256` varbinary(32) DEFAULT NULL,
`simhash` varbinary(64) DEFAULT NULL,
`mimetype` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`mimetype_detail` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`crx_etag`,`path`,`typ`),
KEY `crxfile_md5_typ` (`md5`,`typ`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

54
database/schemas/extension.sql Normal file
View File

@ -0,0 +1,54 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `extension`
--
DROP TABLE IF EXISTS `extension`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `extension` (
`extid` varchar(32) COLLATE utf8mb4_unicode_ci NOT NULL,
`date` timestamp(6) NOT NULL,
`name` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`version` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`description` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`downloads` int(11) DEFAULT NULL,
`rating` double DEFAULT NULL,
`ratingcount` int(11) DEFAULT NULL,
`fulldescription` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`developer` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`itemcategory` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`crx_etag` varchar(44) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`lastupdated` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`extid`,`date`) KEY_BLOCK_SIZE=8,
KEY `extension_crx_etag` (`crx_etag`),
KEY `extension_date` (`date`),
KEY `extension_date_extid` (`date`,`extid`),
KEY `extension_extid_crx_etag` (`extid`,`crx_etag`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

56
database/schemas/libdet.sql Normal file
View File

@ -0,0 +1,56 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `libdet`
--
DROP TABLE IF EXISTS `libdet`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `libdet` (
`md5` varbinary(16) NOT NULL,
`typ` enum('AS_IS','NORMALIZED','DECOMPRESSED','DECOMPRESSED_NORMALIZED') COLLATE utf8mb4_unicode_ci NOT NULL,
`sha1` varbinary(20) DEFAULT NULL,
`sha256` varbinary(32) DEFAULT NULL,
`size` bigint(20) DEFAULT NULL,
`loc` bigint(20) DEFAULT NULL,
`description` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`encoding` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`mimetype` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`library` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`version` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`classification_type` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`detect_method` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`detect_method_details` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`evidence_start_pos` bigint(20) DEFAULT NULL,
`evidence_end_pos` bigint(20) DEFAULT NULL,
`evidence_text` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`mimetype_detail` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`mimetype_magic` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`md5`,`typ`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

40
database/schemas/permission.sql Normal file
View File

@ -0,0 +1,40 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `permission`
--
DROP TABLE IF EXISTS `permission`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `permission` (
`crx_etag` varchar(44) COLLATE utf8mb4_unicode_ci NOT NULL,
`permission_md5` varbinary(16) NOT NULL,
`permission` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`crx_etag`,`permission_md5`) KEY_BLOCK_SIZE=8
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

46
database/schemas/reply.sql Normal file
View File

@ -0,0 +1,46 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `reply`
--
DROP TABLE IF EXISTS `reply`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `reply` (
`extid` varchar(32) COLLATE utf8mb4_unicode_ci NOT NULL,
`date` timestamp(6) NOT NULL,
`author` varchar(98) COLLATE utf8mb4_unicode_ci NOT NULL,
`commentdate` timestamp NOT NULL,
`displayname` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`replyto` varchar(98) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`language` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`shortauthor` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`commentmd5` varbinary(16) DEFAULT NULL,
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`extid`,`date`,`author`,`commentdate`) KEY_BLOCK_SIZE=8
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

39
database/schemas/reply_comment.sql Normal file
View File

@ -0,0 +1,39 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `reply_comment`
--
DROP TABLE IF EXISTS `reply_comment`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `reply_comment` (
`commentmd5` varbinary(16) NOT NULL,
`comment` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`commentmd5`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

46
database/schemas/review.sql Normal file
View File

@ -0,0 +1,46 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `review`
--
DROP TABLE IF EXISTS `review`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `review` (
`extid` varchar(32) COLLATE utf8mb4_unicode_ci NOT NULL,
`date` timestamp(6) NOT NULL,
`author` varchar(98) COLLATE utf8mb4_unicode_ci NOT NULL,
`commentdate` timestamp NOT NULL,
`displayname` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`rating` double DEFAULT NULL,
`language` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`shortauthor` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`commentmd5` varbinary(16) DEFAULT NULL,
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`extid`,`date`,`author`,`commentdate`) KEY_BLOCK_SIZE=8
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

39
database/schemas/review_comment.sql Normal file
View File

@ -0,0 +1,39 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `review_comment`
--
DROP TABLE IF EXISTS `review_comment`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `review_comment` (
`commentmd5` varbinary(16) NOT NULL,
`comment` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`commentmd5`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

42
database/schemas/status.sql Normal file
View File

@ -0,0 +1,42 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `status`
--
DROP TABLE IF EXISTS `status`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `status` (
`extid` varchar(32) COLLATE utf8mb4_unicode_ci NOT NULL,
`date` timestamp(6) NOT NULL,
`crx_status` int(11) DEFAULT NULL,
`overview_status` int(11) DEFAULT NULL,
`overview_exception` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`extid`,`date`) KEY_BLOCK_SIZE=8
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

46
database/schemas/support.sql Normal file
View File

@ -0,0 +1,46 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `support`
--
DROP TABLE IF EXISTS `support`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `support` (
`extid` varchar(32) COLLATE utf8mb4_unicode_ci NOT NULL,
`date` timestamp(6) NOT NULL,
`author` varchar(98) COLLATE utf8mb4_unicode_ci NOT NULL,
`commentdate` timestamp NOT NULL,
`displayname` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`title` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`language` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`shortauthor` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`commentmd5` varbinary(16) DEFAULT NULL,
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`extid`,`date`,`author`,`commentdate`) KEY_BLOCK_SIZE=8
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

39
database/schemas/support_comment.sql Normal file
View File

@ -0,0 +1,39 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `support_comment`
--
DROP TABLE IF EXISTS `support_comment`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `support_comment` (
`commentmd5` varbinary(16) NOT NULL,
`comment` text /*!100301 COMPRESSED*/ COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`last_modified` timestamp NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`commentmd5`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci `PAGE_COMPRESSED`='ON';
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

6
database/scripts/mariabackup-full Executable file
View File

@ -0,0 +1,6 @@
#!/usr/bin/env bash
set -o nounset
set -o errexit
/usr/bin/mariabackup --backup --stream=xbstream --parallel=4 --compress --compress-threads=2

12
database/scripts/mariabackup-inc Executable file
View File

@ -0,0 +1,12 @@
#!/usr/bin/env bash
set -o nounset
set -o errexit
LSN=$1
if ! [[ "$LSN" =~ ^[0-9]+$ ]]; then
>&2 echo "Invalid LSN: $LSN"
exit 1
fi
/usr/bin/mariabackup --backup --stream=xbstream --parallel=4 --compress --compress-threads=2 --incremental-lsn=$LSN

13
database/scripts/mariabackup-schemas Executable file
View File

@ -0,0 +1,13 @@
#!/usr/bin/bash
set -o errexit
set -o nounset
T=$(mktemp -d)
for db in $(mysql -N -e "show databases" | grep -v -e "^mysql$" -e "^information_schema$" -e "^performance_schema$")
do
mkdir -p $T/schemas/$db
mysqldump $db --no-data --single-transaction --tab=$T/schemas/$db
done
(cd $T; tar cz *)
rm -r $T

5
database/scripts/showgrants Executable file
View File

@ -0,0 +1,5 @@
#!/usr/bin/env bash
set -o errexit
set -o nounset
mysql "" --skip-column-names -A -e"SELECT CONCAT('SHOW GRANTS FOR ''',user,'''@''',host,''';') FROM mysql.user WHERE user<>''" | mysql "" --skip-column-names -A | sed 's/$/;/g'

43
database/views/extension_most_recent.sql Normal file
View File

@ -0,0 +1,43 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Final view structure for view `extension_most_recent`
--
/*!50001 DROP TABLE IF EXISTS `extension_most_recent`*/;
/*!50001 DROP VIEW IF EXISTS `extension_most_recent`*/;
/*!50001 SET @saved_cs_client = @@character_set_client */;
/*!50001 SET @saved_cs_results = @@character_set_results */;
/*!50001 SET @saved_col_connection = @@collation_connection */;
/*!50001 SET character_set_client = utf8 */;
/*!50001 SET character_set_results = utf8 */;
/*!50001 SET collation_connection = utf8_general_ci */;
/*!50001 CREATE ALGORITHM=UNDEFINED */
/*!50013 DEFINER=`writer`@`%` SQL SECURITY DEFINER */
/*!50001 VIEW `extension_most_recent` AS select `e3`.`extid` AS `extid`,`e3`.`date` AS `date`,`e3`.`name` AS `name`,`e3`.`version` AS `version`,`e3`.`description` AS `description`,`e3`.`downloads` AS `downloads`,`e3`.`rating` AS `rating`,`e3`.`ratingcount` AS `ratingcount`,`e3`.`fulldescription` AS `fulldescription`,`e3`.`developer` AS `developer`,`e3`.`itemcategory` AS `itemcategory`,`e3`.`crx_etag` AS `crx_etag`,`e3`.`lastupdated` AS `lastupdated` from (((select `e1`.`extid` AS `extid`,max(`e1`.`date`) AS `date` from `extensions`.`extension` `e1` group by `e1`.`extid`)) `e2` join `extensions`.`extension` `e3` on(`e2`.`extid` = `e3`.`extid` and `e2`.`date` = `e3`.`date`)) */;
/*!50001 SET character_set_client = @saved_cs_client */;
/*!50001 SET character_set_results = @saved_cs_results */;
/*!50001 SET collation_connection = @saved_col_connection */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

43
database/views/extension_most_recent_small.sql Normal file
View File

@ -0,0 +1,43 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Final view structure for view `extension_most_recent_small`
--
/*!50001 DROP TABLE IF EXISTS `extension_most_recent_small`*/;
/*!50001 DROP VIEW IF EXISTS `extension_most_recent_small`*/;
/*!50001 SET @saved_cs_client = @@character_set_client */;
/*!50001 SET @saved_cs_results = @@character_set_results */;
/*!50001 SET @saved_col_connection = @@collation_connection */;
/*!50001 SET character_set_client = utf8 */;
/*!50001 SET character_set_results = utf8 */;
/*!50001 SET collation_connection = utf8_general_ci */;
/*!50001 CREATE ALGORITHM=UNDEFINED */
/*!50013 DEFINER=`writer`@`%` SQL SECURITY DEFINER */
/*!50001 VIEW `extension_most_recent_small` AS select `e3`.`extid` AS `extid`,`e3`.`date` AS `date`,`e3`.`name` AS `name`,`e3`.`version` AS `version`,`e3`.`description` AS `description`,`e3`.`downloads` AS `downloads`,`e3`.`rating` AS `rating`,`e3`.`ratingcount` AS `ratingcount`,`e3`.`fulldescription` AS `fulldescription`,`e3`.`developer` AS `developer`,`e3`.`itemcategory` AS `itemcategory`,`e3`.`crx_etag` AS `crx_etag`,`e3`.`lastupdated` AS `lastupdated` from (((select `e1`.`extid` AS `extid`,max(`e1`.`date`) AS `date` from `extensions`.`extension` `e1` where `e1`.`extid` like 'aa%' group by `e1`.`extid`)) `e2` join `extensions`.`extension` `e3` on(`e2`.`extid` = `e3`.`extid` and `e2`.`date` = `e3`.`date`)) */;
/*!50001 SET character_set_client = @saved_cs_client */;
/*!50001 SET character_set_results = @saved_cs_results */;
/*!50001 SET collation_connection = @saved_col_connection */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

46
database/views/extension_most_recent_until_date.sql Normal file
View File

@ -0,0 +1,46 @@
drop function if exists until_date;
create function until_date returns datetime NO SQL DEERMINISTIC return @until_date;
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Final view structure for view `extension_most_recent_until_date`
--
/*!50001 DROP TABLE IF EXISTS `extension_most_recent_until_date`*/;
/*!50001 DROP VIEW IF EXISTS `extension_most_recent_until_date`*/;
/*!50001 SET @saved_cs_client = @@character_set_client */;
/*!50001 SET @saved_cs_results = @@character_set_results */;
/*!50001 SET @saved_col_connection = @@collation_connection */;
/*!50001 SET character_set_client = utf8 */;
/*!50001 SET character_set_results = utf8 */;
/*!50001 SET collation_connection = utf8_general_ci */;
/*!50001 CREATE ALGORITHM=UNDEFINED */
/*!50013 DEFINER=`root`@`localhost` SQL SECURITY DEFINER */
/*!50001 VIEW `extension_most_recent_until_date` AS select `e1`.`extid` AS `extid`,`e1`.`date` AS `date`,`extensions`.`extension`.`name` AS `name`,`extensions`.`extension`.`version` AS `version`,`extensions`.`extension`.`description` AS `description`,`extensions`.`extension`.`downloads` AS `downloads`,`extensions`.`extension`.`rating` AS `rating`,`extensions`.`extension`.`ratingcount` AS `ratingcount`,`extensions`.`extension`.`fulldescription` AS `fulldescription`,`extensions`.`extension`.`developer` AS `developer`,`extensions`.`extension`.`itemcategory` AS `itemcategory`,`extensions`.`extension`.`crx_etag` AS `crx_etag`,`extensions`.`extension`.`lastupdated` AS `lastupdated`,`extensions`.`extension`.`last_modified` AS `last_modified` from (((select `extensions`.`extension`.`extid` AS `extid`,max(`extensions`.`extension`.`date`) AS `date` from `extensions`.`extension` where `extensions`.`extension`.`date` <= `until_date`() group by `extensions`.`extension`.`extid`)) `e1` join `extensions`.`extension` on(`e1`.`extid` = `extensions`.`extension`.`extid` and `e1`.`date` = `extensions`.`extension`.`date`)) */;
/*!50001 SET character_set_client = @saved_cs_client */;
/*!50001 SET character_set_results = @saved_cs_results */;
/*!50001 SET collation_connection = @saved_col_connection */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

43
database/views/extension_second_most_recent.sql Normal file
View File

@ -0,0 +1,43 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Final view structure for view `extension_second_most_recent`
--
/*!50001 DROP TABLE IF EXISTS `extension_second_most_recent`*/;
/*!50001 DROP VIEW IF EXISTS `extension_second_most_recent`*/;
/*!50001 SET @saved_cs_client = @@character_set_client */;
/*!50001 SET @saved_cs_results = @@character_set_results */;
/*!50001 SET @saved_col_connection = @@collation_connection */;
/*!50001 SET character_set_client = utf8 */;
/*!50001 SET character_set_results = utf8 */;
/*!50001 SET collation_connection = utf8_general_ci */;
/*!50001 CREATE ALGORITHM=UNDEFINED */
/*!50013 DEFINER=`root`@`localhost` SQL SECURITY DEFINER */
/*!50001 VIEW `extension_second_most_recent` AS select `e1`.`extid` AS `extid`,`e1`.`date` AS `date`,`extensions`.`extension`.`name` AS `name`,`extensions`.`extension`.`version` AS `version`,`extensions`.`extension`.`description` AS `description`,`extensions`.`extension`.`downloads` AS `downloads`,`extensions`.`extension`.`rating` AS `rating`,`extensions`.`extension`.`ratingcount` AS `ratingcount`,`extensions`.`extension`.`fulldescription` AS `fulldescription`,`extensions`.`extension`.`developer` AS `developer`,`extensions`.`extension`.`itemcategory` AS `itemcategory`,`extensions`.`extension`.`crx_etag` AS `crx_etag`,`extensions`.`extension`.`lastupdated` AS `lastupdated`,`extensions`.`extension`.`last_modified` AS `last_modified` from (((select `extensions`.`extension`.`extid` AS `extid`,max(`extensions`.`extension`.`date`) AS `date` from `extensions`.`extension` where !((`extensions`.`extension`.`extid`,`extensions`.`extension`.`date`) in (select `extensions`.`extension`.`extid`,max(`extensions`.`extension`.`date`) AS `date` from `extensions`.`extension` group by `extensions`.`extension`.`extid`)) group by `extensions`.`extension`.`extid`)) `e1` join `extensions`.`extension` on(`e1`.`extid` = `extensions`.`extension`.`extid` and `e1`.`date` = `extensions`.`extension`.`date`)) */;
/*!50001 SET character_set_client = @saved_cs_client */;
/*!50001 SET character_set_results = @saved_cs_results */;
/*!50001 SET collation_connection = @saved_col_connection */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

46
database/views/extension_second_most_recent_until_date.sql Normal file
View File

@ -0,0 +1,46 @@
drop function if exists until_date;
create function until_date returns datetime NO SQL DEERMINISTIC return @until_date;
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Final view structure for view `extension_second_most_recent_until_date`
--
/*!50001 DROP TABLE IF EXISTS `extension_second_most_recent_until_date`*/;
/*!50001 DROP VIEW IF EXISTS `extension_second_most_recent_until_date`*/;
/*!50001 SET @saved_cs_client = @@character_set_client */;
/*!50001 SET @saved_cs_results = @@character_set_results */;
/*!50001 SET @saved_col_connection = @@collation_connection */;
/*!50001 SET character_set_client = utf8 */;
/*!50001 SET character_set_results = utf8 */;
/*!50001 SET collation_connection = utf8_general_ci */;
/*!50001 CREATE ALGORITHM=UNDEFINED */
/*!50013 DEFINER=`root`@`localhost` SQL SECURITY DEFINER */
/*!50001 VIEW `extension_second_most_recent_until_date` AS select `e1`.`extid` AS `extid`,`e1`.`date` AS `date`,`extensions`.`extension`.`name` AS `name`,`extensions`.`extension`.`version` AS `version`,`extensions`.`extension`.`description` AS `description`,`extensions`.`extension`.`downloads` AS `downloads`,`extensions`.`extension`.`rating` AS `rating`,`extensions`.`extension`.`ratingcount` AS `ratingcount`,`extensions`.`extension`.`fulldescription` AS `fulldescription`,`extensions`.`extension`.`developer` AS `developer`,`extensions`.`extension`.`itemcategory` AS `itemcategory`,`extensions`.`extension`.`crx_etag` AS `crx_etag`,`extensions`.`extension`.`lastupdated` AS `lastupdated`,`extensions`.`extension`.`last_modified` AS `last_modified` from (((select `extensions`.`extension`.`extid` AS `extid`,max(`extensions`.`extension`.`date`) AS `date` from `extensions`.`extension` where `extensions`.`extension`.`date` <= `until_date`() and !((`extensions`.`extension`.`extid`,`extensions`.`extension`.`date`) in (select `extensions`.`extension`.`extid`,max(`extensions`.`extension`.`date`) AS `date` from `extensions`.`extension` where `extensions`.`extension`.`date` <= `until_date`() group by `extensions`.`extension`.`extid`)) group by `extensions`.`extension`.`extid`)) `e1` join `extensions`.`extension` on(`e1`.`extid` = `extensions`.`extension`.`extid` and `e1`.`date` = `extensions`.`extension`.`date`)) */;
/*!50001 SET character_set_client = @saved_cs_client */;
/*!50001 SET character_set_results = @saved_cs_results */;
/*!50001 SET collation_connection = @saved_col_connection */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

43
database/views/extension_small.sql Normal file
View File

@ -0,0 +1,43 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Final view structure for view `extension_small`
--
/*!50001 DROP TABLE IF EXISTS `extension_small`*/;
/*!50001 DROP VIEW IF EXISTS `extension_small`*/;
/*!50001 SET @saved_cs_client = @@character_set_client */;
/*!50001 SET @saved_cs_results = @@character_set_results */;
/*!50001 SET @saved_col_connection = @@collation_connection */;
/*!50001 SET character_set_client = utf8 */;
/*!50001 SET character_set_results = utf8 */;
/*!50001 SET collation_connection = utf8_general_ci */;
/*!50001 CREATE ALGORITHM=UNDEFINED */
/*!50013 DEFINER=`writer`@`%` SQL SECURITY DEFINER */
/*!50001 VIEW `extension_small` AS select `extension`.`extid` AS `extid`,`extension`.`date` AS `date`,`extension`.`name` AS `name`,`extension`.`version` AS `version`,`extension`.`description` AS `description`,`extension`.`downloads` AS `downloads`,`extension`.`rating` AS `rating`,`extension`.`ratingcount` AS `ratingcount`,`extension`.`fulldescription` AS `fulldescription`,`extension`.`developer` AS `developer`,`extension`.`itemcategory` AS `itemcategory`,`extension`.`crx_etag` AS `crx_etag`,`extension`.`lastupdated` AS `lastupdated` from `extension` where `extension`.`extid` like 'aa%' */;
/*!50001 SET character_set_client = @saved_cs_client */;
/*!50001 SET character_set_results = @saved_cs_results */;
/*!50001 SET collation_connection = @saved_col_connection */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

43
database/views/extension_update.sql Normal file
View File

@ -0,0 +1,43 @@
-- MySQL dump 10.16 Distrib 10.3.8-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: extensions
-- ------------------------------------------------------
-- Server version 10.3.8-MariaDB-log
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Final view structure for view `extension_update`
--
/*!50001 DROP TABLE IF EXISTS `extension_update`*/;
/*!50001 DROP VIEW IF EXISTS `extension_update`*/;
/*!50001 SET @saved_cs_client = @@character_set_client */;
/*!50001 SET @saved_cs_results = @@character_set_results */;
/*!50001 SET @saved_col_connection = @@collation_connection */;
/*!50001 SET character_set_client = utf8 */;
/*!50001 SET character_set_results = utf8 */;
/*!50001 SET collation_connection = utf8_general_ci */;
/*!50001 CREATE ALGORITHM=UNDEFINED */
/*!50013 DEFINER=`root`@`%` SQL SECURITY DEFINER */
/*!50001 VIEW `extension_update` AS select `e3`.`extid` AS `extid`,`e3`.`first_date_with_new_crx_etag` AS `first_date_with_new_crx_etag`,`e3`.`new_crx_etag` AS `new_crx_etag`,`e3`.`last_date_with_previous_crx_etag` AS `last_date_with_previous_crx_etag`,`e4`.`crx_etag` AS `previous_crx_etag` from (((select `e1`.`extid` AS `extid`,`e1`.`date` AS `first_date_with_new_crx_etag`,`e1`.`crx_etag` AS `new_crx_etag`,max(`e2`.`date`) AS `last_date_with_previous_crx_etag` from (((select `extensions`.`extension`.`extid` AS `extid`,`extensions`.`extension`.`crx_etag` AS `crx_etag`,min(`extensions`.`extension`.`date`) AS `date` from `extensions`.`extension` where `extensions`.`extension`.`crx_etag` is not null group by `extensions`.`extension`.`extid`,`extensions`.`extension`.`crx_etag`)) `e1` join (select `extensions`.`extension`.`extid` AS `extid`,`extensions`.`extension`.`crx_etag` AS `crx_etag`,max(`extensions`.`extension`.`date`) AS `date` from `extensions`.`extension` where `extensions`.`extension`.`crx_etag` is not null group by `extensions`.`extension`.`extid`,`extensions`.`extension`.`crx_etag`) `e2` on(`e1`.`extid` = `e2`.`extid`)) where `e1`.`date` > `e2`.`date` group by `e1`.`crx_etag`)) `e3` join `extensions`.`extension` `e4` on(`e3`.`extid` = `e4`.`extid` and `e3`.`last_date_with_previous_crx_etag` = `e4`.`date`)) */;
/*!50001 SET character_set_client = @saved_cs_client */;
/*!50001 SET character_set_results = @saved_cs_results */;
/*!50001 SET collation_connection = @saved_col_connection */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
-- Dump completed on 2018-08-09 12:31:29

52
queries/get_added_content_scripts.sql
View File

@ -1,52 +0,0 @@
select extension_info.downloads, extension_info.extid, extension_info.name, url, crx_most_recent_and_prev.crx_etag as updated_crx
from (
-- We generate a table containing every crx_etag that we crawled within the
-- last day and is an update, and a random date where we encountered the its
-- previous crx_etag. We only use it to find the crx_etag again later.
select
crx_first_date1.crx_etag,
crx_first_date1.extid,
crx_first_date1.md as most_recent_update,
max(crx_first_date2.md) as last_date_with_previous_version
from (
select crx_etag,extid,min(date) as md
from extension
where crx_etag is not null
group by crx_etag
) crx_first_date1
inner join (
select extid,min(date) as md
from extension
where crx_etag is not null
group by crx_etag
) crx_first_date2
on crx_first_date1.extid=crx_first_date2.extid
where
crx_first_date1.md>crx_first_date2.md
-- the first time we have seen the crx_etag shall be zero days in the past, rounded to days,
-- measured from the current date and hour
and
TIMESTAMPDIFF(DAY, crx_first_date1.md, TIMESTAMPADD(HOUR,HOUR(UTC_TIME),UTC_DATE)) = 0
group by crx_first_date1.crx_etag
) crx_most_recent_and_prev
inner join content_script_url
on crx_most_recent_and_prev.crx_etag=content_script_url.crx_etag
inner join extension extension_info
on crx_most_recent_and_prev.extid=extension_info.extid
and extension_info.date=most_recent_update
and extension_info.crx_etag=crx_most_recent_and_prev.crx_etag
where
url in (
"http://*/*",
"https://*/*",
"*://*/*",
"<all_urls>"
)
and
url not in (
select url
from extension natural join content_script_url
where extid=extension_info.extid and date=last_date_with_previous_version
)
order by extension_info.downloads desc;

53
queries/get_added_permissions.sql
View File

@ -1,53 +0,0 @@
select extension_info.downloads, extension_info.extid, extension_info.name, permission, crx_most_recent_and_prev.crx_etag as updated_crx
from (
-- We generate a table containing every crx_etag that we crawled within the
-- last day and is an update, and a random date where we encountered the its
-- previous crx_etag. We only use it to find the crx_etag again later.
select
crx_first_date1.crx_etag,
crx_first_date1.extid,
crx_first_date1.md as most_recent_update,
max(crx_first_date2.md) as last_date_with_previous_version
from (
select crx_etag,extid,min(date) as md
from extension
where crx_etag is not null
group by crx_etag
) crx_first_date1
inner join (
select extid,min(date) as md
from extension
where crx_etag is not null
group by crx_etag
) crx_first_date2
on crx_first_date1.extid=crx_first_date2.extid
where
crx_first_date1.md>crx_first_date2.md
-- the first time we have seen the crx_etag shall be zero days in the past, rounded to days,
-- measured from the current date and hour
and
TIMESTAMPDIFF(DAY, crx_first_date1.md, TIMESTAMPADD(HOUR,HOUR(UTC_TIME),UTC_DATE)) = 0
group by crx_first_date1.crx_etag
) crx_most_recent_and_prev
inner join permission
on crx_most_recent_and_prev.crx_etag=permission.crx_etag
inner join extension extension_info
on crx_most_recent_and_prev.extid=extension_info.extid
and extension_info.date=most_recent_update
and extension_info.crx_etag=crx_most_recent_and_prev.crx_etag
where
permission in (
"<all_url>",
"http://*/*",
"https://*/*",
"webRequest",
"webRequestBlocking"
)
and
permission not in (
select permission
from extension natural join permission
where extid=extension_info.extid and date=last_date_with_previous_version
)
order by extension_info.downloads desc;