jebe8883/DIDECT2S
Template
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
DIDECT2S/D3-Collector
History
Jesper Bergman 57f585de74 Delete 'D3-Collector/wget-log'
2022-10-03 14:43:26 +02:00
..
__pycache__
Added README.
2021-12-03 17:58:48 +01:00
blacklisted_onion_urls.html
Added README.
2021-12-03 17:58:48 +01:00
blacklisted_onion_urls.md5
Added README.
2021-12-03 17:58:48 +01:00
onion_links_150.txt
Added README.
2021-12-03 17:58:48 +01:00
onion_links.txt
Added README.
2021-12-03 17:58:48 +01:00
README.md
Added README.
2021-12-03 17:58:48 +01:00
similarity_measure.py
Upload files to 'D3-Collector'
2022-10-03 14:42:47 +02:00
site_scrape_cw.py
Upload files to 'D3-Collector'
2022-10-03 14:42:47 +02:00
site_scrape_dw.py
Upload files to 'D3-Collector'
2022-10-03 14:42:47 +02:00
site_scrape_marketplace.py
Upload files to 'D3-Collector'
2022-10-03 14:42:47 +02:00
start-d3-collector.sh
Added README.
2021-12-03 17:58:48 +01:00

README.md

D3-Collector

This is the, currently very primitive, scraper of DIDECTS. It is triggered when a URL is added from the D3-Annotator add-on and D3-Centraliser invokes D3-Collector. D3-Collector then simply sends a URL request through torsocks and saves the returned web page to the configured database.

Licence

GPL-3 (see LICENCE file)