Collection of Java-based Web crawlers

Prerequisites

• Maven 3
• JDK 21

How to build

mvn clean install

Common crawler functionality

• Your crawler should extend WebCrawler base crawler class
• DTO class which describes collected data should implement CrawlerData marker interface

Crawler for Orthodox torrent tracker pravtor.ru

Check PravtorRuWebCrawler for details

To make search - use run-search script in pravtor.ru-crawler folder.
Collected data will be placed into result.xls file in sandbox folder

Crawler for vacancies aggregator rabota.by

Check RabotaByWebCrawler for details

To make search - use run-search script in rabota.by-crawler folder.

Video with description of the project