Craiscraper

Description

Scan (and monitor) Craigslist for new apartments within the defined parameters.

Features

• sends notifications (after first run) for any new apartment
• extract features like parking, gym, pool, availability and apartment size from the description
• calculate distance from a location (i.e.: your work place)
• stores everything conveniently in a sqlite database

Requirements

• Python 3.10+

Usage

Docker

1. create a folder for the app and change your workdir: mkdir craigscraper; cd craigscraper
2. create a notification file using the instructions below vi notifications.yaml
3. prepare a .env file using the .env_template as model. Remember to uncomment the last two lines to allow data persistence
4. OPTIONAL: change container listening port adding UI_PORT=... in the .env file
5. run docker: docker run --env-file ./craigscraper/.env -p 8001:8501 -v ./craigscraper:/persist ghcr.io/porelli/craigscraper:main

if you want to examinate the database

• docker run -it --rm -p 8080:8080 -v ./craigscraper:/data -e SQLITE_DATABASE=rents.db coleifer/sqlite-web

Local or dev

1. clone the package: git clone git@github.com:porelli/craigscraper.git && cd craigscraper
2. copy env file: cp .env_template .env
3. replace env values: nano .env
4. create venv: python3 -m venv .venv
5. activate venv: source .venv/bin/activate
6. install dependencies: pip3 install -r requirements.txt
7. profit: scrapy crawl rent
8. launch UI: streamlit run ui/ui.py

Periodic scan

• suppress the notification test in the .env file using SUPPRESS_TEST_NOTIFICATION='True'
• specify the scan interval in the .env file using MINUTES_INTERVAL
• run: python3 main.py

Caveats (PRs are welcome!)

• currently it works only for Vancouver, BC
• code is not very organized and does not follow all the scrapy best practices

Notifications

By default, the script will try to send a desktop notification. A first notification is sent at the beginning as a test. Subsequent notifications are sent when new apartments appear in the search on there is a price change.

You can override the default behavior specifying your own notification provider(s) with an apprise compatible configuration file and using the -a notifications_file=<NOTIFICATION_FILE> CLI option or via ENV NOTIFICATION_FILE.

Please note, that for notifications to work on Mac or Windows, you may need to install additional packages. If you experience any errors with this, please refer to the apprise wiki.

MacOS

brew install terminal-notifier

Windows

pip install pywin32