4chan text scraping still possible?

[cameronbeansontoast]

Hi everyone!

I've just successfully installed 4CAT, only to find that 4chan is no longer an option under "Available data sources". Is there any way around this? Or are 4chan, reddit, 8chan no longer supported as a source?

If this is the case, can anyone recommend a similar data scraping programme for collecting text from 4chan? I'm looking for a way to aggregate and analyse text data in /pol, if possible over time and using keywords.

Many thanks!

[dale-wahl]

4chan and 8kun are still supported. 4chan/8kun API posts/threads disappear over time, thus you have to build up your data by first collecting it. There are instructions on how to set that up here:
https://github.com/digitalmethodsinitiative/4cat/wiki/Enabling-local-data-sources
Once you collect data, you can then search your records via the 4cat interface. We are exploring other collection options for locally stored datasources like those in the future that will hopefully be easier to set up.

We did stop supporting Reddit due to their API changes ($$$). You can find reddit data dumps online (which you could then import into 4cat if you were interested in using its analysis methods).

[cameronbeansontoast]

Thank you so much for the quick response! That makes sense.
I imagine then if I wanted to search across historical data for 4chan, I would be able to import 4chan data dumps and analyse them, just like with reddit?

Again, thanks for the reply!

[dale-wahl]

Correct. There is a helper script to upload 4chan data. There are some import scripts to help with that here. I will admit that I have not tested them in ages, but we have made no real changes to those datasources.