Proposal: split AwfulScraping into standalone library

[sockbot]

Rationale

If the AwfulCore AwfulScraping library were split into a standalone library, then other applications could be developed using the library independently from Awful.app.

I'd like to write a Awful forums CLI/TUI reader for MacOS and Linux and reuse the AwfulCore AwfulScraping library parser code.

Current State

Currently the AwfulCore AwfulScraping library and its dependencies are internal to the Awful.app project. This means that any projects that want to use AwfulCore AwfulScraping must be coupled to Awful.app, either internally or as a fork.

End State

• AwfulCore AwfulScraping is maintained as its own package library
• Awful.app imports AwfulCore AwfulScraping as an external dependency

[nolanw]

Hello! That sounds amazing!

I see a few steps in our future:

1. Ensure everything scraping is in AwfulScraping, and everything networking/persisting is in AwfulCore, all usable without any code in the Awful app target.
2. Get AwfulCore and AwfulScraping working on Linux (with at least perfunctory "does this build and run on Linux" tests so we don't break it).
3. Move AwfulCore and AwfulScraping to their own repo.

I think you need steps 1 and 2 complete to embark on your project, while step 3 is not immediately necessary. For now, could you include this Awful.app repo as a git submodule (or subtree) and depend on AwfulCore as a "local" dependency? I'll happily accept PRs to move code around, add Linux support, etc. (and you get the commit bit after your first PR, so I won't be slowing you down).

I'm absolutely willing to do step 3 at some point, but for now I'd prefer to procrastinate.

[nolanw]

Thinking about this some more, I realized AwfulCore is based on using Core Data for persistence. As far as I know, there's no Core Data for Linux.

It might make more sense to share AwfulScraping between apps, as the idea there is "making Forums URLs and scraping Forums HTML into structured data". Then Awful.app can turn that into Core Data modelled types, and you can use whatever you'd like (e.g. GRDB.swift). And as I'm writing this I remember AwfulScraping depends on HTMLReader which is an Objective-C library, and I don't know if Swift on Linux supports a Linux-capable objc runtime.

Anyway, if this works at all, I'd extend the same logic to anything shareable cross-platform, e.g. Forums cookie parsing/storage.

[sockbot]

Great! I was hoping you would be open to this proposal. Using git submodules is a new workflow for me, as is the Swift ecosystem. I don't even have a Mac as my main dev machine, so I will be relying on your automated tests to tell me whether I am making breaking changes to Awful.app.