Support for CIF feeds

[davidski]

Feature request to support Collective Intelligence Framework feeds. A fine intermediate step would be to allow importing from local files.

[alexcpsec]

Can you be more specific here? Do you mean connecting to a running instance of CIF and pulling everything that is there?

Or do you mean replicating the feed parsing of the ones located in their "feeds" directory that they produce as sample? In this case, I believe (have to check, though) that all the good ones (arbitrary subjective measure here) are already in the list.

[davidski]

Some of my intelligence partners host CIF instances and I would like to pull their feeds down for munging in combine. The harvest file would then be used either for tiq-test and for internal alerting/lookups/etc.

Referencing issues #48 and #23, a good first step would be to read a local file (cif feed already downloaded) for ingestion into combine. The stretch goal would be to have a framework where reaper could reach out to the cif instance and pull down a feed just like the current sample feeds. That's more complicated as CIF feeds aren't straight HTTP downloads but instead require API calls, hence the request for local file processing first. 😄

[krmaxwell]

Yes, local file processing is coming Real Soon Now ™️ - but does CIF no longer have the ability to output JSON or CSV feeds? I know there was a move to protocol buffers a while ago, but I hope it's somewhat able to produce common formats.

CybOX is also on the menu here but I recall that Wes didn't really want to deal with that, at least back a year or two ago.

[davidski]

Yay for local file processing! 😄

CIF can produce JSON, CSV, and XML feeds. As far as I know (and my CIF experience is still limited), those feeds cannot be retrieved directly via the HTTP mechanism combine uses today and would need to go through the cif utility (using whatever API CIF exposes). Local file import would make CIF imports easy to do, while a plugin system would allow me to hack calls to the cif util to directly retrieve the files and make retrievals automated.

Thanks for the help and dialog on this!

[alexcpsec]

Yeah. Connecting directly to CIF sounds like a worthwhile goal (and we will keep this open), but first things first. When the local files importing is ready, it should begin to help out with challenges like this.

[alexcpsec]

@davidski is there a default CSV format from CIF we should consider to import first?

[davidski]

I'm afraid my use case for this has changed. I'll close out this request for now. Thanks for taking the time to review!

[krmaxwell]

Reopening only because other people have privately expressed interest in the same feature even if @davidski doesn't need it anymore. :)

[alexcpsec]

Heh, I was about to do the same. Thanks, Kyle.

---

This e-mail message and any files transmitted with it contain legally
privileged, proprietary information, and/or confidential information,
therefore, the recipient is hereby notified that any unauthorized
dissemination, distribution or copying is strictly prohibited. If you have
received this e-mail message inappropriately or accidentally, please notify
the sender and delete it from your computer immediately.

[coolacid]

Query - Why connect to CIF if you can get and produce the same data? What is the end goal here?

[alexcpsec]

I can think of a few things:

1. Export to tiq-test
2. Export to different formats
3. Perform the winnower enrichments