Decide on Redis Persistance #14
Description
https://redis.io/topics/persistence
AOF looks like a worthwhile venture. They recommend using the snapshots with AOF.
We could probably release without it at first.
Brainstorming notes:
How much resources do we expect a server to have for Redis? In high traffic orgs, we may need snapshot or AOF files to cache extraneous data that cannot fit in memory as we write to zeek files
What is our max file size for a generated zeek log>
Controlling this could potentially eliminate any issues with oversized zeek logs being imported into RITA (think memories from intel man)
What is the expected data size for flow data from logstash for an organization of (let’s say…) 1000 remote workers/endpoints? Can we filter packetbeat data in a logstash pipeline before spitting to Redis to reduce unnecessary memory consumption within Redis?
Filter using packetbeat: https://www.elastic.co/guide/en/beats/packetbeat/master/filtering-and-enhancing-data.html