Feature/valkey json notifications

[conitrade-as]

This change set adds the ability to push JSON events into a Valkey / Redis stream. This allows for the decoupling of downstream processors of these events.

[AGWA]

Thanks for your interest in improving certspotter! I would like to avoid adding third-party dependencies. Is there a reason this functionality couldn't be implemented as a hook?

[conitrade-as]

Thanks for your feedback and this great CT log monitoring tool. I can understand that you want to avoid third-party dependencies.

Using a hook works great, as long as your volume of notifications is below a certain level (we are monitoring some cloud applications which issue a lot of certificates). At one point the system becomes saturated and a lot of CPU cycles are used to set up the environment and read from the file system parsing .json files. Thus the reason for using a queuing system comes down to a performance optimization. With this in place, a very modest machine can do all the work required without the overhead of calling an external hook.

May be this functionality could be implemented as a loadable module? https://pkg.go.dev/plugin#pkg-overview However, that may create additional headaches. May be you see another way of implementing the proposed functionality?

[AGWA]

Thanks for the explanation. I'd like to investigate if the performance of hooks can be improved. When you say "a lot of certificates", how many per second do you mean?

[conitrade-as]

That sounds like an possible improvement. The generic nature of the hooks is of course very nice. However, setting up environment variables and invoking a program takes a toll for sure. Another possibility could be to write to a (unix domain) socket to deliver a stream of .json messages.

For the volume I'll have to run some numbers for you. One use case is e.g. to monitor all domains from Switzerland's TLD .ch to get a better understanding of the attack surface.