With log pipelines, is there a way to use a different field if one field is not available?

[jerrac]

When processing my docker logs, I have 2 or 3 different places I can get the timestamp.

Is there a way to try to set it using one field, and then if that field is missing, then try a different one?

Specifically I'm using the regex processor to pull the timestamp out of the actual log message. But that doesn't always work, so is there a way I can fall back to the timestamp that my log shipper adds? Vector in this case.

So, normally the timestamp is pulled out of the "message" field, but if that doesn't work, I want to use the "timestamp" field.

I'm asking because I am seeing messages in my greptime logs like:

2025-07-01T21:02:16.048669Z  WARN servers::error: Failed to handle HTTP request  err=0: Pipeline error, at src/servers/src/pipeline.rs:130:14
1: Processor date: missing field: message_log_timestamp, at src/pipeline/src/etl/processor/date.rs:212:26

I don't see anything in the docs about this being possible, but I figured I'd ask.

[shuiyisong]

Hi, jerrac
First off, you can use the ignore_missing option to continue the process without breaking and returning an error.
Secondly, we don't have a pick-one-valid-from-an-array processor or mechanism. This can surely be done in a vrl script. But this is an interesting idea, we might need to do some investigation to see if there exists a general idea to form a processor.

However, you can try to use a workaround. With ignore_missing added, rename the output of the date processor to timestamp. In this case, if date processor outputs the result, it overrides the timestamp and is used in the result; otherwise the processor is skipped and the original timestamp is used. This is also possible with multiple options, notice that the processor runs in sequence. I've not tested it but I think it might work.

Let me know if there are any further questions!

[jerrac]

Ah, your workaround makes sense. I'll try it in a bit. :)

[jerrac]

Yep. That worked. Thanks!