Combination of logfiles with time-only and datetime fields

[carstengehling]

On a single server, I have multiple logfiles, that I need to send to LogStash, the problem is, that logfiles from one application has a datetime column:

2018-06-29 13:44:22,569 [26] DEBUG yadayadayada

and logfiles from another application has only time column:

07:06:27.105 [26] DEBUG yadayadayada

The way I understand FileBeat, I cannot send these files into seperate pipelines (one FileBeat service is running on the server) - is that correct?

If so: Is there a way to configure a LogStash pipeline, so in both logfile cases, the time/datetime column gets used for the @timestamp - but in the second case, the time first gets "enriched" with current date?

Or another solution: Can FileBeat add the date itself to this single input, before it sends it to LogStash?

[Badger]

There are several options. You could use one prospector for each application, and add a tag using the prospector, then conditionalize the parsing, possibly using a routing pipeline to send the two tags to different processing pipelines.

However, if the rest of the processing is common, then just grok the date conditionally using

grok { match => { "message" => [ "^%{TIMESTAMP_ISO8601:date},%{NUMBER:ms} \[", "^%{TIME:time} \[" ] } }

[carstengehling]

Thanks for your reply, that gets me part of the way. From one logfile, I will now have:

"datetime" => "2017-12-17 17:08:36",
"ms" => "911",

and from another I have:

"time" => "06:04:53.809",

What if I want both of these to end up with - let's call it "final_datetime"? In pseudocode computed like this:

if <datetime and ms exists> then
    add_field => {"final_datetime" => "%{datetime}.%{ms}"}
else if <time exists> then
    add_field => {"final_datetime" => "%{<date-part of @timestamp>} %{{time}"}
end

So that no matter what logline comes in, final_datetime is the same format? (ie. YYYY-MM-DD hh:mm:ss.mss)

[carstengehling]

I figured it out - turned out that I needed to make two different groks, that is where the conditional parsing is split:

filter {
  grok {
    match => {
      "message" => "^%{TIMESTAMP_ISO8601:datetime},%{NUMBER:ms} \["
    }
    add_field => {
      "final_datime" => "%{datetime}.%{ms}"
    }
  }
  grok {
    match => {
      "message" => "^%{TIME:time} \["
    }
    add_field => {
      "final_datime" => "%{+YYYY-MM-dd} %{time}"
    }
  }
}