No default Year/month/day for only HH:mm:ss,SSS logs

Elastic Stack Logstash
1

We have some logfiles wich store only the time in format "HH:mm:ss,SSS" ( no day, month, year info)

11:38:58,654 INFO [ajp-195.72.2xx.17-8109-722] [lm.PerfilLoginModule] : [TOKEN_OK] authenticated user pepito

This has never been a problem until about a month (I haven't been able to find the exact date), we catched the hour, and logstash automatically added the year/mont/date, and added it to the daily index...

We use this:

if [logname] == "egovern" {
grok {
match => [
#Jboss
"message", "(%{TIMESTAMP_ISO8601:timestamp}|%{DATESTAMP:timestamp}|%{TIME:timestamp})%{SPACE}%{LOGLEVEL:level}%{SPACE}([%{DATA:logger}]+) [%{DATA:method}] : %{GREEDYDATA:message}"

            ]
    overwrite => [ "message" ]
  }

   date {
              match => [ "timestamp" , "HH:mm:ss,SSS" ]
    #          target => "@timestamp"
    }

And the destination index is:

index => "%{[@metadata][destindex]}-%{+YYYY.MM.dd}"

Since that misterious moment, those logs are being sent to index_name-2020-01-01 instead of for example index_name-2020-01-07 (this was happening too in 2019)

I've been reading and it seems that in those cases the date info was provided internally by logstash, or at least it was the same that it was in the @timestamp field, and I think this was the case until some time ago, but actually it's not :_(

How could I solve it?

2

If the year or month is not supplied the date filter will definitely get it wrong sometimes. See this issue.

3

Yeah, I undestand, but is not sometimes, it's always...

And is always 2020-01-01the destination index...
I'm almost sure that was working, and it started failing before the end of the year...

4

If that is your index option and the documents are indexed into index_name-2020-01-01 then the documents are not going through that elasticsearch output (. vs. -)

5

Sorry, it was my fault, the destination index is logstash-aplicacions-2020.01.01

I've seen in other forum post a possible workaround, but I'm not sure if it would be a good solution, because I would have to apply it for every log input with this format, making even more complicated the processing

I've also seen in the elastic docs, that it seems that is necessary to provide the year month day info, I don't know how it's been working until a few days... I'm starting to thing that maybe developers changed the log format...

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.