How to add data to Elasticsearch DB as per log line timestamp

Elastic Stack Logstash
1

Hello,
I am pushing some log files to Elasticsearch DB every day. For example: on 6th March morning, I will push some log files with log lines:
03-02 10:21:13.042 227 I Test: Invalid Solid Color Type
03-02 10:21:13.042 227 I Test: Valid Topic Type
03-02 10:21:13.042 227 I Test: Invalid service hms
Is there a way to add timestamp of 03-02 instead of 03-06 in ElasticDB?
Main idea is even though I push data on 6th March, I want data to be tagged for 2nd March (which is present in the log line). So on the time range of Kibana, these lines should show up for 2nd March instead of 6th March. Is it possible? Can someone please guide?

2 
    dissect { mapping => { "message" => "%{[@metadata][ts]} %{+[@metadata][ts]} %{}" } }
    date { match => [ "[@metadata][ts]", "MM-dd HH:mm:ss.SSS" ] }

Note that your messages do not say what year they are from, so logstash has to guess. It's guess may not be what you want if you ingest old logs. If they only a few days old you should be OK.

3

Thanks for your quick reply Badger. I have already present grok pattern. So I modified it with your suggestion. Is this correct though?

grok {
match => { "message" => "%{MONTHNUM:Month}-%{MONTHDAY:Day}\s*%{TIME:Timestamp}\s*%{NONNEGINT:SID}\s*%{NUMBER:Num}\s*%{WORD:Level}\s*(?<Function>(.?)):\s%{GREEDYDATA:Message}"}
}
date{
match => ["%{MONTHNUM:Month}-%{MONTHDAY:Day}\s*%{TIME:Timestamp}", "MM-dd HH:mm:ss.SSS"]
}

4

That grok pattern does not appear to match your data, but once you get it working you can use

    mutate { add_field => { "[@metadata][ts]" => "%{Month}-%{Day} %{Timestamp}" } }
    date{ match => ["[@metadata][ts]", "MM-dd HH:mm:ss.SSS"] }
5

Perfect this worked. Thanks Badger.

6

Hi Badger,

Quick question related to this: How can I ensure that data is not inserted in the future value? For example one of the timestamp says:
12-31 10:21:13.042 227 I Test: Invalid Solid Color Type

If I find this log now, I want to make sure this gets inserted in the 31st December 2018 and not 31st December 2019.

7

Not sure. It is probably going to require a ruby filter.

    ruby {
        code => '
            t = Time.at(event.get("@timestamp").to_f)
            if t > Time.now
                # It is in the future!
            end
        '
    }

The problem is how to subtract one year from Time. I suspect it will require converting the time to a string, then extracting the year, decrementing it and re-attaching it, then parsing the string. It strikes me as inelegant, so I am still trying to think of a better approach.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.