How to create a date scripted field for extracting actual date from logs?

[sagarleo (sagar)]

I want to create a date scripted field in kibana so that i can plot actual log dates in date histogram plot instead of using timestamp.

My log format is:
2019-01-18 18:49:24.375 Progress: creating : resume

I'm attaching code for scripted field i was trying to do
painless script:

new SimpleDateFormat('YYYY-MM-DD:HH:mm:ss.SSS').parse(doc['message'].value);

on field creation it gives me errors that "shards failed". Please help me

[ppisljar (Peter Pisljar)]

could you show screenshot from index pattern showing fields of your documents ?

thanks

[sagarleo (sagar)]

yes sure Sir,
index is logstash-*

i want scripted field to appear with timestamp while plotting date histogram. please help me out sir

[ppisljar (Peter Pisljar)]

then i guess you arleady have a @timestamp field ? could you try to better explain what you are tring to achieve ? also what is your message field ?

[sagarleo (sagar)]

Sir,
Using scripted field i am trying to extract the correct date from message field so that i can save it in a date type field and use that field instead of @timestamp field in Date Histogram Graph.
Message field contains info like date, warnings,progress etc
It's format is
2019-01-18 12:49:45.231 Progress: Targets : Resume

I'm using painless scripting in it.

So i'm facing error in the script part i showed in screenshot. If any more information is needed then i'll provide that too.

[Christian_Dahlqvist]

You should really do that at indexing time before you send the documents to Elasticsearch as that will scale and perform much better.

[sagarleo (sagar)]

I did try with converting field to date type in logstash but that was not working so i was trying using scripted field.
can u help me with the painless script to extract date from message field and storing in date type scripted field. I want script for it.
Here is my script
new SimpleDateFormat('YYYY-MM-DD:HH:mm:ss.SSS').parse(doc['message'].value);

[Christian_Dahlqvist]

The right place to do this is in my view in Logstash, so if you can describe the problems you faced there I can help you with that.

[sagarleo (sagar)]

okay Sir, I tried to convert the event_timestamp field to a date type field so that logs actual time can be retrieved but event_timestamp field shows to be "string type". In order to be able to plot data histogram i need event_timestamp field to be a date type which i am not able to convert.

sample log format:
2019-02-07 13:48:16.603 Progress: code completed successfully in 708 millisecond

my logstash config file:

input {
beats {
port => "5044"
}
}

filter {
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:event_timestamp} %{GREEDYDATA:message}" ]
}

date {
match => ["event_timestamp", "ISO8601"]
}
}

output {
elasticsearch {
hosts => [ "171.46.134.22:9200" ]
}
stdout {codec => rubydebug}
}

Please help me out sir. I need to show the plot

[Christian_Dahlqvist]

The config above stores the timestamp in the @timestamp field, so this is what you should use when building your visualisation. If you instead want to store it in the existing event_timestamp field, you need to set this as target in your date filter.

[sagarleo (sagar)]

here i made changes in config file

input {
beats {
port => "5044"
}
}

The filter part of this file is commented out to indicate that it is

optional.

filter {
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:event_timestamp} %{GREEDYDATA:message}" ]
}

date {
match => ["event_timestamp", "ISO8601"]
target => "event_timestamp"
}
}

output {
elasticsearch {
hosts => [ "171.34.345.25:9200" ]
}
stdout {codec => rubydebug}
}

Sir, im trying to convert it into date type but its not happening.

[Christian_Dahlqvist]

Once the type had been defined in an index it will not change, so you will need to index into a new index for the mapping to change.

[sagarleo (sagar)]

sir, I created a new index still its giving me event_timestamp field as string only. Is there anymore changes required in my logstash config file?
or any other way is there to convert string type to date type field.
while running logstash i'm getting an error
"Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}"