Change date format need help

Elastic Stack Logstash
1

i all i am new to logstash
i have a input log
{"message":"08/30/2015 16:17:36.442,16.003765,20.503670","@version":"1","@timestamp":"2015-08-30T13:17:46.745Z","host":"ERIKP-PC","LogDate":"08/30/2015 16:17:36.442","RamUsed":16.003765,"CpuPercent":20.50367}

I like to change the "LogDate" from "08/30/2015 16:17:36.442" to "08-30-2015 16:17:36.442"

How can i do it?

2

Normally one uses the date filter to parse the event's timestamp into the @timestamp field, which has a standardized format in UTC. What do you plan to do with the LogDate field, i.e. why does its date format matter?

3

My PC send the log to ELK server and in the logs i get
"message" => "08/30/2015 15:25:55.353,14.651681,5.618097",
"@version" => "1",
"@timestamp" => "2015-08-30T12:26:05.705Z",
"host" => "ERIKP-PC",
"LogDate" => "08/30/2015 15:25:55.353",
"RamUsed" => 14.651681,
"CpuPercent" => 5.618097,
"tags" => [
[0] "_dateparsefailure"

I cant understand the _dateparsefailure error

4

The Logstash log will contain clues about why the date filter couldn't parse the date. If you post your configuration we'll be in a better position to help.

5

This is the server sends data to rabbitmq

input {
perfmon {
interval => 10
counters => [
"\Memory% Committed Bytes In Use",
"\Processor(_Total)% Processor Time"]
}
}

filter {
grok {
match => {
"message" => "%{DATESTAMP:LogDate},%{NUMBER:RamUsed:float},%{NUMBER:CpuPercent:float}"
}
}

}

output {
rabbitmq {
exchange => "Logs.Logstash"
exchange_type => "topic"
host => "blabla"
user => "blabla"
password => "blabla"
port => blabla
durable => "true"
ssl => false
key => "cpu.#"
}
file {
path => "C:\perfmon_output.txt"
}
}

This the server that bring the data from rabbitmq

input {
rabbitmq {
host => "blabla"
port => blabla
ssl => false
user => "blabla"
password => "blabla"
queue => "Logstash.Perfmon.CPU"
durable => true
}
}
output {
elasticsearch {
host => "blabla"
cluster => "blabla"
protocol => transport
}

stdout { codec => rubydebug }
}

6

Where's the date filter that adds the _dateparsefailure tag?

7

I dont have any

this is what i see in the log

"message" => "08/31/2015 10:32:05.767,18.271383,6.800951",
"@version" => "1",
"@timestamp" => "2015-08-31T07:32:11.105Z",
"host" => "ERIKP-PC",
"LogDate" => "08/31/2015 10:32:05.767",
"RamUsed" => 18.271383,
"CpuPercent" => 6.800951,
"tags" => [
[0] "_dateparsefailure"
]

8

The _dateparsefailure tag is only added by the date filter so there must be a date filter in there somewhere. Do you by any chance have an old file laying around in /etc/logstash/conf.d?

9

Yes i found it

date {
match => [ "LogDate", "yyyy-MM-dd HH:mm:ss,SSS" ]
target => "EventDate"
}

its in different conf file not related to my conf file

10

Okay. Well, that date format clearly doesn't match what's in your LogDate field but it should be an easy adjustment.

11

Now i understand what happened the date in the different conf took over my conf file
now i understand how its working

So how can i change my date format so it will be the same?

12

Adjust the match pattern to match what your field contains. This should work:

date {
  match => ["LogDate", "MM/dd/yyyy HH:mm:ss.SSS"]
  remove_field => ["LogDate"]
}

I added a remove_field option to remove the field that was just parsed and deleted the target option since @timestamp is the default timestamp field that you should use unless you have a compelling reason.

13

Thanks a lot
how can i manipulate the date so it will be in different format?

14

That's not supported out of the box but you could use a ruby filter. But why do you feel you need to do that?

15

Thanks Removed the date all is working :slight_smile: