Changing date format through Logstash

[zaeemmasood]

Hello. We are getting feed from Kafka topic. One of the fields rdt appears in the following format:

I need to change the date format to YYYY-MM-dd

This was added in the logstash conf file:

    if "KafkCollect" in [tags] {

        file {
                   path => "/opt/total/logs/kafkaCollect_rubydebug.txt"
                   codec => rubydebug
                }

        filter {
                mutate {
                     strip => ["rdt"]
                     }
                date {
                      match => ["rdt", "YYYY-MM-dd"]
                      target => "rdt"
                     }
               }

    elasticsearch {
     hosts => [ "xxx.xx.xx.xx:43045" ]
     hosts => [ "xxx.xx.xx.xx:43045" ]
     hosts => [ "xxx.xx.xx.xx:43045" ]
     hosts => [ "xxx.xx.xx.xx:43045" ]
     user => "datater"
     password => "xxxxxxxxxx"
     index => "dateter-coll-%{+YYYY.MM.dd}"
     manage_template => true
     template_overwrite => true
     template => "/opt/total/logstash/config/data_kafkaCollect_template.json"
     template_name => "dataCollect_template"
                }
        }

This gives syntax error at startup.

Please guide

Thanks

[leandrojmp (Leandro Pereira)]

You want to change how the date appears on Kibana?

If so, this is not related to Logstash, this is how Kibana shows date fields, you can change it, but it will change how kibana show all date fields.

To change it you need to go in Stack Management > Kibana > Advanced Settings

There you will have this setting:

But changing this will change how all date fields are show in Kibana, including the @timestamp field.

Another option would be to store the rdt field as a string, but you will probably need to change the mapping on your template and reindex your data.

[zaeemmasood]

@leandrojmp

Another option would be to store the rdt field as a string, but you will probably need to change the mapping on your template and reindex your data.

Thanks. This is new set up so I can afford re-indexing.

How do I change the field to string?

[leandrojmp (Leandro Pereira)]

@zaeemmasood

How do I change the field to string?

You seem to have a template for your index.

template => "/opt/total/logstash/config/data_kafkaCollect_template.json"

So you would need to change the type of the mapping for the field rdt to keyword.

Just remember that doing this you will not be able to use the field rdt as a date field, it will be just a keyword field with the repsentation of the date.

[zaeemmasood]

thanks