How to Change Date Format in Logstash

Elastic Stack Logstash
1

I use ELK 6.1.1 and File beat to send some applications log files to logstash.

Logstash automatically, try to changing some of my fields type to Date and this change is not suitable for me.

I need just numbers in date but logstash change month to the name of month like attached picture.

57 AM
Screen Shot 2018-01-16 at 12.07.57 AM.png946×78 2.8 KB

For example it changes 1396/10/25 to October 25th 1396, which 1396 is year, 10 is month and 25 is day. It also add my time zone automatically and i want to remove it too.

Here is my configuration and sample data.

===============Logstash Config=================

input {
  beats {
    port => 5044
    #codec => plain { charset => "UTF-16" }
  }
}

filter {
	if [fields][log_type] == "dispatcher-packet"{
    csv {
        columns => ["PDate", "Date", "Time", "PacketSerial"]
        separator => "|"
        remove_field => [ "host", "message", "path" ]
    }
	mutate {
    convert => { "PacketSerial" => "integer" }
   }


output {

  if [fields][log_type] == "dispatcher-log"{
  elasticsearch { 
    hosts => ["localhost:9200"]
	index => "dispatcher-log-%{+YYYY.MM.dd}"
	}
}
  else if [fields][log_type] == "dispatcher-packet"{
  elasticsearch { 
    hosts => ["localhost:9200"]
	index => "dispatcher-packet4-%{+YYYY.MM.dd}"
	}
}
  stdout { codec => rubydebug }
}

=======================Sample File====================

PDate|Date|Time|PacketSerial
1396/10/25|2018-01-15|00:00:00.025|873672432

=======================get /dispatcher-packet4-2018.01.15============

{
  "dispatcher-packet4-2018.01.15": {
    "aliases": {},
    "mappings": {
      "doc": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          
          "Date": {
            "type": "date"
          },
          
          "PDate": {
            "type": "date",
            "format": "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis"
       
          "PacketSerial": {
            "type": "long"
          },

          "Time": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
 
    "settings": {
      "index": {
        "creation_date": "1516043848975",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "gNRUDFb5RxuWy7psju9-OA",
        "version": {
          "created": "6010199"
        },
        "provided_name": "dispatcher-packet4-2018.01.15"
      }
    }
  }
}

===============================================

What can i do?
Many thanks in advance for your response.

2

Please format your code using </> icon, it will make your post more readable and will help us help you :slight_smile:

Alternatively use markdown style like this:

```
CODE
```
3

Is it readable now?

1 Like
4

Yes, thank you so much for that! :smiley:

Ok, so looking at all of that, it seems to be doing what it should be. Is 1396/10/25 not at date?

5

1396/10/25 is a date in Solar calendar.

6

My problem is that Elasticsearch changes month which is a number "10" to month name "October". It's so useful that Elasticsearch changes my field to Date format, but I want only numbers. As I said before, our calendar is a Solar calendar and Elasticsearch uses a Gregorian calendar.

7

That's not Elasticsearch, it's Kibana. Elasticsearch stores everything as UTC and then Kibana converts that to local, human readable time.

You may want to look at the advanced setting in Kibana called dateFormat:scaled to start.

1 Like
8

The problem is solved by changing the dateFormat on Kibana's advanced setting, from MMMM Do YYYY, HH:mm:ss.SSS to MM Do YYY, HH:mm:ss.SSS.

Tanks a lot.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.