# Wrong values are populated into date field using Logstash

**URL:** <https://discuss.elastic.co/t/wrong-values-are-populated-into-date-field-using-logstash/167865>\
**Category:** Logstash\
**Created:** [February 11, 2019, 1:27pm UTC](https://discuss.elastic.co/t/wrong-values-are-populated-into-date-field-using-logstash/167865 "2019-02-11T13:27:28Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ram18](https://avatars.discourse-cdn.com/v4/letter/r/73ab20/32.png) [@Ram18](https://discuss.elastic.co/u/Ram18)\
**Post date:** [February 11, 2019, 1:27pm UTC](https://discuss.elastic.co/t/wrong-values-are-populated-into-date-field-using-logstash/167865/1 "2019-02-11T13:27:28Z")

</div>

Hello,  
I could see the date field are populated wrongly into Elasticsearch. I am generating CSV file and inserting the data through logstash conf file. Below is the screenshot where I have highlighted the date columns which are wrongly inserted for ex: In CSV the vendor\_start\_dt, vendor\_end\_dt and event\_tm columns are 01-feb-2019, but when I see the data through kibana, it shows as 01-Jan-2019. This is the case for all the values.

 ![date_csv](https://us1.discourse-cdn.com/elastic/original/3X/1/8/184806d024b5e863326dc905df7ada1aa53e671f.jpeg)

 ![kibana_dt](https://us1.discourse-cdn.com/elastic/original/3X/0/1/01bd00baf486512c9ffdb7822ea13e1dbd776e10.jpeg)

Below is the contents from conf file of logstash

> input {  
> file {  
> path =\> "/data01/logstash/data/daily\_final\_report\_201902110807.csv"  
> start\_position =\> "beginning"  
> sincedb\_path =\> "/dev/null"  
> }  
> }  
> filter {  
> csv {  
> separator =\> ","  
> columns =\> ["filename","fileformat","filesize","data\_received\_dt","vendor\_start\_dt","vendor\_end\_dt","bucketname","createdt","filetype","event\_tm","processing\_status"]  
> }  
> }  
> output {  
> elasticsearch {  
> hosts =\> "addd.xxxx.local:9200"  
> index =\> "rr\_log\_gen"  
> document\_type =\> "\_doc"  
> workers =\> 1  
> }  
> stdout {}  
> }

So let me know what was the wrong in this case?

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [February 12, 2019, 7:16am UTC](https://discuss.elastic.co/t/wrong-values-are-populated-into-date-field-using-logstash/167865/2 "2019-02-12T07:16:18Z")

</div>

What does your Elasticsearch mapping look like for those fields?

```auto
GET addd.xxxx.local:9200/rr_log_gen/_mapping/_doc

```

I've previously seen issues where a user configured a date field with the format including `D` (day of year) instead of `d` (day of month), causing similar effect.

---

<div class="post-metadata">

**Author:** ![Ram18](https://avatars.discourse-cdn.com/v4/letter/r/73ab20/32.png) [@Ram18](https://discuss.elastic.co/u/Ram18)\
**Post date:** [February 12, 2019, 7:22am UTC](https://discuss.elastic.co/t/wrong-values-are-populated-into-date-field-using-logstash/167865/3 "2019-02-12T07:22:07Z")

</div>

> [@yaauie](#):
>
> GET addd.xxxx.local:9200/rr\_log\_gen/\_mapping/\_doc

Below is the mappings listed. I have issue in fields like vendor\_start\_dt, vendor\_end\_dt and event\_tm

> {  
> "rr\_log\_gen" : {  
> "mappings" : {  
> "\_doc" : {  
> "properties" : {  
> "@timestamp" : {  
> "type" : "date"  
> },  
> "@version" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "bucketname" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "createdt" : {  
> "type" : "date"  
> },  
> "data\_received\_dt" : {  
> "type" : "date"  
> },  
> "event\_tm" : {  
> "type" : "date",  
> "format" : "yyyy-mm-dd HH:mm:ss"  
> },  
> "fileformat" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "filename" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "filesize" : {  
> "type" : "long"  
> },  
> "filetype" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "host" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "message" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "path" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "processing\_status" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "tags" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "vendor\_end\_dt" : {  
> "type" : "date",  
> "format" : "yyyy-mm-dd HH:mm:ss"  
> },  
> "vendor\_start\_dt" : {  
> "type" : "date",  
> "format" : "yyyy-mm-dd HH:mm:ss"  
> },  
> "vendorname" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> }  
> }  
> }  
> }

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [February 12, 2019, 7:39am UTC](https://discuss.elastic.co/t/wrong-values-are-populated-into-date-field-using-logstash/167865/4 "2019-02-12T07:39:39Z")

</div>

What do those columns look like in your CSV?

---

<div class="post-metadata">

**Author:** ![Ram18](https://avatars.discourse-cdn.com/v4/letter/r/73ab20/32.png) [@Ram18](https://discuss.elastic.co/u/Ram18)\
**Post date:** [February 12, 2019, 7:59am UTC](https://discuss.elastic.co/t/wrong-values-are-populated-into-date-field-using-logstash/167865/5 "2019-02-12T07:59:14Z")

</div>

> [@Ram18](#):
>
> yyyy-mm-dd HH:mm:ss

I have attached the screenshot in my initial topic itself. I am using shell script to generate the date column with the format +%Y-%m-%d %T and if we viewin csv file, it shows the data as m/d/yyyy hh:mi:ss format.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [February 12, 2019, 6:53pm UTC](https://discuss.elastic.co/t/wrong-values-are-populated-into-date-field-using-logstash/167865/6 "2019-02-12T18:53:46Z")

</div>

> [@Ram18](#):
>
> I have attached the screenshot in my initial topic itself. I am using shell script to generate the date column with the format +%Y-%m-%d %T

The dates in your screenshot are not of the format you say you have specified. Sometimes visual editors like Excel can present data in a different way than the raw bytes. Please paste the raw bytes from your CSV (perhaps open it with Notepad or some other basic editor that doesn't manipulate presentation).

---

<div class="post-metadata">

**Author:** ![Ram18](https://avatars.discourse-cdn.com/v4/letter/r/73ab20/32.png) [@Ram18](https://discuss.elastic.co/u/Ram18)\
**Post date:** [February 12, 2019, 7:02pm UTC](https://discuss.elastic.co/t/wrong-values-are-populated-into-date-field-using-logstash/167865/7 "2019-02-12T19:02:23Z")

</div>

The csv file has YYYY-MM-DD HH:MI:SS format  
[2018-08-01 00:00:00]

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 12, 2019, 7:06pm UTC](https://discuss.elastic.co/t/wrong-values-are-populated-into-date-field-using-logstash/167865/8 "2019-02-12T19:06:53Z")

</div>

> [@Ram18](#):
>
> "vendor\_end\_dt" : {  
> "type" : "date",  
> "format" : "yyyy-mm-dd HH:mm:ss"  
> },

Really? Your mapping has lower case mm for both month and minute?

---

<div class="post-metadata">

**Author:** ![Ram18](https://avatars.discourse-cdn.com/v4/letter/r/73ab20/32.png) [@Ram18](https://discuss.elastic.co/u/Ram18)\
**Post date:** [February 12, 2019, 7:08pm UTC](https://discuss.elastic.co/t/wrong-values-are-populated-into-date-field-using-logstash/167865/9 "2019-02-12T19:08:21Z")

</div>

yes!!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 12, 2019, 7:14pm UTC](https://discuss.elastic.co/t/wrong-values-are-populated-into-date-field-using-logstash/167865/10 "2019-02-12T19:14:08Z")

</div>

That seems odd. Unfortunately I do not have an elasticsearch node I can test with. I really should get one built out.

---

<div class="post-metadata">

**Author:** ![Ram18](https://avatars.discourse-cdn.com/v4/letter/r/73ab20/32.png) [@Ram18](https://discuss.elastic.co/u/Ram18)\
**Post date:** [February 12, 2019, 7:16pm UTC](https://discuss.elastic.co/t/wrong-values-are-populated-into-date-field-using-logstash/167865/11 "2019-02-12T19:16:33Z")

</div>

I have a doubt, Can we exclude the HH:MI:SS alone from the field and populate into new field with mappings as date

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 12, 2019, 7:16pm UTC](https://discuss.elastic.co/t/wrong-values-are-populated-into-date-field-using-logstash/167865/12 "2019-03-12T19:16:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.