# Dates dates dates

**URL:** <https://discuss.elastic.co/t/dates-dates-dates/103035>\
**Category:** Logstash\
**Created:** [October 6, 2017, 2:52pm UTC](https://discuss.elastic.co/t/dates-dates-dates/103035 "2017-10-06T14:52:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dorj1234](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorj1234/32/21339_2.png) [@dorj1234](https://discuss.elastic.co/u/dorj1234)\
**Post date:** [October 6, 2017, 2:52pm UTC](https://discuss.elastic.co/t/dates-dates-dates/103035/1 "2017-10-06T14:52:44Z")

</div>

Thanks in advance sorry it's a bit long. Trying not to be frustrated here, but the handling of dates is not behaving as documented.  
Below is a pretty complicated conf file, able to handle multiple .csv formats  
I have 2 date strings inside the .csv files, both failed to register as "date" fields and were converted to strings.  
I need one of them to be dates.

Either "run\_date" formatted as yyyyMMDDHH (example input: 2017100615)  
or "datetime" formatted as yyyy-MM-DD HH:mm:ss (example input 2017-10-06 15:31:06)

Below I tried to use the date plugin. If I try it on the datetime field the field becomes messed up or gets another field's value. How is it possible? I don't know.

I tried different targets for the "date" plugin. Nothing works.  
Tried putting the date plugin in different location. No change.  
Spent hours already trying to resolve this, no luck.

Example call to date plugin:

> date {  
> target =\> ["datetime"]  
> match =\> ["datetime", "yyyyMMdd HH:mm:ss"]  
> }

Here is most of the conf file for logstash:

> input {  
> file {  
> path =\> ["/some\_path/\*.csv"]  
> start\_position =\> beginning  
> sincedb\_path =\> "/dev/null"  
> }  
> }
> 
> filter {  
> grok {  
> patterns\_dir =\> ["patterns"]  
> match =\> ["path", "_%{HOURLYDAILY:timeframe}_%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day}%{HOUR:hour}\_%{VMPM:vm\_or\_pm}.csv$"]  
> add\_field =\> ["received\_from", "%{host}"]  
> add\_tag =\> ["%{timeframe}", "%{vm\_or\_pm}"]  
> }  
> if [vm\_or\_pm] == "PM" {  
> csv {  
> separator =\> " "  
> skip\_empty\_columns =\> false  
> columns =\> ["run\_date",  
> "location\_continent",  
> "location\_state\_country",  
> "location\_city",  
> "datacenter",  
> "cluster",  
> "datetime",  
> "uuid",  
> "host",  
> "primary\_key",  
> "num\_cpus",  
> "num\_sockets", "cpu\_avg\_util", "cpu\_max\_util", "cpu\_avg\_ghz", "cpu\_max\_ghz", "cpu\_cap\_ghz",  
> "memory\_avg\_util", "memory\_max\_util", "memory\_avg\_gb", "memory\_max\_gb", "memory\_cap\_gb", "memory\_baloon\_avg\_util", "memory\_baloon\_max\_util", "memory\_baloon\_cap\_gb",  
> "swapping\_avg\_util", "swapping\_max\_util", "swapping\_cap\_gb", "network\_avg\_util","network\_max\_util","network\_cap\_gb","io\_avg\_util","io\_max\_util","io\_cap\_gb",  
> "avg\_q1vcpu\_util","max\_q1vcpu\_util","avg\_q2vcpu\_util","max\_q2vcpu\_util","avg\_q4vcpu\_util","max\_q4vcpu\_util","avg\_q8vcpu\_util","max\_q4vcpu\_util","avg\_q16vcpu\_util","max\_q16vcpu\_util"]  
> } # end of PM csv clause  
> } # end if PM  
> else if [vm\_or\_pm] == "VM" {  
> csv {  
> #source =\> "message"  
> separator =\> " "  
> skip\_empty\_columns =\> false  
> #quote\_char =\> """  
> columns =\> ["run\_date", "location\_continent", "location\_state\_country", "location\_city", "datacenter", "cluster", "primary\_key", "uuid", "vmname", "group", "datetime", "host", "num\_vcpus", "cpu\_avg\_util", "cpu\_avg\_ghz", "cpu\_max\_util", "cpu\_max\_ghz", "cpu\_cap\_ghz",  
> "memory\_avg\_util", "memory\_avg\_gb", "memory\_max\_util", "memory\_max\_gb", "memory\_cap\_gb", "storage\_util\_avg",  
> "storage\_util\_max", "storage\_cap\_gb", "vdrive", "storage\_used\_gb", "iops\_avg", "iops\_max", "iops\_cap",  
> "io\_thruput\_avg", "io\_thruput\_max", "io\_thruput\_cap\_gb", "memory\_baloon\_avg\_util", "memory\_baloon\_max\_util", "memory\_baloon\_cap\_gb", "network\_avg", "network\_max", "network\_cap\_gb", "storage\_latency\_avg", "storage\_latency\_max", "storage\_latency\_cap"]  
> } # end of VM csv clause  
> } # end if VM  
> else {  
> csv {  
> remove\_tag =\> ["%{timeframe}"]  
> separator =\> " "  
> skip\_empty\_columns =\> false  
> columns =\> ["datacenter","cluster","vm\_uuid","vmname","primary\_key","DeployedState","datetime","host","numvcpus","cpu\_cap\_ghz","memory\_cap\_gb","storage\_cap\_gb"]  
> } # end of VM csv clause  
> } # and of else - vmremlist/vmaddlist  
> date {  
> target =\> ["datetime"]  
> match =\> ["datetime", "yyyyMMdd HH:mm:ss"]  
> }  
> mutate {  
> remove\_field =\> ["month", "day", "year", "hour"]  
> }  
> mutate {
> 
> ```
> # mutual PM and VM fields
> convert => ["cpu_avg_ghz","float"]
> convert => ["cpu_cap_ghz","float"]
> convert => ["cpu_max_ghz","float"]
> convert => ["memory_avg_gb","float"]        	
> convert => ["memory_avg_util","float"] 
> convert => ["memory_max_util","float"]		
> convert => ["network_cap_gb","float"]
> convert => ["memory_max_gb","integer"]
> convert => ["memory_cap_gb","integer"]
> convert => ["memory_baloon_avg_util","float"]
> convert => ["memory_baloon_max_util","float"]
> convert => ["memory_baloon_cap_gb","float"]
>     	
> # addremvm/addrempm fields
> convert => ["numvcpus","integer"]
>     	
> # PM hourly fields
> convert => ["num_cpus","integer"]
> convert => ["num_sockets","integer"]
> convert => ["cpu_avg_util","float"]
> convert => ["cpu_max_util","float"]
> convert => ["swapping_avg_util","float"]
> convert => ["swapping_max_util","float"]
> convert => ["swapping_cap_gb","float"]
> convert => ["network_avg_util","float"]
> convert => ["network_max_util","float"]
> convert => ["io_avg_util","float"]
> convert => ["io_max_util","float"]
> convert => ["io_cap_gb","float"]
> convert => ["avg_q1vcpu_util","float"]
> convert => ["max_q1vcpu_util","float"]
> convert => ["avg_q2vcpu_util","float"]
> convert => ["max_q2vcpu_util","float"]
> convert => ["avg_q4vcpu_util","float"]
> convert => ["max_q4vcpu_util","float"]
> convert => ["avg_q8vcpu_util","float"]
> convert => ["max_q8vcpu_util","float"]
> convert => ["avg_q16vcpu_util","float"]
> convert => ["max_q16vcpu_util","float"]  
> 
> ```
> 
> ## VM hourly fields
> 
> ```
> convert => ["num_vcpus","integer"]
> convert => ["storage_util_avg","float"]
> convert => ["storage_util_max","float"]
> convert => ["storage_cap_gb","integer"]
> convert => ["storage_used_gb","float"]
> convert => ["iops_avg","float"]
> convert => ["iops_max","float"]
> convert => ["iops_cap","float"]
> convert => ["io_thruput_avg","float"]
> convert => ["io_thruput_max","float"]
> convert => ["io_thruput_cap_gb","float"]
> convert => ["network_avg","float"]
> convert => ["network_max","integer"]
> convert => ["storage_latency_avg","integer"]
> convert => ["storage_latency_max","integer"]
> convert => ["storage_latency_cap","integer"]
> 
> ```
> 
> } # end mutate  
> } # end filter
> 
> output {  
> elasticsearch {  
> hosts =\> "localhost"  
> index =\> "tokiomarine"  
> document\_type =\> "%{vm\_or\_pm}\_%{timeframe}\_report"  
> }  
> stdout { }  
> }

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 6, 2017, 9:44pm UTC](https://discuss.elastic.co/t/dates-dates-dates/103035/2 "2017-10-06T21:44:49Z")

</div>

> [@dorj1234](#):
>
> Below I tried to use the date plugin. If I try it on the datetime field the field becomes messed up or gets another field's value. How is it possible? I don't know.

Without seeing the data it's difficult to answer this.

---

<div class="post-metadata">

**Author:** ![ppuschmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppuschmann/32/146751_2.png) [@ppuschmann](https://discuss.elastic.co/u/ppuschmann)\
**Post date:** [October 7, 2017, 6:55am UTC](https://discuss.elastic.co/t/dates-dates-dates/103035/3 "2017-10-07T06:55:25Z")

</div>

> [@dorj1234](#):
>
> ```
> date {
> target => ["datetime"]
> match => ["datetime", "yyyyMMdd HH:mm:ss"]
> }
> 
> ```

I rather use something like

```
date {
    match => ["datetime", "yyyyMMdd HH:mm:ss"]
    target => ["newdatetime"]
    remove_field => ["datetime"]
}

```

to convert the content from the field `datetime`into the field newdatetime`, but after a successful conversion the old field is deleted.

This way we easily can check if the conversion happened. (And also by looking at the field `tags`).

@dorj1234 Can you please properly format your code as `code`? Would be much easier to read with proper indentation.

---

<div class="post-metadata">

**Author:** ![dorj1234](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorj1234/32/21339_2.png) [@dorj1234](https://discuss.elastic.co/u/dorj1234)\
**Post date:** [October 12, 2017, 1:56pm UTC](https://discuss.elastic.co/t/dates-dates-dates/103035/4 "2017-10-12T13:56:30Z")

</div>

Thank you ! eventually that worked after I removed and recreated the index. Something may be buggy there but at least there is an override.

Thanks again for reading and helping.  
JD

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 9, 2017, 1:56pm UTC](https://discuss.elastic.co/t/dates-dates-dates/103035/5 "2017-11-09T13:56:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.