# Date filter problem

**URL:** <https://discuss.elastic.co/t/date-filter-problem/97211>\
**Category:** Logstash\
**Created:** [August 16, 2017, 8:44am UTC](https://discuss.elastic.co/t/date-filter-problem/97211 "2017-08-16T08:44:27Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jonesthomas](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\
**Post date:** [August 16, 2017, 8:44am UTC](https://discuss.elastic.co/t/date-filter-problem/97211/1 "2017-08-16T08:44:27Z")

</div>

- 1 : In kibana Time field is showing string , not as date field.

- 2 : this is my sample log file..  
{"Metrics":"type=COUNTER, name=FacilityMgmtResource Counter, count=0,Time:2017-08-16\_01:51:14.926"}  
{"Metrics":"type=METER, name=systems.ellora.core.exception.UnhandledRuntimeMapper, count=0, mean\_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate\_unit=events/second,Time:2017-08-16\_01:51:17.699"}  
{"Metrics":"type=TIMER, name=FacilityMgmtResource Timer, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean\_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate\_unit=events/second, duration\_unit=milliseconds,Time:2017-08-16\_01:51:17.700"}

- 3 : my filter:  
filter {  
csv {  
source =\> "message"  
}  
csv {  
source =\> "Metrics"  
}  
date {  
match =\> ["Time","yyyy-MM-dd\_HH:mm:ss,SSS"]  
target =\> "Time"  
}  
kv{  
value\_split =\> "="  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2017, 8:51am UTC](https://discuss.elastic.co/t/date-filter-problem/97211/2 "2017-08-16T08:51:25Z")

</div>

Why are you using a csv filter to parse the `message` field? The input lines contain JSON data.

Show what an event produced by Logstash looks like. Copy/paste from Kibana's JSON tab or use a `stdout { codec => rubydebug }` output. Format everything you paste as preformatted text.

---

<div class="post-metadata">

**Author:** ![Jonesthomas](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\
**Post date:** [August 16, 2017, 8:53am UTC](https://discuss.elastic.co/t/date-filter-problem/97211/3 "2017-08-16T08:53:12Z")

</div>

> [@magnusbaeck](#):
>
> Why are you using a csv filter to parse the message field? The input lines contain JSON data.
> 
> Show what an event produced by Logstash looks like. Copy/paste from Kibana’s JSON tab or use a stdout { codec =\> rubydebug } output. Format everything you paste as preformatted text.

its a json file with csv format, and when i try with json ,i get a json parsing failure error in logstash.log

{  
"\_index": "filebeat-2017.08.16",  
"\_type": "log",  
"\_id": "AV3qPHYgJmWWVh1I-bI8",  
"\_version": 1,  
"\_score": null,  
"\_source": {  
"mean\_rate": "0.11281360328467671",  
"offset": 211,  
"m1": "0.2",  
"input\_type": "log",  
"count": "1",  
"m5": "0.2",  
"rate\_unit": "events/second",  
"Time": "2017-08-16\_12:29:59,457",  
"source": "/ext/logs/perff.log",  
"message": "{"Metrics" : " type=METER , name=systems.ellora.core.exception.NoDataFoundMapper , count=1 , mean\_rate=0.11281360328467671 , m1=0.2 , m5=0.2 , m15=0.2 , rate\_unit=events/second , Time=2017-08-16\_12:29:59,457 "}",  
"type": "METER",  
"tags": [  
"beats\_input\_codec\_plain\_applied"  
],  
"m15": "0.2",  
"@timestamp": "2017-08-16T08:50:12.152Z",  
"@version": "1",  
"beat": {  
"hostname": "sdatabase",  
"name": "sdatabase",  
"version": "5.5.0"  
},  
"host": "sdatabase",  
"name": "systems.ellora.core.exception.NoDataFoundMapper",  
"{"Metrics"": " type=METER , name=systems.ellora.core.exception.NoDataFoundMapper , count=1 , mean\_rate=0.11281360328467671 , m1=0.2 , m5=0.2 , m15=0.2 , rate\_unit=events/second , Time=2017-08-16\_12:29:59,457 "  
},  
"fields": {  
"@timestamp": [  
1502873412152  
]  
},  
"sort": [  
1502873412152  
]  
}

---

<div class="post-metadata">

**Author:** ![Jonesthomas](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\
**Post date:** [August 16, 2017, 9:11am UTC](https://discuss.elastic.co/t/date-filter-problem/97211/4 "2017-08-16T09:11:06Z")

</div>

> [@Jonesthomas](#):
>
> “Time”: “2017-08-16\_12:29:59,457”,

Still time as string field

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2017, 9:24am UTC](https://discuss.elastic.co/t/date-filter-problem/97211/5 "2017-08-16T09:24:44Z")

</div>

**Format everything you paste as preformatted text.**

> its a json file with csv format, and when i try with json ,i get a json parsing failure error in logstash.log

What's the error? The `message` field looks okay to me. I don't think this looks like CSV at all. Use a json filter or codec, then a kv filter to parse the `Metrics` field.

Your problem is that the date filter comes before the kv filter so that the `Time` field doesn't exist when the date filter runs.

When that has been fixed and the date filter works you need to reindex the existing data so the `Time` field can be detected as a date field. If you don't care about the existing data in the index you can just delete the index.

---

<div class="post-metadata">

**Author:** ![Jonesthomas](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\
**Post date:** [August 16, 2017, 9:53am UTC](https://discuss.elastic.co/t/date-filter-problem/97211/6 "2017-08-16T09:53:26Z")

</div>

- I reindexed.

- After changing filter to json: This is the result  
{  
"\_index": "filebeat-2017.08.16",  
"\_type": "log",  
"\_id": "AV3qc2UWJmWWVh1I-r8i",  
"\_version": 1,  
"\_score": 2,  
"\_source": {  
"mean\_rate": "0.0,",  
"Time": "2017-08-16\_02:54:53.211"}",  
"source": "/ext/logs/perff.log",  
"type": "log",  
"p95": "0.0,",  
"p98": "0.0,",  
"p75": "0.0,",  
"m15": "0.0,",  
"p99": "0.0,",  
"min": "0.0,",  
"@version": "1",  
"beat": {  
"hostname": "sdatabase",  
"name": "sdatabase",  
"version": "5.5.0"  
},  
"host": "sdatabase",  
"p999": "0.0,",  
"stddev": "0.0,",  
"{"Metrics":"type": "TIMER,",  
"offset": 937,  
"m1": "0.0,",  
"max": "0.0,",  
"input\_type": "log",  
"count": "0,",  
"m5": "0.0,",  
"rate\_unit": "events/second,",  
"message": "{"Metrics":"type=TIMER, name=FacilityMgmtResource Timer, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean\_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate\_unit=events/second, duration\_unit=milliseconds, Time=2017-08-16\_02:54:53.211"}",  
"duration\_unit": "milliseconds,",  
"tags": [  
"beats\_input\_codec\_plain\_applied"  
],  
"@timestamp": "2017-08-16T09:50:11.563Z",  
"median": "0.0,",  
"mean": "0.0,",  
"name": "FacilityMgmtResource",  
"{"Metrics"": "type=TIMER, name=FacilityMgmtResource Timer, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean\_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate\_unit=events/second, duration\_unit=milliseconds, Time=2017-08-16\_02:54:53.211"  
},  
"fields": {  
"@timestamp": [  
1502877011563  
]  
}  
}

- My filter  
filter {  
json {  
source =\> "message"  
}  
json {  
source =\> "Metrics"  
}  
kv{  
value\_split =\> "="  
}  
date {  
match =\> ["Time","yyyy-MM-dd\_HH:mm:ss,SSS"]  
target =\> "Time"  
}  
}

---

<div class="post-metadata">

**Author:** ![Jonesthomas](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\
**Post date:** [August 16, 2017, 9:56am UTC](https://discuss.elastic.co/t/date-filter-problem/97211/7 "2017-08-16T09:56:10Z")

</div>

> [@Jonesthomas](#):
>
> “Time”: “2017-08-16\_02:54:53.211”}",

And why this field is set like this??? Like after Time value =\> "}"

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2017, 10:03am UTC](https://discuss.elastic.co/t/date-filter-problem/97211/8 "2017-08-16T10:03:50Z")

</div>

Why are you repeatedly ignoring my request that you post pasted log data as preformatted text?

I'm not spending any more time on this until you post logs and configuration that hasn't been mangled.

---

<div class="post-metadata">

**Author:** ![Jonesthomas](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\
**Post date:** [August 16, 2017, 10:12am UTC](https://discuss.elastic.co/t/date-filter-problem/97211/9 "2017-08-16T10:12:13Z")

</div>

Sorry for the inconvenience, this is my sample log data  
`{"Metrics":"type=COUNTER, name=FacilityMgmtResource Counter, count=0, Time=2017-08-16_02:54:53.180"} {"Metrics":"type=METER, name=FacilityMgmtResource Meter, count=0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, Time=2017-08-16_02:54:53.180"} {"Metrics":"type=METER, name=systems.ellora.core.exception.NoDataFoundMapper, count=1, mean_rate=0.9657515433602983, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, Time=2017-08-16_02:54:53.180"} {"Metrics":"type=METER, name=systems.ellora.core.exception.UnhandledRuntimeMapper, count=0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, Time=2017-08-16_02:54:53.180"} {"Metrics":"type=TIMER, name=FacilityMgmtResource Timer, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds, Time=2017-08-16_02:54:53.211"} {"Metrics":"type=COUNTER, name=FacilityMgmtResource Counter, count=1, Time=2017-08-16_02:54:54.032"} {"Metrics":"type=METER, name=FacilityMgmtResource Meter, count=1, mean_rate=0.5260485800519025, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, Time=2017-08-16_02:54:54.043"} {"Metrics":"type=METER, name=systems.ellora.core.exception.NoDataFoundMapper, count=1, mean_rate=0.5268436948972698, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, Time=2017-08-16_02:54:54.043"} {"Metrics":"type=METER, name=systems.ellora.core.exception.UnhandledRuntimeMapper, count=0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, Time=2017-08-16_02:54:54.053"} {"Metrics":"type=TIMER, name=FacilityMgmtResource Timer, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds, Time=2017-08-16_02:54:54.053"} {"Metrics":"type=COUNTER, name=FacilityMgmtResource Counter, count=1, Time=2017-08-16_02:54:55.032"} {"Metrics":"type=METER, name=FacilityMgmtResource Meter, count=1, mean_rate=0.3448138089928063, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, Time=2017-08-16_02:54:55.042"} {"Metrics":"type=METER, name=systems.ellora.core.exception.NoDataFoundMapper, count=1, mean_rate=0.3439532315922082, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, Time=2017-08-16_02:54:55.052"} {"Metrics":"type=METER, name=systems.ellora.core.exception.UnhandledRuntimeMapper, count=0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, Time=2017-08-16_02:54:55.053"}`

- Filter conf  
`filter { json { source => "message" } json { source => "Metrics" } kv{ value_split => "=" } date { match => ["Time","yyyy-MM-dd_HH:mm:ss,SSS"] target => "Time" } }`

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2017, 10:18am UTC](https://discuss.elastic.co/t/date-filter-problem/97211/10 "2017-08-16T10:18:40Z")

</div>

Three problems:

- The second json filter doesn't make sense because the `Metrics` field doesn't contain JSON.
- The kv filter must be configured to parse the `Metrics` field.
- The date filter uses the wrong pattern (comma vs. period right before the milliseconds).

---

<div class="post-metadata">

**Author:** ![Jonesthomas](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\
**Post date:** [August 16, 2017, 10:25am UTC](https://discuss.elastic.co/t/date-filter-problem/97211/11 "2017-08-16T10:25:29Z")

</div>

thank you so much @magnusbaeck

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 13, 2017, 10:40am UTC](https://discuss.elastic.co/t/date-filter-problem/97211/12 "2017-09-13T10:40:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.