# Date filter not working
**URL:** https://discuss.elastic.co/t/date-filter-not-working/79679
**Category:** Logstash
**Created:** [March 23, 2017, 4:02am UTC](https://discuss.elastic.co/t/date-filter-not-working/79679 "2017-03-23T04:02:22Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@techols](https://discuss.elastic.co/u/techols)
#### Post date: [March 23, 2017, 4:02am UTC](https://discuss.elastic.co/t/date-filter-not-working/79679/1 "2017-03-23T04:02:22Z")
I have an xml file being ingested by logstash. The xml file has a timestamp element with a unix\_ms format (e.g., 1458827492928). My date filter is not properly parsing this and setting the @timestamp variable. I created a field to verify the value that I was using for my match statement and I can see that the added field contains a valid and the expected pre-converted timestamp value. Here is my filter section in my config file.
filter {
xml {
source =\> "message"
target =\> "doc"
}
## use the embedded Unix timestamp
date {
match =\> ["%{[doc][timestamp]}", "UNIX\_MS"]
}
}
The timestamp is being set with a current date even though the timestamp is for 2016 date.
Is there anything I am missing?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [March 23, 2017, 6:02am UTC](https://discuss.elastic.co/t/date-filter-not-working/79679/2 "2017-03-23T06:02:24Z")
Please show the result of a `stdout { codec => rubydebug }` output so we can see exactly what your events look like.
---
### Author:  [@techols](https://discuss.elastic.co/u/techols)
#### Post date: [March 23, 2017, 1:17pm UTC](https://discuss.elastic.co/t/date-filter-not-working/79679/3 "2017-03-23T13:17:23Z")
Here is a snippet of the output....
{
"path" =\> "/var/log/tdrs/scf/TDR.AES\_1.SCF\_1.ScfAgent\_5.20160324.1351-UTC.xml",
"@timestamp" =\> 2017-03-23T03:41:31.841Z,
"@metadata" =\> {
"path" =\> "/var/log/tdrs/scf/TDR.AES\_1.SCF\_1.ScfAgent\_5.20160324.1351-UTC.xml",
"host" =\> "leda-1"
},
"@version" =\> "1",
"host" =\> "leda-1",
"doc" =\> {
"corrID" =\> [
[0] "1458827492927\_daad49a7-b67a-4dbd-8db4-fa4a1a3c386e"
],
"dn" =\> [
[0] "AES\_1.SCF\_1.ScfAgent\_5.ServiceComponentNnsPXDC\_105"
],
"OrgName" =\> [
[0] "000-SLP013"
],
"eventType" =\> [
[0] "beginTrans"
],
"originator" =\> [
[0] "AES\_1.SCF\_1.ScfAgent\_5.ServiceComponentNnsPXDC\_105"
],
"version" =\> [
[0] "1.0"
],
"spID" =\> [
[0] "admin"
],
"AppName" =\> [
[0] "SLP013-APP003"
],
"OrgId" =\> [
[0] "pv"
],
"flowDirection" =\> [
[0] "0"
],
"AppId" =\> [
[0] "a75"
],
"operationType" =\> [
[0] "PxDCService\_GetCapabilities"
],
"seqNumber" =\> [
[0] "1"
],
"timestamp" =\> [
[0] "1458827492928"
]
},
"message" =\> "\n beginTrans\n PxDCService\_GetCapabilities\\n AES\_1.SCF\_1.ScfAgent\_5.ServiceComponentNnsPXDC\_105\n 1.0\n 1458827492928\n 1\n 1458827492927\_daad49a7-b67a-4dbd-8db4-fa4a1a3
c386e\n AES\_1.SCF\_1.ScfAgent\_5.ServiceComponentNnsPXDC\_105\n 0\\n pv\n 000-SLP013\n a75\n SLP013-A
PP003\n admin\n",
"type" =\> "scf\_xml",
"tags" =\> [
[0] "multiline"
]
}
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [March 23, 2017, 1:34pm UTC](https://discuss.elastic.co/t/date-filter-not-working/79679/4 "2017-03-23T13:34:01Z")
`[doc][timestamp]` is an array so if you want Logstash to parse its first element you need to reference it as `[doc][timestamp][0]`.
Are you running a very old Logstash? IIRC reasonably recent ones don't make arrays out of values unless it's necessary.
---
### Author:  [@techols](https://discuss.elastic.co/u/techols)
#### Post date: [March 23, 2017, 2:11pm UTC](https://discuss.elastic.co/t/date-filter-not-working/79679/5 "2017-03-23T14:11:35Z")
I have installed logstash-5.2.2-1.noarch. Updated the config as follows:
input {
file {
path =\> "/var/log/tdrs/scf/\*.xml"
sincedb\_path =\> "/var/log/logstash/.sincedb"
type =\> "scf\_xml"
start\_position =\> "beginning"
codec =\> multiline {
pattern =\> "(^\s|\)"
what =\> "previous"
}
}
}
filter {
## strip the XML prolog and envelope to get the actual TDRs
if [message] =~ "\" {
drop{}
}
xml {
source =\> "message"
target =\> "doc"
}
## use the embedded Unix timestamp
date {
match =\> ["%{[doc][timestamp][0]}", "UNIX\_MS"]
}
}
output {
# elasticsearch {
# hosts =\> ["localhost:9200"]
# index =\> "ngeag-tdrs-%{+YYYY.MM.dd}"
# }
stdout { codec =\> rubydebug { metadata =\> true } }
}
This config yields the same result.
[root@leda-1 logstash]# bin/logstash -f /etc/logstash/conf.d/\*.conf | more
WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
09:02:49.967 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=\>"main", "pipeline.workers"=\>24, "pipeline.batch.size"=\>125, "pi
peline.batch.delay"=\>5, "pipeline.max\_inflight"=\>3000}
09:02:50.182 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
09:02:50.242 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=\>9602}
{
"path" =\> "/var/log/tdrs/scf/TDR.AES\_1.SCF\_1.ScfAgent\_5.20160324.1351-UTC.xml",
"@timestamp" =\> 2017-03-23T14:08:33.842Z,
"@metadata" =\> {
"path" =\> "/var/log/tdrs/scf/TDR.AES\_1.SCF\_1.ScfAgent\_5.20160324.1351-UTC.xml",
"host" =\> "leda-1"
},
"@version" =\> "1",
"host" =\> "leda-1",
"doc" =\> {
"corrID" =\> [
[0] "1458827492723\_ea5b9d3d-ee7a-4966-850c-0131a3d2a9f5"
],
"subscriberID" =\> [
[0] "16308111111"
],
"dn" =\> [
[0] "AES\_1.SCF\_1.ScfAgent\_5.EnablerNnsMMS\_81"
],
"OrgName" =\> [
[0] "000-SLP009"
],
"eventType" =\> [
[0] "finalTrans"
],
"originator" =\> [
[0] "AES\_1.SCF\_1.ScfAgent\_5.EnablerNnsMMS\_81"
],
"version" =\> [
[0] "1.0"
],
"AppName" =\> [
[0] "SLP009-APP003"
],
"OrgId" =\> [
[0] "pl"
],
"flowDirection" =\> [
[0] "1"
],
"AppId" =\> [
[0] "a4x"
],
"operationType" =\> [
[0] "MMSEnabler\_NotifyMessageReception"
],
"seqNumber" =\> [
[0] "4"
],
"timestamp" =\> [
[0] "1458827492860"
]
},
"message" =\> "\n finalTrans\n MMSEnabler\_NotifyMessageReception\n AES\_
1.SCF\_1.ScfAgent\_5.EnablerNnsMMS\_81\n 1.0\n 1458827492860\n 4\n \ 1458827492723\_ea5b9d3d-ee7a-4966-850c-0131a3d2a9f5\n AES\_1.SCF\_1.ScfAgent\_5.EnablerNnsMMS\_81\n 1\<
> /flowDirection\>\n pl\n 000-SLP009\n a4x\n SLP009-APP003\n 16308111111\n",
> "type" =\> "scf\_xml",
> "tags" =\> [
> [0] "multiline"
> ]
> }
---
### Author:  [@GrahamHannington](https://discuss.elastic.co/u/GrahamHannington)
#### Post date: [April 1, 2017, 11:29pm UTC](https://discuss.elastic.co/t/date-filter-not-working/79679/6 "2017-04-01T23:29:31Z")
> [@techols](#):
>
> Is there anything I am missing?
Yes.
Insert the following setting in your `xml` filter:
```
force_array => false
```
(then you can remove the trailing `[0]` from subsequent field references to the new fields stored under `doc`.)
and insert the following `mutate` filter between your `xml` filter and your `date` filter:
```
mutate {
convert => {
"doc[timestamp]" => "integer"
}
}
```
Why? From the [docs](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-match):
> `UNIX_MS` - will parse int value
`int` (integer) value, not a string.
Example output:
```
"@timestamp" => 2016-03-24T13:51:32.860Z
```
That done: I don’t like the way the `xml` filter forces you to store content in a new first-level field (such as `doc`). Here’s a workaround, to be inserted after your `xml` filter:
```
# Copy XML content to first-level fields with all-lowercase names
ruby {
code => '
event.get("doc").each do |key, value|
event.set(key.downcase, value)
end
'
}
```
If you use this workaround, remove the leading `doc` (or `[doc]`) qualifier from subsequent field references. And remove the `doc` field before output. (I set an `xml` target under `@metadata`.)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 29, 2017, 11:29pm UTC](https://discuss.elastic.co/t/date-filter-not-working/79679/7 "2017-04-29T23:29:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.