# Date Filter on Unix Epoch Failing **URL:** \ **Category:** Logstash\ **Created:** [April 10, 2020, 12:12am UTC](https://discuss.elastic.co/t/date-filter-on-unix-epoch-failing/227422 "2020-04-10T00:12:39Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![ben-10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben-10/32/12306_2.png) [@ben-10](https://discuss.elastic.co/u/ben-10)\ **Post date:** [April 10, 2020, 12:12am UTC](https://discuss.elastic.co/t/date-filter-on-unix-epoch-failing/227422/1 "2020-04-10T00:12:39Z")
The date filter fails when attempting to match a unix epoch integer field and I can't figure out why. The data ingested is from xml using the following logstash config: ```auto input { file { path => ["/home/ben/dmarc-reports/AvidXchange.com!domain.com!1586232003!1586318403.xml"] start_position => "beginning" mode => "read" sincedb_path => "/home/ben/.sincedb-dmarctest" exit_after_read => true file_completed_action => "log" file_completed_log_path => "/home/ben/dmarccompleted.log" codec => multiline { pattern => "" negate => "true" what => "previous" } } } filter { xml { source => "message" target => "parsed_xml" store_xml => false xpath => [ "/feedback/report_metadata/org_name/text()", "reporting_org", "/feedback/report_metadata/report_id/text()", "report_id", "/feedback/report_metadata/date_range/begin/text()", "report_start", "/feedback/report_metadata/date_range/end/text()", "report_end", "/feedback/record/row/source_ip/text()", "email_server_ip", "/feedback/record/row/policy_evaluated/dkim/text()", "policy_dkim", "/feedback/record/row/policy_evaluated/spf/text()", "policy_spf", "/feedback/auth_results/dkim/result/text()", "auth_dkim", "/feedback/auth_results/spf/result/text()", "auth_spf" ] } mutate { convert => { "report_start" => "integer" "report_end" => "integer" } } date { match => ["report_start", "UNIX", "UNIX_MS"] #target => "report_start" } date { match => ["report_end", "UNIX", "UNIX_MS"] target => "report_end_time" } dns { reverse => ["email_server_ip"] } if '' in [message] { drop {} } } output { elasticsearch { index => "logstash_dmarcxml_%{+YYYY.MM.dd}" } } ``` Also does anyone know a list of of xpath functions that logstash accepts? ---
**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\ **Post date:** [April 10, 2020, 12:34am UTC](https://discuss.elastic.co/t/date-filter-on-unix-epoch-failing/227422/2 "2020-04-10T00:34:27Z")
Hello @ben-10 Would it be possible to replace the `elasticsearch { }` output with `stdout { codec => rubydebug }` so you can troubleshoot which is the actual value on `report_start` and `report_end` ? Regarding `XPath`, Logstash uses the Ruby library [https://github.com/sparklemotion/nokogiri](https://github.com/sparklemotion/nokogiri) and should support a wide range of XPath expressions. See [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-xml.html#plugins-filters-xml-xpath). I would also suggest to move the following line to the beginning of the pipeline (so it avoids going through the other filters). ```auto if '' in [message] { drop {} } ``` Even better, I would write the `multiline` codec to include `` (if it is repeated for every XML root. Also, please upgrade the XML filter to version 4.1.0 as it includes a fix for a memory leak in case of malformed XML: > ---
**Author:** ![ben-10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben-10/32/12306_2.png) [@ben-10](https://discuss.elastic.co/u/ben-10)\ **Post date:** [April 10, 2020, 1:38am UTC](https://discuss.elastic.co/t/date-filter-on-unix-epoch-failing/227422/3 "2020-04-10T01:38:56Z")
Thanks for the tip on upgrading the plugin! And I moved the conditional statement higher up the list. Here's the snippet of output for those two fields from the ruby debug: ```auto "report_start" => [ [0] 1586232003 ], "report_end" => [ [0] 1586318403 ], ``` Also forgot to mention the functions link on that page going to w3c is dead. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 10, 2020, 5:03am UTC](https://discuss.elastic.co/t/date-filter-on-unix-epoch-failing/227422/4 "2020-04-10T05:03:32Z")
The fields you want to convert are arrays of strings, not strings. I would try something like this: ``` date { match => ["report_end[0]", "UNIX", "UNIX_MS" ] target => "report_end_time" } ``` ---
**Author:** ![ben-10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben-10/32/12306_2.png) [@ben-10](https://discuss.elastic.co/u/ben-10)\ **Post date:** [April 10, 2020, 1:07pm UTC](https://discuss.elastic.co/t/date-filter-on-unix-epoch-failing/227422/5 "2020-04-10T13:07:27Z")
Was worth a shot, but that came with: ```auto Invalid FieldReference: `report_start[0]` ``` Why would that field be an array of strings after mutating it into an integer before running the date filter? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 10, 2020, 1:18pm UTC](https://discuss.elastic.co/t/date-filter-on-unix-epoch-failing/227422/6 "2020-04-10T13:18:53Z")
Sorry, it seems to be an array of integers. Well, only one integer actually, but still an array. ---
**Author:** ![ben-10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben-10/32/12306_2.png) [@ben-10](https://discuss.elastic.co/u/ben-10)\ **Post date:** [April 10, 2020, 1:23pm UTC](https://discuss.elastic.co/t/date-filter-on-unix-epoch-failing/227422/7 "2020-04-10T13:23:02Z")
No worries, with XML documents I feel like I know nothing. I foresee my day filed with reading technical documents 🤓 ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 10, 2020, 1:40pm UTC](https://discuss.elastic.co/t/date-filter-on-unix-epoch-failing/227422/8 "2020-04-10T13:40:02Z")
> [@ben-10](#): > > Invalid FieldReference: `report_start[0]` Try [report\_start][0] ---
**Author:** ![ben-10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben-10/32/12306_2.png) [@ben-10](https://discuss.elastic.co/u/ben-10)\ **Post date:** [April 10, 2020, 4:09pm UTC](https://discuss.elastic.co/t/date-filter-on-unix-epoch-failing/227422/9 "2020-04-10T16:09:43Z")
And that worked! I also created a template in elasticsearch with static mappings to cast the "report\_start" and "report\_end" as integers instead of relying on logstash to mutate and convert. Also create target fields "report\_start\_time" and "report\_end\_time" mapped as dates. The last time I tried all this was back in the 5.x stack storing the xml into sub-fields instead of using xpaths. Lots of new things to learn! Thank you for all the help! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 8, 2020, 4:17pm UTC](https://discuss.elastic.co/t/date-filter-on-unix-epoch-failing/227422/10 "2020-05-08T16:17:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.