XML parsing- Filter to format a date field

Elastic Stack Logstash
1

Hi,

I have the following index:

"hits" : [
  {
    "_index" : "test",
    "_type" : "doc",
    "_id" : "OWZhKmkBDjX7TOqqZgwX",
    "_score" : 1.0,
    "_source" : {
      "@timestamp" : "2019-02-26T15:16:52.527Z"
      "theXML" : {
        "statistics" : {
          "total" : {
            "stat" : [
              {
                "pass" : "1",
                "content" : "Critical Tests",
                "fail" : "0"
              },
              {
                "pass" : "1",
                "content" : "All Tests",
                "fail" : "0"
              }
            ]
          },
          "suite" : {
            "stat" : {
              "pass" : "1",
              "content" : "Test1",
              "id" : "s1",
              "name" : "Test1",
              "fail" : "0"
            }
          }
        },
        "errors" : { },
        "generated" : "20190215 15:03:20.437",
        "generator" : "Robot 3.0.3.dev20170213 (Python 2.7.15 on win32)",
        "suite" : {
          "status" : {
            "starttime" : "20190215 15:03:20.444",
            "endtime" : "20190215 15:03:44.198",
            "status" : "PASS"
          },

And I would like the "endtime" and "starttime" fields to be formatted at date (as opposed to strings).

I had the following filter which did not work:

filter {
xml { source => "message" target => "theXML" store_xml => true force_array => false }
split { field => "[theXML][suite][test][kw]" remove_field => "message"}
date {match => [ "%{[theXML][suite][status][endtime]}", "yyyyMMdd HH:mm:ss.SSS"}
date {match => [ "%{[theXML][suite][status][endtime]}", "yyyyMMdd HH:mm:ss.SSS"}
}

When I check the mapping of this index, starttime and endtime are still stored as string.

Could you point out what I am doing wrong here ?

Thank you in advance,

2

You are missing a ] before the final }.

That will parse endtime and store the result in @timestamp. If you want to overwrite endtime than add

target => "[theXML][suite][status][endtime]"

That will not fix the mapping on the existing index, but once you roll to a new index it should start appearing as a date.

However, the @timestamp on your message does not match either startime or endtime, so there is something else happening here.

3

Thanks, I was indeed missing a bracket..

However, after deleting the index and fixing the configuration file, which is now:

filter {
xml { source => "message" target => "theXML" store_xml => true force_array => false }
split { field => "[theXML][suite][test][kw]" remove_field => "message"}
date {match => [ "%{[theXML][suite][status][endtime]}", "yyyyMMdd HH:mm:ss.SSS"] target => "[theXML][suite][status][endtime]"}
date {match => [ "%{[theXML][suite][status][starttime]}", "yyyyMMdd HH:mm:ss.SSS"] target => "[theXML][suite][status][starttime]" }
}

The endtime and starttime fields are still stored as strings after reindexing.

4

Remove the %{}

1 Like
5

Thanks, that did it!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.