# Parsing different dates from XML & Logs in one Date filed

**URL:** https://discuss.elastic.co/t/parsing-different-dates-from-xml-logs-in-one-date-filed/139205
**Category:** Logstash
**Created:** [July 9, 2018, 2:58pm UTC](https://discuss.elastic.co/t/parsing-different-dates-from-xml-logs-in-one-date-filed/139205 "2018-07-09T14:58:15Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![bab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bab/32/86201_2.png) [@bab](https://discuss.elastic.co/u/bab)
#### Post date: [July 9, 2018, 2:58pm UTC](https://discuss.elastic.co/t/parsing-different-dates-from-xml-logs-in-one-date-filed/139205/1 "2018-07-09T14:58:15Z")

</div>

Hi all,  
I have different types of timestamps coming into my logstash input from different logfiles xml and log type:  
My Configuration for filebeat looks like below:

```
filebeat.inputs:
- type: log
  paths:
    - /home/Dev/logs/server_error.log
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

- type: log
  paths:
    - /home/Dev/logs/icn.log
  multiline.pattern: '^\[[0-9]{1}/[0-9]{2}/[0-9]{2}'
  multiline.negate: true
  multiline.match: after

- type: log
  paths:
    - /home/Dev/logs/cpe.log
  multiline.pattern: '^\[[0-9]{1}/[0-9]{2}/[0-9]{2}'
  multiline.negate: true
  multiline.match: after

- type: log
  paths:
    - /home/Dev/logs/default_XML.log
  multiline.pattern: '^<record>'
  multiline.negate: true
  multiline.match: after

output.logstash:
    hosts: ["localhost:5044"]

```

my configuration for logstash looks like below:

```
input { beats {port => 5044 } }
    filter {

 if [fields][log_type] == "server_error" {
  grok {
   match => [ "message", 
             "%{TIMESTAMP_ISO8601:logtime} %{DATA:thread} %{DATA:sub} []* %{DATA:category} \- %{LOGLEVEL:sev} %{GREEDYDATA:message}" ]
  overwrite => ["message"]
  }
  mutate {
    replace => ["type", "server_error_log"]
    }
 }

 if [fields][log_type] == "icn" {
  grok {
   match => [ "message", 
             "%{DATESTAMP:logtime} %{DATA} %{DATA:thread} %{DATA:source} []* %{DATA:sev} %{DATA:module} %{DATA:log-level} %{DATA} \[\] %{DATA:java-method} %{GREEDYDATA:message}" ]
   overwrite => ["message"]
  }
  mutate {
    replace => ["type", "icn_log"]
    }
 }

 if [fields][log_type] == "cpe" {
  grok {
   match => [ "message",
             "%{DATESTAMP:logtime} %{DATA} %{DATA:thread} %{DATA:java-class} %{DATA:sev} %{DATA:java-package} %{DATA:java-method} %{GREEDYDATA:message}" ]

   overwrite => ["message"]
  }
  mutate {
    replace => ["type", "cpe_log"]
   }
 }

 if [fields][log_type] == "default_xml" {
  grok {
   match => [ "message", 
              "%{GREEDYDATA:raw_xml}" ]
  }
  xml {
   source => "raw_xml"
   #target => "xmldata"
   store_xml => "false"
   xpath => ["/record/date/text()","logtime"]
   xpath => ["/record/millis/text()","millis"]
   xpath => ["/record/sequence/text()","sequence"]
   xpath => ["/record/logger/text()","logger"]
   xpath => ["/record/level/text()","level"]
   xpath => ["/record/class/text()","class"]
   xpath => ["/record/method/text()","method"]
   xpath => ["/record/thread/text()","thread"]
   xpath => ["/record/message/text()","msg"]
  } }

date {
       match => ["logtime", "yyyy-MM-dd'T'HH:mm:ss.SSS", "yyyy-MM-dd'T'HH:mm:ss", "M/dd/yy HH:mm:ss:SSS"]
       target => "Timestamp"
       }
    mutate { remove_field => ["logtime"] }
    }

output {   
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "index_test"
     }
}

```

I have 4 different log files, one of them is as xml file, I have parsed everything by grok in logstash and saved in individual feild (for later visualization in kibana) but I also have the dates of each log file through (Date Filter parsed), every things work fine.  
But I have a problem with the date from xml file I can not parse in (Date Filter). on the same Filed (logtime)  
can someone help

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 9, 2018, 6:17pm UTC](https://discuss.elastic.co/t/parsing-different-dates-from-xml-logs-in-one-date-filed/139205/2 "2018-07-09T18:17:54Z")

</div>

How could we possibly help out without knowing the contents of the `logtime` field?

---

<div class="post-metadata">

### Author: ![bab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bab/32/86201_2.png) [@bab](https://discuss.elastic.co/u/bab)
#### Post date: [July 10, 2018, 4:45am UTC](https://discuss.elastic.co/t/parsing-different-dates-from-xml-logs-in-one-date-filed/139205/3 "2018-07-10T04:45:50Z")

</div>

Hi @magnusbaeck,  
sure below is my logs:

the first log -\> server\_error:

> **2018-04-17T15:19:20.313** FC772FA2 CBR FNRCE0000I - INFO binary\_recognizer\_....

the secend log -\> icn & cpe have the same Datestamp format like below:

> **[6/13/18 5:29:50:575 CEST]** 0000006d SystemOut O CIWEB Perf : com.ibm.ecm.configuration....

the third log -\> default\_XML

```
<record>
    <date>2018-05-31T09:06:28</date>
    <millis>1527750388992</millis>
    <sequence>825425</sequence>
    <logger>com.ibm.es.nuvo.inyo.ingest.DocIngestHandlerMulti</logger>
    <level>INFO</level>
    <class>com.ibm.es.nuvo.inyo.ingest.DocIngestHandlerMulti</class>
    <method>addDocument</method>
    <thread>908</thread>
    <message>Last document notification received for collection /</message>

```

according to my configuration the contents of the **logtime** field? is the dates from all logs type except the log ( **default xml** ). i will to pars also this date into **logtime** field with the others dates so that i can in kibana using **logtime** filter for all logs type.

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 10, 2018, 1:17pm UTC](https://discuss.elastic.co/t/parsing-different-dates-from-xml-logs-in-one-date-filed/139205/4 "2018-07-10T13:17:15Z")

</div>

Okay, so in the last example you posted `logtime` _should_ contain "2018-05-31T09:06:28" which _should_ be parseable by the date filter you have. Please remove the `mutate { remove_field => ["logtime"] }` filter and show the raw event produced by Logstash when it's fed an XML document. Copy/paste the text from the JSON tab in Kibana or use a `stdout { codec => rubydebug }` output.

---

<div class="post-metadata">

### Author: ![bab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bab/32/86201_2.png) [@bab](https://discuss.elastic.co/u/bab)
#### Post date: [July 10, 2018, 3:23pm UTC](https://discuss.elastic.co/t/parsing-different-dates-from-xml-logs-in-one-date-filed/139205/5 "2018-07-10T15:23:15Z")

</div>

i removed the mutat filter und i get this:

```
{
       "raw_xml" => "<record>\n <date>2018-05-31T09:06:27</date>\n <millis>1527750387452</millis>\n <sequence>825423</sequence>\n <logger>com.ibm.es.nuvo.inyo.ingest.DocIngestHandlerMulti</logger>\n <level>INFO</level>\n <class>com.ibm.es.nuvo.inyo.ingest.DocIngestHandlerMulti</class>\n <method>startOutputingMessages</method>\n <thread>892</thread>\n <message>Starting subscription for collection /opt/filenet/collections/ADR_Folder_20170628111748_310CC8C62293451FB1A1BAB3998CA2E2 for client 3244</message>\n</record>\n",
        "logger" => [
        [0] "com.ibm.es.nuvo.inyo.ingest.DocIngestHandlerMulti"
    ],
      "@version" => "1",
    "prospector" => {
        "type" => "log"
    },
        "thread" => [
        [0] "892"
    ],
         "input" => {
        "type" => "log"
    },
    "@timestamp" => 2018-07-10T15:12:32.263Z,
      "sequence" => [
        [0] "825423"
    ],
        "source" => "/home/badr/Dev/logs/mini_default0.log",
          "host" => {
        "name" => "badr-VirtualBox"
    },
        "method" => [
        [0] "startOutputingMessages"
    ],
        "millis" => [
        [0] "1527750387452"
    ],
        "fields" => {
             "log_type" => "default0",
          "environment" => "test",
        "document_type" => "default0"
    },
          "beat" => {
        "hostname" => "badr-VirtualBox",
         "version" => "6.3.0",
            "name" => "badr-VirtualBox"
    },
        "offset" => 1008,
           "msg" => [
        [0] "Starting subscription for collection /opt/filenet/collections/ADR_Folder_20170628111748_310CC8C62293451FB1A1BAB3998CA2E2 for client 3244"
    ],
         "class" => [
        [0] "com.ibm.es.nuvo.inyo.ingest.DocIngestHandlerMulti"
    ],
       "message" => "<record>\n <date>2018-05-31T09:06:27</date>\n <millis>1527750387452</millis>\n <sequence>825423</sequence>\n <logger>com.ibm.es.nuvo.inyo.ingest.DocIngestHandlerMulti</logger>\n <level>INFO</level>\n <class>com.ibm.es.nuvo.inyo.ingest.DocIngestHandlerMulti</class>\n <method>startOutputingMessages</method>\n <thread>892</thread>\n <message>Starting subscription for collection /opt/filenet/collections/ADR_Folder_20170628111748_310CC8C62293451FB1A1BAB3998CA2E2 for client 3244</message>\n</record>\n",
       "logtime" => [
        [0] "2018-05-31T09:06:27"
    ],
         "level" => [
        [0] "INFO"
    ],
          "tags" => [
        [0] "beats_input_codec_plain_applied",
        [1] "_dateparsefailure"
    ]
}

```

So the date (2018-05-31T09:06:28) of the XML file is already parsed in `logtime` but that does not help me if I can't parse it in `eventlog` where all dates parsed (_from the other logs_). because `eventlog` is my datefilter on kibana at the index pattern creating and there I find only the dates from the other logs (p8, icn, cpe). i need to know how i can get this issue, is important _(for me )_ to go on with the Visualiz on kibana

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 10, 2018, 4:45pm UTC](https://discuss.elastic.co/t/parsing-different-dates-from-xml-logs-in-one-date-filed/139205/6 "2018-07-10T16:45:31Z")

</div>

When you use xpath you always get an array, usually containing a single object. You can add this to the section of the config that processes xml

```
if [logtime] { mutate { replace => { "logtime" => "%{[logtime][0]}" } } }
```

---

<div class="post-metadata">

### Author: ![bab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bab/32/86201_2.png) [@bab](https://discuss.elastic.co/u/bab)
#### Post date: [July 11, 2018, 7:54am UTC](https://discuss.elastic.co/t/parsing-different-dates-from-xml-logs-in-one-date-filed/139205/7 "2018-07-11T07:54:00Z")

</div>

@Badger thank you again, it works fine now, i get all dates in one filed (`eventlog`)

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 8, 2018, 7:54am UTC](https://discuss.elastic.co/t/parsing-different-dates-from-xml-logs-in-one-date-filed/139205/8 "2018-08-08T07:54:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.