# Logstash 7.2 can not match date on JSON data from Cisco telemetry

**URL:** <https://discuss.elastic.co/t/logstash-7-2-can-not-match-date-on-json-data-from-cisco-telemetry/191557>\
**Category:** Logstash\
**Created:** [July 22, 2019, 1:49am UTC](https://discuss.elastic.co/t/logstash-7-2-can-not-match-date-on-json-data-from-cisco-telemetry/191557 "2019-07-22T01:49:42Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![tlittirut](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tlittirut/32/45910_2.png) [@tlittirut](https://discuss.elastic.co/u/tlittirut)\
**Post date:** [July 22, 2019, 1:49am UTC](https://discuss.elastic.co/t/logstash-7-2-can-not-match-date-on-json-data-from-cisco-telemetry/191557/1 "2019-07-22T01:49:42Z")

</div>

Hello !

I'm a newbie on logstash and trying to decode cisco telemetry data in json format. My code in logstash are the following:  
filter {  
if [type] == "xjson" {  
json {  
source =\> "message"  
}  
mutate {  
remove\_field =\> ["message"]  
}  
date {  
match =\> ["[Rows][Timestamp]", "UNIX\_MS", "UNIX"]  
}  
date {  
match =\> ["[Telemetry][msg\_timestamp]", "UNIX\_MS", "UNIX" ]  
}  
}  
}  
The json data are the following:  
{  
"\_index": "logstash-2019.07.21-000001",  
"\_type": "\_doc",  
"\_id": "5fRfFGwBWU64Pkw1pznQ",  
"\_version": 1,  
"\_score": null,  
"\_source": {  
"@timestamp": "2019-07-21T11:51:48.864Z",  
"type": "xjson",  
"@version": "1",  
"Telemetry": {  
"collection\_id": 124647,  
"msg\_timestamp": 1563709831176,  
"collection\_end\_time": 1563709831185,  
"node\_id\_str": "ncs55",  
"subscription\_id\_str": "int2",  
"collection\_start\_time": 1563709831176,  
"encoding\_path": "Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/generic-counters"  
},  
"Rows": [  
{  
"Content": {  
"last-discontinuity-time": 1563431358,  
"packets-received": 0,  
"runt-packets-received": 0,  
"seconds-since-packet-sent": 4294967295,  
"input-queue-drops": 0,  
"multicast-packets-sent": 0,  
"seconds-since-packet-received": 4294967295,  
"bytes-sent": 0,  
"packets-sent": 0,  
"input-aborts": 0,  
"seconds-since-last-clear-counters": 0,  
"parity-packets-received": 0,  
"output-underruns": 0,  
"availability-flag": 0,  
"throttled-packets-received": 0,  
"crc-errors": 0,  
"framing-errors-received": 0,  
"broadcast-packets-received": 0,  
"output-buffers-swapped-out": 0,  
"applique": 0,  
"broadcast-packets-sent": 0,  
"input-drops": 0,  
"output-drops": 0,  
"resets": 0,  
"input-ignored-packets": 0,  
"output-buffer-failures": 0,  
"output-queue-drops": 0,  
"giant-packets-received": 0,  
"input-errors": 0,  
"last-data-time": 1563709820,  
"multicast-packets-received": 0,  
"output-errors": 0,  
"input-overruns": 0,  
"carrier-transitions": 0,  
"unknown-protocol-packets-received": 0,  
"bytes-received": 0  
},  
"Timestamp": 1563709831182,  
"Keys": {  
"interface-name": "Null0"  
}  
},  
{  
"Content": {  
"last-discontinuity-time": 1563431445,  
"packets-received": 0,  
"runt-packets-received": 0,  
"seconds-since-packet-sent": 4294967295,  
"input-queue-drops": 0,  
"multicast-packets-sent": 0,  
"seconds-since-packet-received": 4294967295,  
"bytes-sent": 0,  
"packets-sent": 0,  
"input-aborts": 0,  
"seconds-since-last-clear-counters": 0,  
"parity-packets-received": 0,  
"output-underruns": 0,  
"availability-flag": 0,  
"throttled-packets-received": 0,  
"crc-errors": 0,  
"framing-errors-received": 0,  
"broadcast-packets-received": 0,  
"output-buffers-swapped-out": 0,  
"applique": 0,  
"broadcast-packets-sent": 0,  
"input-drops": 0,  
"output-drops": 0,  
"resets": 0,  
"input-ignored-packets": 0,  
"output-buffer-failures": 0,  
"output-queue-drops": 0,  
"giant-packets-received": 0,  
"input-errors": 0,  
"last-data-time": 1563709791,  
"multicast-packets-received": 0,  
"output-errors": 0,  
"input-overruns": 0,  
"carrier-transitions": 0,  
"unknown-protocol-packets-received": 0,  
"bytes-received": 0  
},  
"Timestamp": 1563709831182,  
"Keys": {  
"interface-name": "HundredGigE0/0/1/2"  
}  
},  
{  
"Content": {  
"last-discontinuity-time": 1563431445,  
"packets-received": 0,  
"runt-packets-received": 0,  
"seconds-since-packet-sent": 4294967295,  
"input-queue-drops": 0,  
"multicast-packets-sent": 0,  
"seconds-since-packet-received": 4294967295,  
"bytes-sent": 0,  
"packets-sent": 0,  
"input-aborts": 0,  
"seconds-since-last-clear-counters": 0,  
"parity-packets-received": 0,  
"output-underruns": 0,  
"availability-flag": 0,  
"throttled-packets-received": 0,  
"crc-errors": 0,  
"framing-errors-received": 0,  
"broadcast-packets-received": 0,  
"output-buffers-swapped-out": 0,  
"applique": 0,  
"broadcast-packets-sent": 0,  
"input-drops": 0,  
"output-drops": 0,  
"resets": 0,  
"input-ignored-packets": 0,  
"output-buffer-failures": 0,  
"output-queue-drops": 0,  
"giant-packets-received": 0,  
"input-errors": 0,  
"last-data-time": 1563709791,  
"multicast-packets-received": 0,  
"output-errors": 0,  
"input-overruns": 0,  
"carrier-transitions": 0,  
"unknown-protocol-packets-received": 0,  
"bytes-received": 0  
},  
"Timestamp": 1563709831182,  
"Keys": {  
"interface-name": "HundredGigE0/0/1/1"  
}  
}  
],  
"Source": "172.16.200.60:23549"  
},  
"fields": {  
"@timestamp": [  
"2019-07-21T11:51:48.864Z"  
]  
},  
"sort": [  
1563709908864  
]  
}

Every time, the Rows.Timestamp and Telemetry.msg\_timestamp always matched to number.  
Could someone guide me how to match this data?

Thank you in advance!  
Tosamon L.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 22, 2019, 12:22pm UTC](https://discuss.elastic.co/t/logstash-7-2-can-not-match-date-on-json-data-from-cisco-telemetry/191557/2 "2019-07-22T12:22:41Z")

</div>

Rows is an array, so you could use

```
date { match => ["[Rows][0][Timestamp]", "UNIX_MS", "UNIX"] }
```

---

<div class="post-metadata">

**Author:** ![tlittirut](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tlittirut/32/45910_2.png) [@tlittirut](https://discuss.elastic.co/u/tlittirut)\
**Post date:** [July 22, 2019, 11:58pm UTC](https://discuss.elastic.co/t/logstash-7-2-can-not-match-date-on-json-data-from-cisco-telemetry/191557/3 "2019-07-22T23:58:50Z")

</div>

> [@tlittirut](#):
>
> {  
> "\_index": "logstash-2019.07.21-000001",  
> "\_type": "\_doc",  
> "\_id": "5fRfFGwBWU64Pkw1pznQ",  
> "\_version": 1,  
> "\_score": null,  
> "\_source": {  
> "@timestamp": "2019-07-21T11:51:48.864Z",  
> "type": "xjson",  
> "@version": "1",  
> "Telemetry": {  
> "collection\_id": 124647,  
> "msg\_timestamp": 1563709831176,  
> "collection\_end\_time": 1563709831185,  
> "node\_id\_str": "ncs55",  
> "subscription\_id\_str": "int2",  
> "collection\_start\_time": 1563709831176,  
> "encoding\_path": "Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/generic-counters"  
> },  
> "Rows": [  
> {  
> "Content": {  
> "last-discontinuity-time": 1563431358,  
> "packets-received": 0,  
> "runt-packets-received": 0,  
> "seconds-since-packet-sent": 4294967295,  
> "input-queue-drops": 0,  
> "multicast-packets-sent": 0,  
> "seconds-since-packet-received": 4294967295,  
> "bytes-sent": 0,  
> "packets-sent": 0,  
> "input-aborts": 0,  
> "seconds-since-last-clear-counters": 0,  
> "parity-packets-received": 0,  
> "output-underruns": 0,  
> "availability-flag": 0,  
> "throttled-packets-received": 0,  
> "crc-errors": 0,  
> "framing-errors-received": 0,  
> "broadcast-packets-received": 0,  
> "output-buffers-swapped-out": 0,  
> "applique": 0,  
> "broadcast-packets-sent": 0,  
> "input-drops": 0,  
> "output-drops": 0,  
> "resets": 0,  
> "input-ignored-packets": 0,  
> "output-buffer-failures": 0,  
> "output-queue-drops": 0,  
> "giant-packets-received": 0,  
> "input-errors": 0,  
> "last-data-time": 1563709820,  
> "multicast-packets-received": 0,  
> "output-errors": 0,  
> "input-overruns": 0,  
> "carrier-transitions": 0,  
> "unknown-protocol-packets-received": 0,  
> "bytes-received": 0  
> },  
> "Timestamp": 1563709831182,  
> "Keys": {  
> "interface-name": "Null0"  
> }  
> },  
> {  
> "Content": {  
> "last-discontinuity-time": 1563431445,  
> "packets-received": 0,  
> "runt-packets-received": 0,  
> "seconds-since-packet-sent": 4294967295,  
> "input-queue-drops": 0,  
> "multicast-packets-sent": 0,  
> "seconds-since-packet-received": 4294967295,  
> "bytes-sent": 0,  
> "packets-sent": 0,  
> "input-aborts": 0,  
> "seconds-since-last-clear-counters": 0,  
> "parity-packets-received": 0,  
> "output-underruns": 0,  
> "availability-flag": 0,  
> "throttled-packets-received": 0,  
> "crc-errors": 0,  
> "framing-errors-received": 0,  
> "broadcast-packets-received": 0,  
> "output-buffers-swapped-out": 0,  
> "applique": 0,  
> "broadcast-packets-sent": 0,  
> "input-drops": 0,  
> "output-drops": 0,  
> "resets": 0,  
> "input-ignored-packets": 0,  
> "output-buffer-failures": 0,  
> "output-queue-drops": 0,  
> "giant-packets-received": 0,  
> "input-errors": 0,  
> "last-data-time": 1563709791,  
> "multicast-packets-received": 0,  
> "output-errors": 0,  
> "input-overruns": 0,  
> "carrier-transitions": 0,  
> "unknown-protocol-packets-received": 0,  
> "bytes-received": 0  
> },  
> "Timestamp": 1563709831182,  
> "Keys": {  
> "interface-name": "HundredGigE0/0/1/2"  
> }  
> },  
> {  
> "Content": {  
> "last-discontinuity-time": 1563431445,  
> "packets-received": 0,  
> "runt-packets-received": 0,  
> "seconds-since-packet-sent": 4294967295,  
> "input-queue-drops": 0,  
> "multicast-packets-sent": 0,  
> "seconds-since-packet-received": 4294967295,  
> "bytes-sent": 0,  
> "packets-sent": 0,  
> "input-aborts": 0,  
> "seconds-since-last-clear-counters": 0,  
> "parity-packets-received": 0,  
> "output-underruns": 0,  
> "availability-flag": 0,  
> "throttled-packets-received": 0,  
> "crc-errors": 0,  
> "framing-errors-received": 0,  
> "broadcast-packets-received": 0,  
> "output-buffers-swapped-out": 0,  
> "applique": 0,  
> "broadcast-packets-sent": 0,  
> "input-drops": 0,  
> "output-drops": 0,  
> "resets": 0,  
> "input-ignored-packets": 0,  
> "output-buffer-failures": 0,  
> "output-queue-drops": 0,  
> "giant-packets-received": 0,  
> "input-errors": 0,  
> "last-data-time": 1563709791,  
> "multicast-packets-received": 0,  
> "output-errors": 0,  
> "input-overruns": 0,  
> "carrier-transitions": 0,  
> "unknown-protocol-packets-received": 0,  
> "bytes-received": 0  
> },  
> "Timestamp": 1563709831182,  
> "Keys": {  
> "interface-name": "HundredGigE0/0/1/1"  
> }  
> }  
> ],  
> "Source": "172.16.200.60:23549"  
> },  
> "fields": {  
> "@timestamp": [  
> "2019-07-21T11:51:48.864Z"  
> ]  
> },  
> "sort": [  
> 1563709908864  
> ]  
> }

Thank you Badger, I have been tried [Rows][0][Timestamp] but it was not work.  
In kibana is shown this "Objects in arrays are not well supported"

 ![58](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a9b1907a154fdce22feb61cab0e8d942fab25293.png)

---

<div class="post-metadata">

**Author:** ![tlittirut](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tlittirut/32/45910_2.png) [@tlittirut](https://discuss.elastic.co/u/tlittirut)\
**Post date:** [July 23, 2019, 2:19pm UTC](https://discuss.elastic.co/t/logstash-7-2-can-not-match-date-on-json-data-from-cisco-telemetry/191557/4 "2019-07-23T14:19:08Z")

</div>

After tried.  
date {  
match =\> ["[Rows][0][Timestamp]", "UNIX\_MS"]  
target =\> ["[Rows][0][Timestamp]" ]  
}  
I got the error msg from logstash:  
, {:\_id=\>nil, :\_index=\>"logstash", :\_type=\>"\_doc", :routing=\>nil}, #LogStash::Event:0x21098a8d], :response=\>{"index"=\>{"\_index"=\>"logstash-2019.07.23-000001", "\_type"=\>"\_doc", "\_id"=\>"yTQwH2wBNWT9XzBB5\_6g", "status"=\>400, "error"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"mapper [Rows.Timestamp] of different type, current\_type [long], merged\_type [date]"}}}}

Need advice

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 23, 2019, 2:25pm UTC](https://discuss.elastic.co/t/logstash-7-2-can-not-match-date-on-json-data-from-cisco-telemetry/191557/5 "2019-07-23T14:25:25Z")

</div>

> [@tlittirut](#):
>
> "mapper [Rows.Timestamp] of different type, current\_type [long], merged\_type [date]"}

OK, so it successfully parsed the field. The problem is you already have documents in your index where that field is a number like 1563709831182, and it cannot be a number on some documents and a date on others.

One option is to wait until tonight, when the index rolls over to logstash-2019.07.24-000001. At that point the message should disappear. Another option is to change the target option on the date filter.

---

<div class="post-metadata">

**Author:** ![tlittirut](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tlittirut/32/45910_2.png) [@tlittirut](https://discuss.elastic.co/u/tlittirut)\
**Post date:** [July 23, 2019, 3:06pm UTC](https://discuss.elastic.co/t/logstash-7-2-can-not-match-date-on-json-data-from-cisco-telemetry/191557/6 "2019-07-23T15:06:52Z")

</div>

In Rows field, there is the array of nested field as shown below:  
There is the timestamp for each interface. After tried to target date to the [Rows][0][Timestamp] field , but the other fields such as [Rows][1][Timestamp] ... [Rows][n][Timestamp] are still the number. That might the case of logstash error different type.

Rows {  
"Content": {  
"output-drops": 0,  
"resets": 0,  
"input-queue-drops": 0  
},  
"Timestamp": 1563892066066,  
"Keys": {  
"interface-name": "Null0"  
}  
},  
{  
"Content": {  
"output-drops": 0,  
"resets": 0,  
"input-queue-drops": 0  
},  
"Timestamp": 1563892066066,  
"Keys": {  
"interface-name": "MgmtEth0/RP0/CPU0/0"  
}  
},  
{  
"Content": {  
"output-drops": 0,  
"last-discontinuity-time": 1563431445,  
"packets-sent": 6072,  
"resets": 0,  
"input-queue-drops": 0  
},  
"Timestamp": 1563892066066,  
"Keys": {  
"interface-name": "GigabitEthernet0/0/0/0"  
}  
},  
{  
"Content": {  
"output-drops": 0,  
"last-discontinuity-time": 1563431445,  
"packets-sent": 20192179,  
"resets": 0,  
"input-queue-drops": 0  
},  
"Timestamp": 1563892066066,  
"Keys": {  
"interface-name": "GigabitEthernet0/0/0/23"  
}  
},  
{  
"Content": {  
"output-drops": 0,  
"last-discontinuity-time": 1563431445,  
"packets-sent": 0,  
"availability-flag": 0,  
"resets": 0,  
"input-queue-drops": 0  
},  
"Timestamp": 1563892066066,  
"Keys": {  
"interface-name": "GigabitEthernet0/0/0/15"  
}  
}  
}

==  
I tryied the following code, but it added the Rows.Keys.Timestamp only on the first field on the array.

date {  
match =\> ["[Rows][0][Timestamp]", "UNIX\_MS"]  
target =\> ["[@Interfacetime]" ]  
add\_field =\> { "[Rows][0][Keys][Timestamp]" =\> "%{[@Interfacetime]}" }  
}  
How can we match all data in arrays of nested field ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 23, 2019, 3:52pm UTC](https://discuss.elastic.co/t/logstash-7-2-can-not-match-date-on-json-data-from-cisco-telemetry/191557/7 "2019-07-23T15:52:09Z")

</div>

You will need to use ruby to convert the timestamp in every entry in an array.

```
    ruby {
        code => '
            r = event.get("[Rows]")
            r.each_index { |x|
                r[x]["Keys"]["Timestamp"] = LogStash::Timestamp.at(r[x]["Timestamp"].to_f/1000)
            }
            event.set("Rows", r)
        '
    }

```

I was surprised to see that 1563709831182 gets converted to 2019-07-21T11:50:31.181Z, which appears to me to be off by a millisecond.

---

<div class="post-metadata">

**Author:** ![tlittirut](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tlittirut/32/45910_2.png) [@tlittirut](https://discuss.elastic.co/u/tlittirut)\
**Post date:** [July 23, 2019, 11:59pm UTC](https://discuss.elastic.co/t/logstash-7-2-can-not-match-date-on-json-data-from-cisco-telemetry/191557/8 "2019-07-23T23:59:55Z")

</div>

> [@Badger](#):
>
> ruby { code =\> ' r = event.get("[Rows]") r.each\_index { |x| r["Keys"]["Timestamp"] = LogStash::Timestamp.at(r["Timestamp"].to\_f/1000) } event.set("Rows", r) ' }

Thank you so much!!! @Badger, This ruby code fixed my problem.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 20, 2019, 11:59pm UTC](https://discuss.elastic.co/t/logstash-7-2-can-not-match-date-on-json-data-from-cisco-telemetry/191557/9 "2019-08-20T23:59:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.