Does the data input filter work on nested fields?

Elastic Stack Logstash
1

Hello All,

I'm trying to get logstash to parse the date from a nested field in a JSON blob and are having some issues doing so.

Here is what the JSON blob looks like:

{
"count"27,
"data":[{
"hotspotIdentifier":"00193BECD438",
"dateStart":"2015-06-03 19:11:15",
"dateStop":"2015-06-03 19:11:55",
"sessionTime":"39",
"traffic":362264,
"terminateCause":"User-Request",
"username":"4C0EDC30",
"macAddressUser":"24fd5214637c",
"ipAddressUser":"192.168.8.1",
"venueId":"ff8080814d91f65e014dba30cc091d83",
"online":false
}]
}

There are more data nested fields but one will do for this issue.

In my logstash config I have this:

filter {
split {
field => "data"
}
date {
match => [ "[startDate]", "yyyy-MM-dd HH:mm:ss" ]
add_tag => [ "startDate_Matched" ]
}
date {
match => [ "[data][stopDate]", "yyyy-MM-dd HH:mm:ss" ]
add_tag => [ "stopDate_Matched" ]
}
}

Which does split out the fields into data.x when looked at from Kibana but no matter what I try I cannot get the date match to work and I never get to see the startDate_matched/stopDate_Matched tags either. There's no errors and no grokparsefailures or dateparsefailures either.

If I remove the kv section I just get one long line in _source and lots of parse failures.

This is what the document (well a similar one but they all looks the same) looks like in Kibana:

{
"_index": "cloud4wi-2016.07.18",
"_type": "clod4wi_connections",
"_id": "AVX-LS-VyWbKOs9i0BQf",
"_score": 1,
"_source": {
"count": 27,
"data": {
"hotspotIdentifier": "00193BECD438",
"dateStart": "2015-06-03 19:15:48",
"dateStop": "2015-06-03 19:16:17",
"sessionTime": "30",
"traffic": 643328,
"terminateCause": "User-Request",
"username": "4C0EDC30",
"macAddressUser": "24fd5214637c",
"ipAddressUser": "192.168.8.1",
"venueId": "ff8080814d91f65e014dba30cc091d83",
"online": false
},
"@version": "1",
"@timestamp": "2016-07-18T13:23:40.444Z",
"http_poller_metadata": {
"name": "connections",
"host": "LIN-UBU-64-AMC-FE2",
"request": {
"method": "get",
"url": "https://api.cloud4wi.com/v2/connections?dateStart=2015-01-01&dateStop=2016-07-01&limit=1000&offset=0&api_version=v2.0&api_key=xxxx&api_secret=xxxx"
},
"runtime_seconds": 0.326,
"code": 200,
"response_headers": {
"server": "nginx",
"date": "Mon, 18 Jul 2016 13:23:40 GMT",
"content-type": "application/json",
"transfer-encoding": "chunked",
"connection": "keep-alive",
"vary": "Accept-Encoding",
"access-control-allow-origin": "*",
"access-control-allow-methods": "OPTIONS, GET, POST, DELETE, PUT",
"access-control-allow-headers": "Content-Type, api_key, Authorization"
},
"response_message": "OK",
"times_retried": 0
},
"type": "clod4wi_connections",
"tags": [
"connections"
]
},
"fields": {
"@timestamp": [
1468848220444
]
}
}

What am I missing here?

2

The fields are named dateStart and dateStop but your date filters tries to parse startDate and stopDate.

3

Hi Magnus,

Thanks for spotting that, I did find the issue myself but didn't know how to resolve question in here.

But after I corrected the error I can access nested fields from the date filter no problem!

Thanks for your help Magnus.