Kibana showing different Packetbeat data

Hi Guys,

I have two version of Elasticsearch x Kibana (Version 7.6 and version 7.12)

When i am analysing data in Kibana, using Discover, the information is showed in different ways for each ES/Kibana version.

Example:

  • ES and Kibana version 7.6 (http.request.body.content and http.response.body.content are in the same doc)

  • ES and Kibana version 7.12 these two fields are showing in different doc.

Why does it happen ? This test was done collecting data from tha same packetbeat server.

We did make some tweaks to how data is shown in Discover in 7.12, but not like what you're describing. Can you provide a screenshot with more context comparing before and after? What happens when you look at the JSON view before and after?

Wylie,

Follow bellow Json/printscreen for each ES+Kibana Version

- Version 7.6:

{
"_index": "packetbeat-ecms-7.6.1-2021-03-30",
"_type": "_doc",
"_id": "qX5whHgBCQItkYpAQjQn",
"_version": 1,
"_score": 0,
"_source": {
"@timestamp": "2021-03-30T17:19:10.807Z",
"url": {
"port": 10115,
"path": "/Air",
"full": "http://10.110.181.143:10115/Air",
"scheme": "http",
"domain": "10.110.181.143"
},
"server": {
"port": 10115,
"bytes": 9322,
"ip": "10.110.181.143"
},
"event": {
"dataset": "http",
"duration": 8801000,
"start": "2021-03-30T17:19:10.807Z",
"end": "2021-03-30T17:19:10.816Z",
"kind": "event",
"category": "network_traffic"
},
"query": "POST /Air",
"client": {
"ip": "10.110.181.9",
"port": 21299,
"bytes": 1212
},
"type": "http",
"destination": {
"port": 10115,
"bytes": 9322,
"ip": "10.110.181.143"
},
"ecs": {
"version": "1.4.0"
},
"host": {
"name": "ecmsdb2sm2"
},
"http": {
"request": {
"bytes": 1212,
"headers": {
"content-type": "text/xml",
"content-length": 971
},
"method": "post",
"body": {
"content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>GetAccountDetailsoriginHostNameCMAoriginTimeStamp<dateTime.iso8601>20210330T14:19:10-0300</dateTime.iso8601>originNodeTypeADMrequestedInformationFlagsrequestMasterAccountBalanceFlag1originTransactionID8437521subscriberNumber5511947069252subscriberNumberNAI1negotiatedCapabilities",
"bytes": 971
}
},
"response": {
"status_phrase": "ok",
"status_code": 200,
"body": {
"bytes": 9049,
"content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>accountFlagsactivationStatusFlag0ussdEndOfCallNotificationID255"
},
"bytes": 9322,
"headers": {
"content-length": 9049,
"content-type": "text/xml"
}
},
"version": "1.1"
},
"source": {
"ip": "10.110.181.9",
"port": 21299,
"bytes": 1212
},
"network": {
"protocol": "http",
"direction": "outbound",
"community_id": "1:wsZmwsJttSkfUR5QILDq8enTj+0=",
"bytes": 10534,
"type": "ipv4",
"transport": "tcp"
},
"status": "OK",
"method": "post",
"user_agent": {
"original": "CMA/5.0/2.0"
},
"agent": {
"ephemeral_id": "59af44e2-bc19-4312-9339-1e703b6a2204",
"hostname": "ecmsdb2sm2",
"id": "b753bd30-4d64-44b3-b942-488bc985186d",
"version": "7.6.1",
"type": "packetbeat"
}
},
"fields": {
"event.end": [
"2021-03-30T17:19:10.816Z"
],
"@timestamp": [
"2021-03-30T17:19:10.807Z"
],
"event.start": [
"2021-03-30T17:19:10.807Z"
]
},
"highlight": {
"network.protocol": [
"@kibana-highlighted-field@http@/kibana-highlighted-field@"
]
}
}

- Version 7.12

{
"_index": "packetbeat-ecms-7.6.1-2021-03-30",
"_type": "_doc",
"_id": "6ypJhHgBauqIsqsBiy_6",
"_version": 1,
"_score": null,
"fields": {
"event.category": [
"network_traffic"
],
"server.ip": [
"10.160.99.137"
],
"user_agent.original.text": [
"CMA/5.0/2.0"
],
"type": [
"http"
],
"http.request.method": [
"post"
],
"http.response.bytes": [
9322
],
"source.ip": [
"10.160.99.7"
],
"http.request.headers.content-length": [
971
],
"network.community_id": [
"1:K+ibSYSTdJNqb/nICgZBbY6SW6I="
],
"host.name": [
"ecms1df2"
],
"http.response.status_code": [
200
],
"http.version": [
"1.1"
],
"event.kind": [
"event"
],
"user_agent.original": [
"CMA/5.0/2.0"
],
"http.request.body.content.text": [
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>GetAccountDetailsoriginHostNameCMAoriginTimeStamp<dateTime.iso8601>20210330T16:46:39-0300</dateTime.iso8601>originNodeTypeADMrequestedInformationFlagsrequestMasterAccountBalanceFlag1originTransactionID5955131subscriberNumber5562992714402subscriberNumberNAI1negotiatedCapabilities"
],
"method": [
"post"
],
"query": [
"POST /Air"
],
"client.ip": [
"10.160.99.7"
],
"agent.hostname": [
"ecms1df2"
],
"http.response.status_phrase": [
"ok"
],
"url.path": [
"/Air"
],
"source.port": [
47887
],
"agent.id": [
"94a8e0bc-b03e-4807-ba13-b8b81fd70083"
],
"bytes_out": [
9322
],
"client.port": [
47887
],
"ecs.version": [
"1.4.0"
],
"agent.version": [
"7.6.1"
],
"destination.bytes": [
9322
],
"event.start": [
"2021-03-30T19:46:39.999Z"
],
"status": [
"OK"
],
"server.bytes": [
9322
],
"http.request.body.bytes": [
971
],
"destination.port": [
10115
],
"bytes_in": [
1211
],
"event.end": [
"2021-03-30T19:46:40.039Z"
],
"url.scheme": [
"http"
],
"http.request.bytes": [
1211
],
"http.response.body.content.text": [
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>accountFlagsactivationStatusFlag0negativeBarringStatusFlag0serviceFeePeriodExpiryFlag>

ps.: Due to huge size of http.response.body.content field, I had to cut some parts of it to give this reply.

Right, this is actually a more accurate view than before. You seem to be using multi-mapping to map both http.response.body.content and http.response.body.content.text on the same field, and before 7.12 this distinction was impossible to see using Kibana. So I have two suggestions:

  1. You can switch to the .text field
  2. You can disable the new data fetching in the Kibana advanced settings, but I don't think you need to.

I don't have a good explanation for why your keyword field is not indexed properly, but the values you're seeing are exactly what Elasticsearch knows.