Kibana Data Table filtered data missed

Elastic Stack Kibana
1

Hi All, I have some raw message have been parsed with below fields:

EventID:
EventTime:
Hostname:
Message:

Kibana index name begins with logstash-xxx.

The question is now I am using Kibana 4.5.0 and visualize my data with Metric and Data Table. I can easily filtered my data with EventID say 676 and shows total counts in Metric is correct. But the question is that it has a wrong count show in Data Table. An example is just like below:

Enable Filter: EventID:676

Metric View
1141 <-- This is correct number of entries.

Data Table
EveintID.raw Message.raw Count
676 Some message 323 <-- (only 323 showed but total count is 1141)

I have tried not to use Message.raw field and it shows correct count 1141. I tried to review the log in Discovery view, it show that was parsed successfully in message field. Seems parsing has no problem but the data view exist missing counts in Data Table.

Thank you in advance for your help with this!

Best,
Deep

2

Based on your example it looks like the data table is grouping events by their raw Message, meaning that the count 323 is simply the number of events with EventID 676 and the same message. What if you add a filter for the message.raw too, does the metric view match then?

Also, can you include a screenshot by chance?

3

Thanks for some thoughts!

I found it is the Size value which made those entries didn't show completely. I have set it to 1000 and sum it up then the total count is now correct.

Value set to 1000

Kibana_pic1.PNG267×667 13.5 KB

But now there is another question. As below screen shows, I have many event ID. If I enable the filter for EventID 675, total logs is 593. The log and message showed correctly in the Data Table. While I enable the filter Event ID 673, total count is 1171 but no message can show out. I go the Discover tab and search the log with EventId 673, the message field contains the message in message filed, seems has been parsed with no problem. Appreciate you help.

No filter applied:

Kibana_pic0.PNG1251×738 58.3 KB

Filter: EventID: 675

Kibana_pic2.PNG1263×763 46.9 KB

Filter: EventID: 673

Kibana_pic3.PNG1256×737 17 KB

Log in Discover tab:

Kibana_pic4.PNG1035×442 21 KB

Thanks,
Deep

4

Hi, when you are filtering on EventID = 673, could you click the little ^ at the bottom of the MSG visualization panel? This will bring up a spy panel which has useful debugging information. In this spy panel, please click the Request button. This will show the request being sent to Elasticsearch. Please paste this request over here so we can debug further. Thanks!

5

Hi , the request being sent to ES like below. Thanks!

{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "",
"analyze_wildcard": true
}
},
"filter": {
"bool": {
"must": [
{
"$state": {
"store": "appState"
},
"query": {
"match": {
"EventID.raw": {
"query": "673",
"type": "phrase"
}
}
}
},
{
"query": {
"query_string": {
"analyze_wildcard": true,
"query": ""
}
}
},
{
"range": {
"@timestamp": {
"gte": 1461212310614,
"lte": 1461233028309,
"format": "epoch_millis"
}
}
}
],
"must_not": []
}
}
}
},
"size": 0,
"aggs": {
"2": {
"terms": {
"field": "EventID.raw",
"size": 1000,
"order": {
"_count": "desc"
}
},
"aggs": {
"3": {
"terms": {
"field": "Message.raw",
"size": 1000,
"order": {
"_count": "desc"
}
}
}
}
}
}
}

6

Is it possible that the events with eventID 673 have a Message but no Message.raw?

7

As it is using dynamic mapping , and the default behaviors of ES that query from Message field is not allowed. And It return successful eventID 675 with the same request from Message.raw. Is there any misconfiguration or praising issue I made or it is the fault of ES?