Kibana says field contains no data, but CURL query shows it does

[gjws]

Hi all,

I have an ES 5.1 cluster running on AWS. Initially I only had CloudTrail data streaming to the cluster, and created several visualisations and dashboards, all of which work fine. I then started streaming additional data (vpc flow logs) to the same cluster, and these documents are being sent to the same index as the CloudTrail data.

Once the new data had been streaming in for a few hours, I went in to Kibana/Management/Index Patterns and noticed the number of fields wasn't looking right. I hit the "Refresh field list" button and as expected, the number of field increased as expected. Unfortunately, when I head to the "Discover" section, none of the new fields appear, they are only appear when I clear the "Hide Missing Fields" option. Problem is, I KNOW there is data in there, a CURL query proves it:

curl -X GET 'http://esendpoint/cwl-2017.03.26/FlowLog/_search?pretty=true&q=_id:33240388600899054679056960135202949661543123298683912213'
{
"took" : 22,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [
{
"_index" : "cwl-2017.03.26",
"_type" : "testFlowLog",
"_id" : "33240388600899054679056960135202949661543123298683912213",
"_score" : 1.0,
"_source" : {
"srcaddr" : "172.31.45.185",
"dstport" : 2791,
"start" : 1490550576,
"dstaddr" : "172.31.41.151",
"version" : 2,
"packets" : 5,
"protocol" : 6,
"account_id" : 810777683624,
"interface_id" : "eni-43db651a",
"log_status" : "OK",
"bytes" : 582,
"srcport" : 80,
"action" : "ACCEPT",
"end" : 1490550694,
"@id" : "33240388600899054679056960135202949661543123298683912213",
"@timestamp" : "2017-03-26T17:49:36.000Z",
"@message" : "2 810777683624 eni-43db651a 172.31.45.185 172.31.41.151 80 2791 6 5 582 1490550576 1490550694 ACCEPT OK",
"@owner" : "810777683624",
"@log_group" : "testFlowLog",
"@log_stream" : "eni-43db651a-all"
}
}
]
}
}

Kibana tells me all these fields (packets, protocol, version etc.) are empty! Why on earth are the fields in the new documents not available for me to use?

Thanks in advance for your help!

[weltenwort (Felix Stürmer)]

Hi @gjws,

when the "Hide Missing Fields" check box is active in "Discover" the field list only shows the fields that are present in the current result set's _sources. Disabling the checkbox shows all fields that Kibana knows for that index pattern.

Does the behavior you described persist even though the query and time range include documents containing the fields you expect to see?

[gjws]

Hi eltenwort,

Yep, I have zero filters applied, but still the fields show as empty

I'm going straight to the "Discover" tab, select the offending index, and that's it. There is a common field, @log_group, that is included in both the old and the new record types. By now, there are more of the new record type in the index than the old type, however when I expand that field , the quick count still shows 100% of records belonging to the old log group.

This just doesn't make any sense, I mean, Kibana is aware of the new fields, but simply refuses to see any of the data contained in them!

[weltenwort (Felix Stürmer)]

So the "Discover" table actually contains documents that have these new fields while the sidebar on the left does not show them? Is it possible that storing the _source has been disabled in the mapping for these fields then?

[gjws]

Nope, the discover tab doesn't have them either!

I can find them using the dev console, other that that, Kibana doesn't know they exist. This is really becoming a problem, I simply don't know what else to try

[weltenwort (Felix Stürmer)]

It sounds there is a misunderstanding about the field list on the left side of the discover page here. Its behavior should be as follows:

• If "Hide Missing Fields" is checked, it only shows fields that occur in at least one of the documents contained in the current query result.
• If "Hide Missing Fields" is not checked, it shows all fields shown in the "Management / Index Patterns" section for the respective Index Pattern.

[gjws]

Problem solved, I'm an idiot!

The new documents were indeed in Kibana, but they did not contain the same field name for their timestamp, so Kibana could not apply time ranges against them. I recreated the index, using the timestamp field common to both log types and voilà, there they were!

Thanks for sticking with me weltenwort, it's been an interesting ride

[weltenwort (Felix Stürmer)]

Good to hear you resolved your problem. I will add that case to my error diagnosis checklist, so I profited as well.