Hi all,
I have an ES 5.1 cluster running on AWS. Initially I only had CloudTrail data streaming to the cluster, and created several visualisations and dashboards, all of which work fine. I then started streaming additional data (vpc flow logs) to the same cluster, and these documents are being sent to the same index as the CloudTrail data.
Once the new data had been streaming in for a few hours, I went in to Kibana/Management/Index Patterns and noticed the number of fields wasn't looking right. I hit the "Refresh field list" button and as expected, the number of field increased as expected. Unfortunately, when I head to the "Discover" section, none of the new fields appear, they are only appear when I clear the "Hide Missing Fields" option. Problem is, I KNOW there is data in there, a CURL query proves it:
curl -X GET 'http://esendpoint/cwl-2017.03.26/FlowLog/_search?pretty=true&q=_id:33240388600899054679056960135202949661543123298683912213'
{
"took" : 22,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [
{
"_index" : "cwl-2017.03.26",
"_type" : "testFlowLog",
"_id" : "33240388600899054679056960135202949661543123298683912213",
"_score" : 1.0,
"_source" : {
"srcaddr" : "172.31.45.185",
"dstport" : 2791,
"start" : 1490550576,
"dstaddr" : "172.31.41.151",
"version" : 2,
"packets" : 5,
"protocol" : 6,
"account_id" : 810777683624,
"interface_id" : "eni-43db651a",
"log_status" : "OK",
"bytes" : 582,
"srcport" : 80,
"action" : "ACCEPT",
"end" : 1490550694,
"@id" : "33240388600899054679056960135202949661543123298683912213",
"@timestamp" : "2017-03-26T17:49:36.000Z",
"@message" : "2 810777683624 eni-43db651a 172.31.45.185 172.31.41.151 80 2791 6 5 582 1490550576 1490550694 ACCEPT OK",
"@owner" : "810777683624",
"@log_group" : "testFlowLog",
"@log_stream" : "eni-43db651a-all"
}
}
]
}
}
Kibana tells me all these fields (packets, protocol, version etc.) are empty! Why on earth are the fields in the new documents not available for me to use?
Thanks in advance for your help!