How to associate Grok fields with Elasticsearch/Kibana "Available Fields"?

Elastic Stack Elasticsearch
1

I was able to Grok my logs to produce this example line in my logstash.json file. Here's my Grok filter

 grok {
    match => {
      "message" => "%{TIMESTAMP_ISO8601:logdate}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}:%{DATA:thread}%{SPACE}:%{SPACE}%{NUMBER:thread_pool}%{SPACE}\[(?<classname>[^\]]+)\]%{SPACE}%{GREEDYDATA:msgbody}"
    }
}

{"@timestamp":"2016-04-22T20:43:25.172Z","message":"2016-04-22 16:43:25,172 ERROR :WebContainer : 2 [com.mycomapany.SomeClass] My log message.","@version":"1","path":"/home/some_path/trm-error.log","host":"myHost","logdate":"2016-04-22 16:43:25,172","loglevel":"ERROR","thread":"WebContainer","thread_pool":"2","classname":"com.mycompany.SomeClass","msgbody":"My log message."}

So how do I feed my json file into Elasticsearch so I'm able to see the following fields (from my Grok filter) in the "Available Fields" panel in Kibana? Make sense?

  • logdate
  • loglevel
  • thread
  • thread_pool
  • classname
  • msgbody
2

Just add an elasticsearch output in your Logstash configuration to send the events to your ES instance.

3

Yes, I have this already. I DO see the logs in Kibana, I jost don't see the fields in "Available Fields" panel.

output {
  file {
    path => "/home/some_path/trm-logstash.json"
  }
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}
4

Kibana are flagging those as "missing fields?" What does that mean? IOW, if I uncheck the "Hide Missing Fields" checkbox, I do see the fields, but are greyed out.

5

I DO see the logs in Kibana, I jost don't see the fields in "Available Fields" panel.

Have you reloaded the index fields list?

6

When I go to Settings -> indices tab -> select logstash-*, I do see the fields I want there. I went ahead and did a reload anyway as you suggested. Now when i go to the Discover tab, I see the logstash-* index, and in it's Available Fields, the fields that I want are designated as "Missing Fields," that is, I only see them if I uncheck the "Hide Missing Fields" check box.