Available fields not showing in kibana [Issue resolved]

Elastic Stack Kibana
1

Hi,

I'm trying to parse Apache logs which is in this format:

xxx.xxx.xxx.xxx - - [06/Oct/2015:23:58:37 +0530] "GET /search/newver_smartsearchjs_ql.php HTTP/1.1" 200 47532 29330 "http://subdomain.domain.com/search/searchres_ql.php?randid=a37093s&gaact=search&gasrc=WO" "Mozilla/5.0 (iPad; CPU OS 9_0_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13A452 Safari/601.1"

I'm using the following grok filter:

filter {
  if [type] == "apache-access" {
    grok {
      match => [ "message", "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:response_time} %{QS:referrer} %{QS:agent}" ]
    }
date{
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] 
    locale => "en" 
  }
geoip {
      source => "clientip"
      target => "geoip"
      database => "/etc/logstash/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float"]
    }
  }
}

The logstash parse the logs well and in correct format. It identifies each field properly and it parses (I saw this in /var/log/logstash/logstash.stdout

But I'm not getting the Available Fields in the indices properly. FYI:

Its not showing the proper Apache fields which I have given in logstash filter. I have also reloaded the fields in the settings-->indices-->

Its showing fields in settings-->indices as shown below:

Screenshot from 2015-10-08 23:37:26.png1116×615 48.6 KB

Please help why kibana is not displaying the proper available fields.

2

UPDATE at end of this post

Ok. I found the issue why this occurs. Below is my observation:

The issue occurs when any field appears to be blank with no data . Check the below screenshot, where certain fields have no data (like IP address, referrer, etc)

Selection_005.png1104×415 41.2 KB

Check the JSON format of that search:

Below is the apache log with all the fields filled in properly:

Selection_007.png1094×621 71.7 KB

In JSON format:

Selection_008.png1099×618 108 KB

So , its seems its the logstash issue rather than kibana. If so, kindly move to this logstash section or you can explain your answer here.

Thanks.

UPDATE: In logstash filter grok, I made the IPaddress field optional by enclosing the %{IPORHOST:clientip} with ()? . Final format is (%{IPORHOST:clientip})? . Issue resolved.

3 Likes
3

Thanks for posting the resolution!