Visualization - field not available for selection

[rvarttinen (Robert Varttinen)]

Being a newbie with Kibana I might be asking about something that is already answered. So, my apologies on beforehand. Anyhow, I cannot find any appropriate entries, so any pointer will be appreciated.

My problem is centrered around the fact that I do have a number of fields with interesting data, extracted from log files produced by the Jetty webserver. Everything looks nice until I attempt to create a line chart; the only fields seemingly available for aggregation are those whose type is 'number', i.e. 'geoip.latitude' and geoip.longitude'.

After scrutinizing the data I found that e.g. the 'duration' field (one of those fields I'd like to chart) to be a string, eventhough the LogStash config tells it to be a number. Also tried INT, but to no avail.

Could you please advise on what I am missing. Checked some of the examples and responses I did find and they all seem to have things like 'Repsonse Time' and 'Duration' available as fields for aggregation ...

My aplogies as I am probably inquiring about something obvious, and thanks in advance.

P.S. did Management -> "Index Patterns" and Refresh and all fields are visible with their types, etc. I can see that there is both a 'duration' and a 'duration.keyword' field - both are strings but the '.keyword' one is aggregatable. How can I force it to be of type 'number' or 'int'? Would that solve the problem?

[timroes]

Hi Robert,

you done pretty well debugging this already. As you mentioned the main issue is, that these fields, even though you expect them to be numeric, are actually strings. So Elasticsearch has a wrong mapping for your index. This could perhaps indicate an error in your Logstash config, that it actually doesn't generate the right mapping.
Could you paste your Logstash config?

Cheers,
Tim

[rvarttinen (Robert Varttinen)]

Tim,
Thank you for your swift repsonse. Sure, here is my Logstash config file:

input { 
  file {
    type => "log_file_jetty"
    path => "C:/tmp/logs/*.log"
  }
}
filter {
  if [type] == "log_file_jetty" {
    grok {
        match => { "message" => "%{IPORHOST:remote_addr} - - \[%{HTTPDATE:time_local}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} %{INT:duration}" }
    }
  }  
  date {
    match => [ "time_local", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}
output {
  stdout {
   codec => "rubydebug"
  }
  elasticsearch { hosts => ["localhost:9200"] }
}

[timroes]

Using %{NUMBER:x} will only detect numbers, but not use a numeric data type. You should use %{NUMBER:duration:int} instead to also cast it to an integer.

This is also shortly outlines in the grok docs.

Could you try fixing the config like that and see whether that solves the problem?

Cheers,
Tim

[rvarttinen (Robert Varttinen)]

Tim,
Thanks! Great! It seems to work now - i do get the field 'duration' to be a 'number' and selectable for aggregation. It was pretty obvious, just me not finding my way in the documentation yet ...

//Robert

[timroes]

Glad I could help