LOGSTASH / Kibana4 - Question

I have a question, I am trying to graph application response times on NGINX logs. I am trying to graph the response time base on a field called request_duration.

My logs look like this: - - [09/Sep/2015:06:54:26 +0000] "GET /cgi-bin/dota/opac-main.pl?logout.x=1 HTTP/1.1" 200 7423 "https://www.thinktank.info/cgi-bin/dota/acc-main.pl?logout.x=1" - www.thinktank.info 0.186

My filter is this:
%{IPORHOST:clientip} %{NGUSER:indent} %{NGUSER:agent} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:answer} %{NUMBER:byte} "%{URI:referrer}" %{NGUSER:indent1} %{JAVACLASS:web1} %{BASE10NUMBER:request_duration} %{IPORHOST:placksrv}:%{INT:plk_version}

The field that I want to graph is the %{BASE10NUMBER:request_duration} but if you look in the kinaba index show the field as a string.

request_duration string true true 0

Any Ideas, I am new to the ELK stack and it looks good for our shop.

Thanks in advance.

BTW also tried this on the filter..

 mutate {
        convert => [ "request_duration", "float"]

But no change it still looks like a string in kibana.

You need to specify that in the template that exists, ES does the best it can but sometimes that's not 100%.

Take a look at curl HOST:9200/_template/logstash.


Thank you very much for your quick response, however I am not sure I understand what you are asking me to do.

curl localhost:9200/_template/logstash

Do I need to add the field so that it takes effect?



What does curl HOST:9200/logstash/_mapping?pretty show for that request_duration field?

You can get the grok filter to convert it from a string to float by changing %{BASE10NUMBER:request_duration} to %{BASE10NUMBER:request_duration:float} or by using the mutate filter to convert it as in your example. As you have already indexed the field as a string, you should, as Mark points out, be able to see the field represented as a string in the mappings.

As you can not change existing mappings, you will need to either delete your index and reindex your data with the corrected configuration.


Here are the results:

     "request_duration" : {
        "type" : "string",
        "norms" : {
          "enabled" : false
        "fields" : {
          "raw" : {
            "type" : "string",
            "index" : "not_analyzed",
            "ignore_above" : 256

Try to do that query on the .raw value then, that should work.