LOGSTASH / Kibana4 - Question

(Luis Lacayo) #1

I have a question, I am trying to graph application response times on NGINX logs. I am trying to graph the response time base on a field called request_duration.

My logs look like this: - - [09/Sep/2015:06:54:26 +0000] "GET /cgi-bin/dota/opac-main.pl?logout.x=1 HTTP/1.1" 200 7423 "https://www.thinktank.info/cgi-bin/dota/acc-main.pl?logout.x=1" - www.thinktank.info 0.186

My filter is this:
%{IPORHOST:clientip} %{NGUSER:indent} %{NGUSER:agent} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:answer} %{NUMBER:byte} "%{URI:referrer}" %{NGUSER:indent1} %{JAVACLASS:web1} %{BASE10NUMBER:request_duration} %{IPORHOST:placksrv}:%{INT:plk_version}

The field that I want to graph is the %{BASE10NUMBER:request_duration} but if you look in the kinaba index show the field as a string.

request_duration string true true 0

Any Ideas, I am new to the ELK stack and it looks good for our shop.

Thanks in advance.

BTW also tried this on the filter..

 mutate {
        convert => [ "request_duration", "float"]

But no change it still looks like a string in kibana.

(Mark Walkom) #2

You need to specify that in the template that exists, ES does the best it can but sometimes that's not 100%.

Take a look at curl HOST:9200/_template/logstash.

(Luis Lacayo) #3


Thank you very much for your quick response, however I am not sure I understand what you are asking me to do.

curl localhost:9200/_template/logstash

Do I need to add the field so that it takes effect?



(Mark Walkom) #4

What does curl HOST:9200/logstash/_mapping?pretty show for that request_duration field?

(Christian Dahlqvist) #5

You can get the grok filter to convert it from a string to float by changing %{BASE10NUMBER:request_duration} to %{BASE10NUMBER:request_duration:float} or by using the mutate filter to convert it as in your example. As you have already indexed the field as a string, you should, as Mark points out, be able to see the field represented as a string in the mappings.

As you can not change existing mappings, you will need to either delete your index and reindex your data with the corrected configuration.

(Luis Lacayo) #6


Here are the results:

     "request_duration" : {
        "type" : "string",
        "norms" : {
          "enabled" : false
        "fields" : {
          "raw" : {
            "type" : "string",
            "index" : "not_analyzed",
            "ignore_above" : 256

(Mark Walkom) #7

Try to do that query on the .raw value then, that should work.

(system) #8