LOGSTASH / Kibana4 - Question

Elastic Stack Logstash
1

I have a question, I am trying to graph application response times on NGINX logs. I am trying to graph the response time base on a field called request_duration.

My logs look like this:

216.165.152.226 - - [09/Sep/2015:06:54:26 +0000] "GET /cgi-bin/dota/opac-main.pl?logout.x=1 HTTP/1.1" 200 7423 "https://www.thinktank.info/cgi-bin/dota/acc-main.pl?logout.x=1" - www.thinktank.info 0.186 10.237.187.132:5013

My filter is this:
%{IPORHOST:clientip} %{NGUSER:indent} %{NGUSER:agent} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:answer} %{NUMBER:byte} "%{URI:referrer}" %{NGUSER:indent1} %{JAVACLASS:web1} %{BASE10NUMBER:request_duration} %{IPORHOST:placksrv}:%{INT:plk_version}

The field that I want to graph is the %{BASE10NUMBER:request_duration} but if you look in the kinaba index show the field as a string.

request_duration string true true 0

Any Ideas, I am new to the ELK stack and it looks good for our shop.

Thanks in advance.

BTW also tried this on the filter..

 mutate {
        convert => [ "request_duration", "float"]
    }

But no change it still looks like a string in kibana.

2

You need to specify that in the template that exists, ES does the best it can but sometimes that's not 100%.

Take a look at curl HOST:9200/_template/logstash.

3

Mark,

Thank you very much for your quick response, however I am not sure I understand what you are asking me to do.

curl localhost:9200/_template/logstash
{"logstash":{"order":0,"template":"logstash-","settings":{"index.refresh_interval":"5s"},"mappings":{"default":{"dynamic_templates":[{"string_fields":{"mapping":{"index":"analyzed","omit_norms":true,"type":"string","fields":{"raw":{"index":"not_analyzed","ignore_above":256,"type":"string"}}},"match_mapping_type":"string","match":""}}],"properties":{"geoip":{"dynamic":true,"path":"full","properties":{"location":{"type":"geo_point"}},"type":"object"},"@version":{"index":"not_analyzed","type":"string"}},"_all"

Do I need to add the field so that it takes effect?

Thanks,

Luis

4

What does curl HOST:9200/logstash/_mapping?pretty show for that request_duration field?

5

You can get the grok filter to convert it from a string to float by changing %{BASE10NUMBER:request_duration} to %{BASE10NUMBER:request_duration:float} or by using the mutate filter to convert it as in your example. As you have already indexed the field as a string, you should, as Mark points out, be able to see the field represented as a string in the mappings.

As you can not change existing mappings, you will need to either delete your index and reindex your data with the corrected configuration.

6

Mark,

Here are the results:

     "request_duration" : {
        "type" : "string",
        "norms" : {
          "enabled" : false
        },
        "fields" : {
          "raw" : {
            "type" : "string",
            "index" : "not_analyzed",
            "ignore_above" : 256
          }
        }
      },
7

Try to do that query on the .raw value then, that should work.