Web Server Recipies

Elastic Stack Kibana
#1

I've got logstash shoving my log files into elasticsearch, got a stable cluster, status is green. Kibana4 is up and running. Huzzah!

I'm not quiet sure how to turn this data into useful graphs and pictures. Oh I get the process of X and Y, charts and graphs but ... are there any recipes that say 'For total page views do this' and 'for average visitor stay do that'?

#2

The Next step is to make sure your data is parsed

So in your logstash configuration you need to use the filter { } section and to parse the data the filter grok would be what you want.

https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

Here is a good page to help you develop you regex's against
https://grokdebug.herokuapp.com/

Oh and for response time or other numeric fields you will want to use a GROKstatement like %{NUMBER:responsetime:int} this way it gets in to elasticsearch as a int field and not string.

#3

Okay: I'm using filters [1]. I get, in a basic way, how to create visualizations. What I'm struggling with, now, is how to turn this list into visualizations.

  • Total Visits Total
  • Unique Visits
  • Total Page Views
  • Total Hits
  • Unique Page Views
  • Average Visitor Stay Length

Actually I have the Totals - those turn out to be reasonable. I don't have a ready way to calculate Page Views and Visitor Stay Length. Any help is appreciated.

[1] Like so

filter {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    geoip {
      source => "clientip"
      target => "geoip"
      database => "/etc/logstash/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float"]
    }

    if [clientip] in ["10.1.88.11", "10.1.88.12", "10.1.88.13", "10.1.88.14", "10.1.88.15", "10.1.88.16", "10.1.42.117", "10.1.42.118", "10.1.42.119", "10.1.88.21", "10.1.88.22", "10.1.88.23", "10.1.88.24", "10.1.88.25", "10.1.88.26", "10.1.42.127", "10.1.42.128", "10.1.42.129"]  {
       drop {}
    }
}
#4

Well you have to work with Kibana to visualize this information. If you just wan the data to run queries you want to look at all the aggregations options

Unfortunately your followup question is very vague and difficult to give you exact answers

I would start by installing kibana, and using it's functionality then post any questions about that as you need help

#5

Brian,

There are things that can easily be done like the overall page views, geo location etc. and there are things that require a lot of work like unique visitors, visitor stay length etc. These will require scripting in Elasticsearch and very complicated queries.

We have generated some very interesting dashboads around web analytics and technical SEO and you can read our post here - (http://logz.io/blog/log-analysis-technical-seo/)

Would be happy to help if you need assistance.

-- Asaf.

#6