Given Apache logs in ES, how do I get this data out into Elasticsearch

Elastic Stack Kibana
#1

I've got apache logs in Elasticsearch. I can see the data in Kibana.

How do I visualize the following?

  • Total Page Views
  • Total Hits
  • Unique Page Views
  • Average Visitor Stay Length

I'm just a system administrator. After setting up Logstash, Elasticsearch, getting data moving into the es cluster, this particular problem seems so far out of my wheelhouse it's not funny. I almost literally have no idea where to begin.

Any help is appreciated.

#2

You can only do the first two, unique page views is not possible because there is no unique function and Average visitor stay is also not possible because there is no way to group by visitor and deduct the stay time in an easy way.

Seems like what you are looking for can be done in Google Analytics or any other web analytics tool pretty easily.

You can do other cool stuff with Kibana like we wrote here - if you want we can push that dashboard to you.

-- Asaf.

#3

I would be very interested in that dashboard.

#4

In Kibana 4, unique visitors and page views can be achieved with "Unique Count" (or Cardinality) aggregation -- see attached below.

Average stay length is more complicated. I think you'd have to pre-calculate those values outside of Elasticsearch and store them along with your documents. You could take a look at the elapsed filter in Logstash - I haven't tried it myself, but it seems to do what you want.

Screen Shot 2015-07-22 at 5.50.49 AM.png1964×1052 176 KB

#5

@bdunbar - send me an email to asaf@logz.io and I will share the dashboard with you.

@tbragin - Tanya - the Unique Count is not really Unique count, for us in BI accuracy is critical so we couldn't use that but if you're looking for estimates of unique visitors then it should be good enough I think.

I have never used the elapsed filter in Logstash like Tanya indicated and it can be a good solution if your traffic is in low enough volume that can be handled by a single logstash - otherwise this wont work.

-- Asaf.

#6

This will likely help get you started down this path with Elasticsearch and a little help from a customer script or two.

Entity Centric Indexing is the basis for achieving this with Elasticsearch and Mark Harwood, one of our Engineers, has given several talks on the subject. Here are some pointers:

Slides: https://www.elastic.co/elasticon/2015/sf/building-entity-centric-indexes
Recent video: https://www.elastic.co/videos/entity-centric-indexing-mark-harwood?q=entity
Video from Meetup: https://www.elastic.co/videos/entity-centric-indexing-london-meetup-sep-2014

#7