I have been playing with ELK and setup Elasticstack to collect my personal webserver access logs. There will be more to come but I though this would be a good starting point.
A run down of where I am at is below:
- I have installed Filebeat to monitor access logs and output to Logstash on another Ubuntu server – Working .
- I have installed Logstash to input, filter and enrich the weblogs (grok apache and geoip filter I think), and output to Elasticsearch on another Ubuntu server – Working .
- I have installed Elasticsearch to index the weblogs. This is where it get strange. It appears it is working as there are no errors (the connection established and pipeline is working). When I do a search for the indices I can see the index is created and the “docs count” increases as people visit the site. However, when I search to “match all” I can only see a handful of events. It is as if it worked to ingested logs at some point but stopped or it is still working and the docs are there but not “extracted” – I have no idea.
- I have installed Kibana on the same server that is running Elasticsearch. I can access the web interface but there is no data to discover. The strange thing is I can import one of the sample datasets provided with Kibana and search/visualise that data and that data is now available in an index in Elasticsearch so I appears communication is working. The only thing I can see that is different is the sample data is defined as “green” and the other data I have been sending in defined as “yellow”. – Something is wrong.