Best practice for Log-Analyzing


(Illmatic) #1

I need to Analyze the Logs(Exceptions) from a Rest-Application. I have a Log-File with the logged Exceptions and was wondering, what would be best practice to Analyze this Logs and wich Components from the ELK-Stack I should use for that.

Thanks for the Help allready :slight_smile:


(Jake Landis) #2

The first step is to get the data into Elasticsearch.

Take a look at Filebeat [1] (which installs on the same host as your Rest-Application). Pay particular attention to the multi-line support [2] since most exceptions are multi-line.

I would suggest to start with sending data directly to Elasticsearch [3]. If you need any pre-processing of the data prior to indexing, you have a few options [4][5][6].

If you have your own instance of Elasticsearch and Kibana already running, great! If not our cloud offering [7] is really great for getting starting (and production too !). Docker is also great for getting started (and production) [8], and using docker compose with Elasticsearch and Kibana [9] is great for a quick test environment.

Once you have the data in Elasticsearch and Kibana running to view the data, there is a lot available information on this topic via the interwebs. It's a wide topic so specific questions would be needed to properly advise your for your usecase.

[1] https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html
[2] https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
[3] https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html
[4] https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html
[5] https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html
[6] https://www.elastic.co/guide/en/logstash/current/index.html
[7] https://www.elastic.co/cloud
[8] https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
[9] https://www.elastic.co/guide/en/kibana/current/docker.html


(Illmatic) #3

Thank you for this detailed answer! You helped me alot!