Indexing application log files to elasticsearch

Elastic Stack Logstash
1

Hi,

I am investigating what to use to index application log files into elasticsearch.

===Use Case===

I have a miroservice developed using spring boot. Logging framework I use is logback.

Options to log application output are:

  1. File -- Basic file appender
  2. TCP Socket -- LogstashTcpSocketAppender

In the first use case, output is plain text (not json). My options here I assume are to use lightweight shipper such as FileBeats (handle multilines) and then output to logstash.

OR
Output in json format over tcp and have logstash listen on that port

input { tcp { codec => "json" port => 5000 } }

Am i barking up the wrong tree or is that my two options!?

Thanks,
Shane.

2

I'd dump the logs to a local file in JSON format, then use Filebeat to ship that. I don't like shipping logs directly over the network since network or server outages could lead to either a blocked application or dropped logs.

3

Thanks for the quick reply mate.

Is it best practice to log output in json format than plain text?

Is it recommended then to

  1. Ship logs directly from filebeats to elasticsearch
  2. Ship logs from filebeats to logstash which outputs to elasticsearch

I read this interesting article recently and trying to understand the best solution!

Thanks,
Shane.

4

Is it best practice to log output in json format than plain text?

If you can control the logging format I think it's preferable since there's more or less no configuration to do (no multiline worries for example).

Is it recommended then to

  1. Ship logs directly from filebeats to elasticsearch
  2. Ship logs from filebeats to logstash which outputs to elasticsearch

Since Filebeat has basically zero features for processing or parsing events I don't think the first option is very useful.

5

+1 on the json format then! :stuck_out_tongue:

This is the part I am not sure about.
The framework logstash encoder has an encoder called LoggingEventCompositeJsonEncoder that can provide greater flexibilty in the json format.

So I am thinking if I defined patterns at the logging level, do I really need to ship to logstash?

I understand your point about directly over the network. My colleague has mentioned that beats and/or logstash will have a retry mechanism in place for network failures. Is that true?

i see the 12 factor site recommends stdout! The Twelve-Factor App

Thanks,
Shane.

6

So I am thinking if I defined patterns at the logging level, do I really need to ship to logstash?

That depends on what kind of filtering you might want to do in Logstash, and if Elasticsearch is the only output you're interested in. There is no right or wrong here. It depends on your needs and preferences.

I understand your point about directly over the network. My colleague has mentioned that beats and/or logstash will have a retry mechanism in place for network failures. Is that true?

Yes.

1 Like
7

Thank you.