# Indexing application log files to elasticsearch
**URL:** https://discuss.elastic.co/t/indexing-application-log-files-to-elasticsearch/55377
**Category:** Logstash
**Created:** [July 13, 2016, 7:01am UTC](https://discuss.elastic.co/t/indexing-application-log-files-to-elasticsearch/55377 "2016-07-13T07:01:45Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@shane\_lee](https://discuss.elastic.co/u/shane_lee)
#### Post date: [July 13, 2016, 7:01am UTC](https://discuss.elastic.co/t/indexing-application-log-files-to-elasticsearch/55377/1 "2016-07-13T07:01:45Z")
Hi,
I am investigating what to use to index application log files into elasticsearch.
===Use Case===
I have a miroservice developed using spring boot. Logging framework I use is logback.
Options to log application output are:
1. File -- Basic file appender
2. TCP Socket -- LogstashTcpSocketAppender
In the first use case, output is plain text (not json). My options here I assume are to use lightweight shipper such as FileBeats (handle multilines) and then output to logstash.
OR
Output in json format over tcp and have logstash listen on that port
`input { tcp { codec => "json" port => 5000 } }`
Am i barking up the wrong tree or is that my two options!?
Thanks,
Shane.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 13, 2016, 7:20am UTC](https://discuss.elastic.co/t/indexing-application-log-files-to-elasticsearch/55377/2 "2016-07-13T07:20:18Z")
I'd dump the logs to a local file in JSON format, then use Filebeat to ship that. I don't like shipping logs directly over the network since network or server outages could lead to either a blocked application or dropped logs.
---
### Author:  [@shane\_lee](https://discuss.elastic.co/u/shane_lee)
#### Post date: [July 13, 2016, 7:28am UTC](https://discuss.elastic.co/t/indexing-application-log-files-to-elasticsearch/55377/3 "2016-07-13T07:28:01Z")
Thanks for the quick reply mate.
Is it best practice to log output in json format than plain text?
Is it recommended then to
1. Ship logs directly from filebeats to elasticsearch
2. Ship logs from filebeats to logstash which outputs to elasticsearch
I read this interesting article recently and trying to understand the best solution!
> **[Java app monitoring with ELK - Part I - Logstash and Logback](https://balamaci.ro/java-app-monitoring-with-elk-logstash/)**
>
> Logs for developers are undeniably the most important source of information available to track down problems and understand what is happening with your applications. It makes sense to have a good tool in our toolbox that will enable us to...
Thanks,
Shane.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 13, 2016, 7:33am UTC](https://discuss.elastic.co/t/indexing-application-log-files-to-elasticsearch/55377/4 "2016-07-13T07:33:56Z")
> Is it best practice to log output in json format than plain text?
If you can control the logging format I think it's preferable since there's more or less no configuration to do (no multiline worries for example).
> Is it recommended then to
1. Ship logs directly from filebeats to elasticsearch
2. Ship logs from filebeats to logstash which outputs to elasticsearch
Since Filebeat has basically zero features for processing or parsing events I don't think the first option is very useful.
---
### Author:  [@shane\_lee](https://discuss.elastic.co/u/shane_lee)
#### Post date: [July 13, 2016, 8:19am UTC](https://discuss.elastic.co/t/indexing-application-log-files-to-elasticsearch/55377/5 "2016-07-13T08:19:31Z")
+1 on the json format then! 😛
> [@magnusbaeck](#):
>
> Since Filebeat has basically zero features for processing or parsing events I don't think the first option is very useful.
This is the part I am not sure about.
The framework logstash encoder has an encoder called LoggingEventCompositeJsonEncoder that can provide greater flexibilty in the json format.
> **[GitHub - logfellow/logstash-logback-encoder: Logback JSON encoder and appenders](https://github.com/logfellow/logstash-logback-encoder#composite_encoder)**
>
> Logback JSON encoder and appenders. Contribute to logfellow/logstash-logback-encoder development by creating an account on GitHub.
So I am thinking if I defined patterns at the logging level, do I really need to ship to logstash?
I understand your point about directly over the network. My colleague has mentioned that beats and/or logstash will have a retry mechanism in place for network failures. Is that true?
i see the 12 factor site recommends stdout! [The Twelve-Factor App](http://12factor.net/logs)
Thanks,
Shane.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 13, 2016, 10:31am UTC](https://discuss.elastic.co/t/indexing-application-log-files-to-elasticsearch/55377/6 "2016-07-13T10:31:38Z")
> So I am thinking if I defined patterns at the logging level, do I really need to ship to logstash?
That depends on what kind of filtering you might want to do in Logstash, and if Elasticsearch is the only output you're interested in. There is no right or wrong here. It depends on your needs and preferences.
> I understand your point about directly over the network. My colleague has mentioned that beats and/or logstash will have a retry mechanism in place for network failures. Is that true?
Yes.
---
### Author:  [@shane\_lee](https://discuss.elastic.co/u/shane_lee)
#### Post date: [July 13, 2016, 1:47pm UTC](https://discuss.elastic.co/t/indexing-application-log-files-to-elasticsearch/55377/7 "2016-07-13T13:47:47Z")
Thank you.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:48am UTC](https://discuss.elastic.co/t/indexing-application-log-files-to-elasticsearch/55377/8 "2017-07-06T04:48:14Z")