# Logstash Multiline and line codec differences **URL:** \ **Category:** Elasticsearch\ **Created:** [November 19, 2023, 7:59am UTC](https://discuss.elastic.co/t/logstash-multiline-and-line-codec-differences/347466 "2023-11-19T07:59:16Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![randomnamegenerator](https://avatars.discourse-cdn.com/v4/letter/r/7ab992/32.png) [@randomnamegenerator](https://discuss.elastic.co/u/randomnamegenerator)\ **Post date:** [November 19, 2023, 7:59am UTC](https://discuss.elastic.co/t/logstash-multiline-and-line-codec-differences/347466/1 "2023-11-19T07:59:16Z")
Hello All, We have application logs coming in from a number of different hosts (shipped with filebeat) and have obvserved a mixing of datastreams for one of the log types. We changed the logstash input.config from version A to B which appears to have resolved the issue. We are using ELK 7.10. A) input { beats { client\_inactivity\_timeout =\> 1200 port =\> 5044 codec =\> line { charset =\> "ISO-8859-1" } } } B) input { beats { port =\> 5044 } } Below is our filebeat config for reference input\_type: log paths: - C:\bla\bla.\*.exe.log fields: {log\_type: App\_log\_files} fields\_under\_root: true multiline.pattern: '^(20[0-9]{2}(-[0-9]{2}){2} [0-9]{2}(:[0-9]{2}){2})|([0-9]{2} [JFMASOND][a-z]{2} 20[0-9]{2})' multiline.negate: true multiline.match: after close\_removed: false close\_eof: false clean\_removed: true scan\_frequency: 1h backoff: 5m I have read using multiline codec can cause this (mentioned in the KB below) but nothing is mentioned about this in the line code KB. > **[Multiline codec plugin | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html)** > **[Line codec plugin | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-line.html)** Is it also possible that the line code can cause this issue? Also, can anyone suggest an improvement on version B for our input? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [November 19, 2023, 1:13pm UTC](https://discuss.elastic.co/t/logstash-multiline-and-line-codec-differences/347466/2 "2023-11-19T13:13:10Z")
> [@randomnamegenerator](#): > > have obvserved a mixing of datastreams for one of the log types. What kind of mixing? Can you share some examples? > [@randomnamegenerator](#): > > Is it also possible that the line code can cause this issue? If you are using filebeat to send logs to Logstash you should not use the codec you used in your input A. Filebeat will send logs as json using UTF-8. > [@randomnamegenerator](#): > > Also, can anyone suggest an improvement on version B for our input? What kind of improvement? The beats input is pretty simple, normally you just set the host and the port. ---
**Author:** ![randomnamegenerator](https://avatars.discourse-cdn.com/v4/letter/r/7ab992/32.png) [@randomnamegenerator](https://discuss.elastic.co/u/randomnamegenerator)\ **Post date:** [November 19, 2023, 6:52pm UTC](https://discuss.elastic.co/t/logstash-multiline-and-line-codec-differences/347466/3 "2023-11-19T18:52:45Z")
thanks for the prompt reply. > [@leandrojmp](#): > > What kind of mixing? Can you share some examples? Sure, we have 2 different application log file types coming into separate daily logstash indices, one for server and one for the various services. Each with their own input entry in filebeat.yml and separate .conf filter file. We observed from Kibana log lines from the server log consistently appearing in one of the service log fields though this stopped after making the changes to the logstash input conf (version B). There is no crossover between the two different log types or shared processes etc so it was odd seeing server log lines appear in the service logs in kibana. > [@leandrojmp](#): > > If you are using filebeat to send logs to Logstash you should not use the codec you used in your input A. Filebeat will send logs as json using UTF-8. Ok, so if we are sending multiline application logs the multiline aspect of this should be configured in filebeat.yml and we don´t need any additional conf for the logstash input other than what is already in version B? > [@leandrojmp](#): > > What kind of improvement? The beats input is pretty simple, normally you just set the host and the port. I guess a better question would have been, is the config we are currently running best practice? is there are recommended logstash input codec for setup such as this. Also, will we be able to fix (remove the server log lines from the service log indices) by just reindexing the previous daily log indices. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [November 19, 2023, 7:09pm UTC](https://discuss.elastic.co/t/logstash-multiline-and-line-codec-differences/347466/4 "2023-11-19T19:09:36Z")
> [@randomnamegenerator](#): > > so it was odd seeing server log lines appear in the service logs in kibana. You would need to share your entire `filebeat.yml` file and also your entire `conf` file to help understand what happened, if you do, please use the preformatted text option, the `` button. It is pretty hard to understand configuration when it is not properly formatted. But per default, different conf files in logstash does not mean that they are independent from each other, if you do not configure [multiple pipelines](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html), then all the conf files inside `/etc/logstash/conf.d` will be merged as one configuration. > [@randomnamegenerator](#): > > Ok, so if we are sending multiline application logs the multiline aspect of this should be configured in filebeat.yml and we don´t need any additional conf for the logstash input other than what is already in version B? If you have multiline logs and are using filebeat, then the configuration should be done in Filebeat side, not Logstash. The beats inputs is pretty simple, in most of the cases you just need the port and that's it. > [@randomnamegenerator](#): > > I guess a better question would have been, is the config we are currently running best practice? is there are recommended logstash input codec for setup such as this. It depends on your case, you didn't provide enough context, the beats always send data using json and with the UTF-8 codec, so there is no need to change the charset, if you want you can use the `json` codec in the input to parse the data directly in the input, but you can also do the same thing using a `json` filter in the `filter` block. I prefer to leave the codec as the default, which is the `plain` codec, and do the parsing in the filter block. ---
**Author:** ![randomnamegenerator](https://avatars.discourse-cdn.com/v4/letter/r/7ab992/32.png) [@randomnamegenerator](https://discuss.elastic.co/u/randomnamegenerator)\ **Post date:** [November 20, 2023, 8:46pm UTC](https://discuss.elastic.co/t/logstash-multiline-and-line-codec-differences/347466/5 "2023-11-20T20:46:59Z")
> [@leandrojmp](#): > > You would need to share your entire `filebeat.yml` file and also your entire `conf` file to help understand what happened, if you do, please use the preformatted text option, the `` button. > > It is pretty hard to understand configuration when it is not properly formatted. Hopefully this is easier on the eyes, this is our filebeat.yml from the host servers. ```auto filebeat.registry.path: ./filebeat logging.to_files: true logging.files: path: C:\Data\Logs\Monitoring name: filebeat.log rotateeverybytes: 10485760 keepfiles: 2 logging.level: info filebeat.spool_size: 102400 filebeat.idle_timeout: 15s filebeat.inputs: - input_type: log paths: - C:\Data\Logs\Company.Application.Server.exe.log fields: {log_type: Server_log_files} fields_under_root: true multiline.pattern: '^(20[0-9]{2}(-[0-9]{2}){2} [0-9]{2}(:[0-9]{2}){2})|([0-9]{2} [JFMASOND][a-z]{2} 20[0-9]{2})' multiline.negate: true multiline.match: after close_removed: false close_eof: false clean_removed: true scan_frequency: 1h backoff: 5m - input_type: log paths: - C:\Data\Logs\Company.Application.DataImport.*.exe.log fields: {log_type: Ovalis_DI_log_files} fields_under_root: true multiline.pattern: '^(20[0-9]{2}(-[0-9]{2}){2} [0-9]{2}(:[0-9]{2}){2})|([0-9]{2} [JFMASOND][a-z]{2} 20[0-9]{2})' multiline.negate: true multiline.match: after close_removed: false close_eof: false clean_removed: true scan_frequency: 1h backoff: 5m output.logstash: hosts: ["Monitoring.server.com:5044"] bulk_max_size: 8192 worker: 4 compression_level: 0 pipelining: 5 ``` > [@leandrojmp](#): > > But per default, different conf files in logstash does not mean that they are independent from each other, if you do not configure [multiple pipelines](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html), then all the conf files inside `/etc/logstash/conf.d` will be merged as one configuration. Can you explain why multiple pipelines is better than having just a main/default pipeline? Is it for redundancy, performance of both? I am also a little confused how to set this up, is the setup below correct? pipelines.conf ```auto - pipeline.id: data-import-pipeline path.config: "C:\logstash\conf.d\DI_log_files.conf" - pipeline.id: ovalis-server-pipeline path.config: "C:\logstash\conf.d\Server_log_files.conf" ``` input.conf ```auto input { beats { port => 5044 } } ``` output.conf ```auto output } if [log_type] == "Server_log_files" { if "elastic" in [tags] { elasticsearch { hosts => ["localhost:9200"] index => ["logstash-srv-%{+YYYY.MM.dd}"] document_id => "%{fingerprint}" } } stdout {codec =>rubydebug} } if [log_type] == "DI_log_files" { if "elastic" in [tags] { elasticsearch { hosts => ["localhost:9200"] index => ["logstash-di-%{+YYYY.MM.dd}"] document_id => "%{fingerprint}" } } stdout {codec =>rubydebug} } } ``` > [@leandrojmp](#): > > I prefer to leave the codec as the default, which is the `plain` codec, and do the parsing in the filter block. when you say its default do I have to define codec =\> plain (or similar) in the input.conf or this doesn´t just setting defining beats and the port will cover this. Thanks in advance ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [November 20, 2023, 10:09pm UTC](https://discuss.elastic.co/t/logstash-multiline-and-line-codec-differences/347466/6 "2023-11-20T22:09:26Z")
> [@randomnamegenerator](#): > > Can you explain why multiple pipelines is better than having just a main/default pipeline? Is it for redundancy, performance of both? Multiple pipeline are used when you don't to avoid the risk of the data from twow or more different inputs to mix up as using multiple pipelines will run each pipeline independently from each other. If you do not use multiple pipelines Logstash will merge all the configuration files in the configuration path. But in your case this is not the issue, your Filebeat is configured to send logs to Logstash on port `5044`, and you can have only one input listening on this port. Also, you have conditionals in your configuration, I don't see how your logs would be mixed in the output. You are correctly adding a field for each input in filebeat and using this field to filter the logs in your Logstash, this is already the right approach. Are you still experiencing mixed logs? > [@randomnamegenerator](#): > > when you say its default do I have to define codec =\> plain (or similar) in the input.conf or this doesn´t just setting defining beats and the port will cover this. For the beats input you just need to set the `port`, you do not need to set the `codec` as `plain` because this is already the default codec. You will however need a `json` filter in your logstash pipeline if you do not set the `beats` codec to `json`. ---
**Author:** ![randomnamegenerator](https://avatars.discourse-cdn.com/v4/letter/r/7ab992/32.png) [@randomnamegenerator](https://discuss.elastic.co/u/randomnamegenerator)\ **Post date:** [November 23, 2023, 7:42pm UTC](https://discuss.elastic.co/t/logstash-multiline-and-line-codec-differences/347466/7 "2023-11-23T19:42:09Z")
Apologies for the delay in my reply. > [@leandrojmp](#): > > Multiple pipeline are used when you don't to avoid the risk of the data from twow or more different inputs to mix up as using multiple pipelines will run each pipeline independently from each other. > > If you do not use multiple pipelines Logstash will merge all the configuration files in the configuration path. > > But in your case this is not the issue, your Filebeat is configured to send logs to Logstash on port `5044`, and you can have only one input listening on this port. > > Also, you have conditionals in your configuration, I don't see how your logs would be mixed in the output. > > You are correctly adding a field for each input in filebeat and using this field to filter the logs in your Logstash, this is already the right approach. > > Are you still experiencing mixed logs? Just to clarify, our issue was resolved after making the changes to our logstash input.conf (removing line codec and charset). So all good on that front. Is it only possible to have multiple logstash pipelines if logstash is pulling the files statically from disk/network share? > [@leandrojmp](#): > > You will however need a `json` filter in your logstash pipeline if you do not set the `beats` codec to `json`. How would I apply this to our current setup and or is it necessary ? It looks as though the logs are coming as hoped Do I just have to adjust output.conf I am curious about the different logstash logs for logstash itself, if I want to see any errors or just how things are going in general when each log is read/ingested/sent to elasticsearch how do I see this? Looking through logstash-plain only shows me initial logstashs initial connection to elasticsearch and thats about it. Also thanks again ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 21, 2023, 7:42pm UTC](https://discuss.elastic.co/t/logstash-multiline-and-line-codec-differences/347466/8 "2023-12-21T19:42:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.