# Filebeat Line Endings Problem '\\n'

**URL:** <https://discuss.elastic.co/t/filebeat-line-endings-problem-n/157568>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 20, 2018, 2:30pm UTC](https://discuss.elastic.co/t/filebeat-line-endings-problem-n/157568 "2018-11-20T14:30:10Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ahbe2901](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@ahbe2901](https://discuss.elastic.co/u/ahbe2901)\
**Post date:** [November 20, 2018, 2:30pm UTC](https://discuss.elastic.co/t/filebeat-line-endings-problem-n/157568/1 "2018-11-20T14:30:10Z")

</div>

I have the following setting: Filebeat =\> Logstash

My problem is, that Filebeat is not recognizing \n as a line ending and is then packing multiple lines into one message.

Log Input (with Unix line endings \n):

```auto
2018-11-07 17:24:26	178.1.111.11	app/nativeDataCommon/	[]		
2018-11-07 17:24:26	178.1.111.11	app/nativeDataCommon/	[]		
2018-11-07 17:24:28	178.1.111.11	app/nativeDataCommon/	[]				

```

When I have a look at the output from logstash, the message arrives like this:

```auto
"message":"2018-11-07 17:24:26\t178.1.111.11\tapp/nativeDataCommon/\t[]\t\t\n2018-11-07 17:24:26\t178.1.111.11\tapp/nativeDataCommon/\t[]\t\t\n2018-11-07 17:24:28\t178.1.111.11\tapp/nativeDataCommon/\t[]\t\t\n"

```

Anyone an idea what I am doing wrong?  
A bit strange is, that in the stdout there is also the flag 'multiline' set. Even it is not in the config.

```auto
"log":{"flags":["multiline"]}

```

**filebeat.yml:**

```auto
filebeat.inputs:

- type: log
  enabled: false
  paths:
    - /var/log/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

setup.kibana:
  host: "192.168.131.170:5601"

output.logstash:
  hosts: ["localhost:5044"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

```

**filebeat/modules.d/logstash.yml**

```auto
- module: logstash
  log:
    enabled: true
    var.paths: ["/logtemptest8/*"]

  slowlog:
   enabled: false

```

**logstash/conf.d/logstash.conf**

```auto
input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => {"message" => "%{NOTSPACE:date} %{NOTSPACE:time}[\t]%{IP:client}[\t]%{NOTSPACE:request}[\t]%{NOTSPACE:other}"}
  }
  mutate {
    add_field => {
      "timestamp" => "%{date} %{time}"
    }
    remove_field => ["date", "time"]
  }
  date {
    match => ["timestamp", "yyyy-MM-dd HH:mm:ss"]
    timezone => "Europe/Zurich"
  }
}

output {
  stdout {
    codec => json
  }
}

```

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [November 20, 2018, 5:47pm UTC](https://discuss.elastic.co/t/filebeat-line-endings-problem-n/157568/2 "2018-11-20T17:47:52Z")

</div>

Hello,

Filebeat should not have any problem with file ending with '\n' so many tests are covering that use case.  
Lets try to isolate your problem:

1. Define only a single input with the problematic log.
2. Use the [console output](https://www.elastic.co/guide/en/beats/filebeat/current/console-output.html) instead of Logstash.
3. Run filebeat with full debug mode with `-e -v -d "*"`

How the event look like in the console?

---

<div class="post-metadata">

**Author:** ![ahbe2901](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@ahbe2901](https://discuss.elastic.co/u/ahbe2901)\
**Post date:** [November 21, 2018, 8:59am UTC](https://discuss.elastic.co/t/filebeat-line-endings-problem-n/157568/3 "2018-11-21T08:59:49Z")

</div>

Thanks for your answer. Seems all to be okay, when only Filebeat is used.  
I guess the problem is most likely something with the Logstash Module for Filebeat.

The output is the following, the messages are separated correctly.

```auto
{
  "@timestamp": "2018-11-21T08:56:49.807Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.5.0"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "name": "kibana",
    "hostname": "kibana",
    "version": "6.5.0"
  },
  "host": {
    "name": "kibana",
    "architecture": "x86_64",
    "os": {
      "codename": "bionic",
      "platform": "ubuntu",
      "version": "18.04.1 LTS (Bionic Beaver)",
      "family": "debian"
    },
    "id": "4c9d724e74cb4804acc40849f74d4047",
    "containerized": false
  },
  "source": "/logtemptest12/api-app_2.txt",
  "offset": 0,
  "message": "2018-11-07 17:24:26\t178.1.111.11\tapp/tcs/nativeDataCommon/\t[]\t\t",
  "prospector": {
    "type": "log"
  }
}
{
  "@timestamp": "2018-11-21T08:56:49.808Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.5.0"
  },
  "source": "/logtemptest12/api-app_2.txt",
  "offset": 64,
  "message": "2018-11-07 17:24:27\t178.1.111.12\tapp/tcs/nativeDataCommon/\t[]\t\t",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "version": "6.5.0",
    "name": "kibana",
    "hostname": "kibana"
  },
  "host": {
    "name": "kibana",
    "architecture": "x86_64",
    "os": {
      "platform": "ubuntu",
      "version": "18.04.1 LTS (Bionic Beaver)",
      "family": "debian",
      "codename": "bionic"
    },
    "id": "4c9d724e74cb4804acc40849f74d4047",
    "containerized": false
  }
}

```

---

<div class="post-metadata">

**Author:** ![ahbe2901](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@ahbe2901](https://discuss.elastic.co/u/ahbe2901)\
**Post date:** [November 21, 2018, 10:05am UTC](https://discuss.elastic.co/t/filebeat-line-endings-problem-n/157568/4 "2018-11-21T10:05:21Z")

</div>

I finally worked it out.

If the log paths are defined in filebeat.yml all is fine and working correctly. The misbehaviour only happens if the log paths are defined in modules.d/logstash.yml.

So this is **not** working (modules.d/logstash.yml):

```auto
- module: logstash
  log:
    enabled: true
    var.paths: ["/logtemp17/*"]

```

And this is working (filebeat.yml):

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /logtemp17/*

```

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [November 21, 2018, 2:35pm UTC](https://discuss.elastic.co/t/filebeat-line-endings-problem-n/157568/5 "2018-11-21T14:35:10Z")

</div>

It is possible that /logtemp17/ contains other logs than Logstash?

The logstash module uses multiline on the file, if other logs are present in the directory it might not be able to correctly merge the line.

---

<div class="post-metadata">

**Author:** ![ahbe2901](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@ahbe2901](https://discuss.elastic.co/u/ahbe2901)\
**Post date:** [November 21, 2018, 3:48pm UTC](https://discuss.elastic.co/t/filebeat-line-endings-problem-n/157568/6 "2018-11-21T15:48:51Z")

</div>

What you mean exactly with other logs than Logstash?

The directory contained only one file. But this one was with the ending .txt, maybe this has an impact.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 19, 2018, 3:49pm UTC](https://discuss.elastic.co/t/filebeat-line-endings-problem-n/157568/7 "2018-12-19T15:49:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.