Filebeat Line Endings Problem '\n'

Elastic Stack Beats
1

I have the following setting: Filebeat => Logstash

My problem is, that Filebeat is not recognizing \n as a line ending and is then packing multiple lines into one message.

Log Input (with Unix line endings \n):

2018-11-07 17:24:26	178.1.111.11	app/nativeDataCommon/	[]		
2018-11-07 17:24:26	178.1.111.11	app/nativeDataCommon/	[]		
2018-11-07 17:24:28	178.1.111.11	app/nativeDataCommon/	[]

When I have a look at the output from logstash, the message arrives like this:

"message":"2018-11-07 17:24:26\t178.1.111.11\tapp/nativeDataCommon/\t[]\t\t\n2018-11-07 17:24:26\t178.1.111.11\tapp/nativeDataCommon/\t[]\t\t\n2018-11-07 17:24:28\t178.1.111.11\tapp/nativeDataCommon/\t[]\t\t\n"

Anyone an idea what I am doing wrong?
A bit strange is, that in the stdout there is also the flag 'multiline' set. Even it is not in the config.

"log":{"flags":["multiline"]}

filebeat.yml:

filebeat.inputs:

- type: log
  enabled: false
  paths:
    - /var/log/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

setup.kibana:
  host: "192.168.131.170:5601"

output.logstash:
  hosts: ["localhost:5044"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

filebeat/modules.d/logstash.yml

- module: logstash
  log:
    enabled: true
    var.paths: ["/logtemptest8/*"]

  slowlog:
   enabled: false

logstash/conf.d/logstash.conf

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => {"message" => "%{NOTSPACE:date} %{NOTSPACE:time}[\t]%{IP:client}[\t]%{NOTSPACE:request}[\t]%{NOTSPACE:other}"}
  }
  mutate {
    add_field => {
      "timestamp" => "%{date} %{time}"
    }
    remove_field => ["date", "time"]
  }
  date {
    match => ["timestamp", "yyyy-MM-dd HH:mm:ss"]
    timezone => "Europe/Zurich"
  }
}

output {
  stdout {
    codec => json
  }
}
2

Hello,

Filebeat should not have any problem with file ending with '\n' so many tests are covering that use case.
Lets try to isolate your problem:

  1. Define only a single input with the problematic log.
  2. Use the console output instead of Logstash.
  3. Run filebeat with full debug mode with -e -v -d "*"

How the event look like in the console?

3

Thanks for your answer. Seems all to be okay, when only Filebeat is used.
I guess the problem is most likely something with the Logstash Module for Filebeat.

The output is the following, the messages are separated correctly.

{
  "@timestamp": "2018-11-21T08:56:49.807Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.5.0"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "name": "kibana",
    "hostname": "kibana",
    "version": "6.5.0"
  },
  "host": {
    "name": "kibana",
    "architecture": "x86_64",
    "os": {
      "codename": "bionic",
      "platform": "ubuntu",
      "version": "18.04.1 LTS (Bionic Beaver)",
      "family": "debian"
    },
    "id": "4c9d724e74cb4804acc40849f74d4047",
    "containerized": false
  },
  "source": "/logtemptest12/api-app_2.txt",
  "offset": 0,
  "message": "2018-11-07 17:24:26\t178.1.111.11\tapp/tcs/nativeDataCommon/\t[]\t\t",
  "prospector": {
    "type": "log"
  }
}
{
  "@timestamp": "2018-11-21T08:56:49.808Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.5.0"
  },
  "source": "/logtemptest12/api-app_2.txt",
  "offset": 64,
  "message": "2018-11-07 17:24:27\t178.1.111.12\tapp/tcs/nativeDataCommon/\t[]\t\t",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "version": "6.5.0",
    "name": "kibana",
    "hostname": "kibana"
  },
  "host": {
    "name": "kibana",
    "architecture": "x86_64",
    "os": {
      "platform": "ubuntu",
      "version": "18.04.1 LTS (Bionic Beaver)",
      "family": "debian",
      "codename": "bionic"
    },
    "id": "4c9d724e74cb4804acc40849f74d4047",
    "containerized": false
  }
}
4

I finally worked it out.

If the log paths are defined in filebeat.yml all is fine and working correctly. The misbehaviour only happens if the log paths are defined in modules.d/logstash.yml.

So this is not working (modules.d/logstash.yml):

- module: logstash
  log:
    enabled: true
    var.paths: ["/logtemp17/*"]

And this is working (filebeat.yml):

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /logtemp17/*
5

It is possible that /logtemp17/ contains other logs than Logstash?

The logstash module uses multiline on the file, if other logs are present in the directory it might not be able to correctly merge the line.

6

What you mean exactly with other logs than Logstash?

The directory contained only one file. But this one was with the ending .txt, maybe this has an impact.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.