Filebeat Windows/Linux discrepancies

Elastic Stack Beats
1

I see a discrepancy in Filebeat output between Linux and Windows.

ELK setup:

ELK server - Ubuntu 18.04.1 LTS

  • Elasticsearch version 6.3.2
  • Logstash version 6.3.2
  • Kibana version 6.3.2

ELK node 1 - Ubuntu 18.04.1 LTS, Filebeat version 6.3.2 (installed from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.2-amd64.deb)

ELK node 2 - Windows Server 2012R2, Filebeat version 6.3.2 (installed from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.2-windows-x86_64.zip)

For the sake of the tests I am starting Logstash manually with:
logstash -f 10-log-filter.conf

The content of 10-log-filter.conf file is:

input {
beats {
port => 5043
ssl => false
}
}

filter {
if [type] == "log" {
grok {
match => { "message" => "%{LOGTIMESTAMP:logtimestamp} %{GREEDYDATA:everythingelse}" }
}
mutate {
add_field => { "newtimestamp" => "%{logtimestamp}" }
remove_field => ["logtimestamp"]
}
date {
match => [ "newtimestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
target => "@timestamp"
}
}
}

output { stdout { codec => rubydebug } }

where LOGTIMESTAMP is a custom pattern.

Then harvesting a single file with Filebeat with the following content:

2018-10-17 08:17:18.079 Information Connecting to message broker.

The following is received in Logstash stdout when the test file is harvested by Filebeat under Linux:

{
"@timestamp" => 2018-10-17T08:17:18.079Z,
"source" => "/opt/test/AuthenticationService.log",
"message" => "2018-10-17 08:17:18.079 Information Connecting to message broker.",
"host" => "elk-node01",
"newtimestamp" => "2018-10-17 08:17:18.079",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"fields" => nil,
"@version" => "1",
"offset" => 0,
"type" => "log",
"beat" => {
"hostname" => "elk-node01",
"name" => "elk-node01"
},
"input_type" => "log",
"everythingelse" => "Information Connecting to message broker.",
"count" => 1
}

The following is received in Logstash stdout when the test file is harvested by Filebeat under Windows:

{
"@timestamp" => 2018-11-26T10:28:12.795Z,
"source" => "C:\Microservices\test\AuthenticationService.log",
"message" => "2018-10-17 08:17:18.079 Information Connecting to message broker.",
"host" => {
"name" => "elk-node02"
},
"beat" => {
"hostname" => "elk-node02",
"name" => "elk-node02",
"version" => "6.3.2"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"prospector" => {
"type" => "log"
},
"input" => {
"type" => "log"
},
"@version" => "1",
"offset" => 0
}

There are differences in how the file is being parsed by Logstash.

Therefore I also output the log as a file rather than sending it to Logstash.

The following is the record in output file when the test file is harvested by Filebeat under Linux:

{"@timestamp":"2018-11-26T10:26:42.911Z","beat":{"hostname":"elk-node01","name":"elk-node01"},"count":1,"fields":null,"input_type":"log","message":"2018-10-17 08:17:18.079 Information Connecting to message broker.","offset":0,"source":"/opt/test/AuthenticationService.log","type":"log"}

The following is the record in output file when the test file is harvested by Filebeat under Windows:

{"@timestamp":"2018-11-26T10:31:50.147Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.3.2"},"message":"2018-10-17 08:17:18.079 Information Connecting to message broker.","input":{"type":"log"},"prospector":{"type":"log"},"beat":{"name":"elk-node02","hostname":"elk-node02","version":"6.3.2"},"host":{"name":"elk-node02"},"source":"C:\Microservices\test\AuthenticationService.log","offset":0}

Obviously there is a difference in the output files between Filebeat used under Linux and Windows.

Why is this happening and how to get it fixed (if it should be fixed at all). I also tried with Filebeat version 6.5.1 under Windows but the result is still different.

Filebeat under Linux is working as I would like but not Filebeat under Windows. So what is wrong with Filebeat under Windows ?

2

Well, that's pity. 3 days and not a single reply. I expected more from Elastic team.

3

I am sorry to keep you waiting. Could you please share the configuration of your Filebeat instances formatted using </> and your debug logs.

4

Hi Noémi,

Thank you for the reply.

Filebeat configuratin under Linux is:

filebeat:
  prospectors:
    -
      paths:
        - /opt/test/*.log
      input_type: log
      document_type: log
      multiline:
        pattern: '^(^[0-9]{4}-[0-9]{2}-[0-9]{2})'
        negate: true
        match: after
  idle_timeout: 5s
  registry_file: /var/lib/filebeat/registry
output:
  logstash:
    hosts: ["172.16.11.53:5043"]
    bulk_max_size: 1024
    index: filebeat
shipper:
logging:
  to_files: true
  files:
    path: /var/log/filebeat
    name: filebeat
    rotateeverybytes: 10485760 # = 10MB
    keepfiles: 7
  level: debug

Filebeat configuration under Windows is:

filebeat.inputs:
- type: log
  enabled: true
  paths:
  multiline.pattern: '^(^[0-9]{4}-[0-9]{2}-[0-9]{2})'
  multiline.negate: True
  multiline.match: after
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
output.logstash:
  hosts: 172.16.11.53:5043
  index: filebeat
logging.level: debug

The debug outputs are quite long so I am wondering if there is a way to attach the logs here.

5

For starters ./filebeat version is enough.
I also see a few weird thing in your events. beat.version is added to all events sent by Filebeat since 5.0, but I don't see it in your example Linux event. However, Windows event has this field. Also, host.name is added to events since Filebeat 6.3. But this field is missing from the Linux event. Are you sure you are running Filebeat 6.3 on Linux?

6

Hi Noémi,

You are correct. I was using old Filebeat version under Linux which is my bad as it seems I didn't install Filebeat from a .deb package as I mentioned above but used wrong APT repository. It seems the discrepancies occurred because of the different versions - apologies for that.
I logged the output to a file and now the output files (Windows and Linux) are similar.

Unfortunately this didn't resolve my issue with @timestamp but this is a question for another topic :grinning:
I believe we can resolve this request as the issue with the discrepancies I was observing is sorted out now.
Thank you.

Best regards,

Aleksandar

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.