Logstash errors after upgrading to filebeat-6.3.0

[ceekay]

Thanks - I'll need to test the processor on Monday though. It's well outside work hours now in my timezone.

As for the host field, we set it directly in the Beat's config (resolved from Puppet or Ansible facts) - it's not added at the LS Beats input. This is the only reliable way we've found to get a system's FQDN, as Go doesn't appear to be able to resolve them. The beat.hostname field is a short hostname, so that would cause the name collisions mentioned above.

Given that we're adding host as part of the Beat's config, will the processor rename the new host object before our custom field is added? If not I'm not sure how this solution would work.

Here's an example config to explain:

filebeat:
  registry_file: /var/lib/filebeat/registry
  config.prospectors:
    enabled: true
    path: /etc/filebeat/conf.d/*.yml
fields:
  host: sample.host.fqdn
  timezone: Pacific/Auckland
fields_under_root: true
output:
  logstash:
    hosts: ["ingress1.xxx.xxx:10514", "ingress2.xxx.xxx:10514"]
    compression_level: 3
    ssl:
      certificate_authorities: ["/etc/ssl/certs/ingress.crt"]
      verification_mode: full
[...]

[ruflin]

I see, you set host directly on the Beats level. I need to check in detail in what order the processors are applied.

I also need to test your above config on what the affect is. It should probably work as from a Beats perspective it's fine to have host and host.name. To remove only 1 field you could drop just host.name instead of host. Let me know if that works.

Side note: My hope is that we can actually solve the FQDN problem with the add_host_metadata processor and have a config option there for making host.name FQDN.

[ceekay]

My initial testing showed that our custom field wasn't making it to the output - there was no FQDN at all in the Filebeat 6.3.0 messages, but thehost.name field was present.

Reverting back to 6.2.4 with the same config returned it to what we'd normally expect to see.

If we can possibly keep the host metadata (but renamed!) that would be great as it looks pretty useful.

[edit]
OK I just tried the rename processor (renaming host -> node) on a dev box and here's what I got. The custom host field was suppressed, unfortunately.

6.3.0 output:
{
  "@timestamp": "2018-06-29T13:57:54.914Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.0"
  },
  "beat": {
    "name": "dev-mgmt1",
    "hostname": "dev-mgmt1",
    "version": "6.3.0"
  },
  "offset": 49993950,
  "node": {
    "name": "dev-mgmt1"
  },
  "prospector": {
    "type": "log"
  },
  "platform": "Development",
  "type": "latency-test",
  "input": {
    "type": "log"
  },
  "timezone": "Pacific/Auckland",
  "source": "/var/log/latency-test.log",
  "message": "1530280673.966",
  "client_id": “dev”,
}

6.2.4 output:
{
  "@timestamp": "2018-06-29T14:03:31.858Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "message": "1530281011.585",
  "source": "/var/log/latency-test.log",
  "offset": 49999020,
  "platform": "Development",
  "prospector": {
    "type": "log"
  },
  "timezone": "Pacific/Auckland",
  "type": "latency-test",
  "client_id": “dev”,
  "host": "dev-mgmt1.fqdn.xxx”,
  "beat": {
    "name": "dev-mgmt1",
    "hostname": "dev-mgmt1",
    "version": "6.2.4"
  }
}

[ruflin]

Thanks for testing. I need to test this also on my end but an other thing you could try is adding host field under and other name and then use the rename processor after you renamed the host.* fields. But not 100% sure it will work.

The general problem is that fields: * existed before the processor existed which makes it a bit special, meaning you can define the order it is executed. That is why I want to replace it by a processor: https://github.com/elastic/beats/issues/7350 If the above does not work, sounds like we should increase the priority for it.

[ceekay]

I'll try the host rename a bit later - it's after 2am here and I probably shouldn't be working.

Just a note on issue #7350: If adding fields is replaced by a processor, will there be allowances to continue adding them in external prospectors? Our prospectors are dropped in as fragments, and also contain root fields, such as client_id and type.

[ceekay]

The rename appears to work fine. I've got the results I need with this config, adding a field named fqdn instead of host:

[...]
fields:
  fqdn: dev-mgmt1.fqdn.xxx
processors:
- rename:
    fields:
     - from: "host"
       to: "node"
     - from: "fqdn"
       to: "host"
    ignore_missing: true
[...]

This gives the following output:

[...]
  "host": "dev-mgmt1.fqdn.xxx,
[...]
  "node": {
    "name": "dev-mgmt1"
  },
[...]

My only reservation here is, would this be considered a hack? Is it going to cause us problems down the line?

Thanks for all your help so far @ruflin

[ruflin]

For your first question about the processor: Processors also work on the prospector level so this feature would keep working. A replacement for fields config option would have to same or more functionality. The interesting part is as soon as we have a processor, we can add more config options like overwrite_existing etc.

I would not consider the above a hack so you should be fine also in the long run. But as you know, these things are tricky

One note for the host.* in general: If you have the option and see a path forward where you can introduce it in your ingestion tooling in the long run, I would recommend it as we will start to use host.* more and more in 7.0.

Let me know if I can help further.

[mksavic (Mel)]

For clarification, this would go in a Logstash conf file?

So the following example conf file...

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "metricbeat"
  }
}

Would be this?

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "metricbeat"
  }
}
mutate {
  remove_field => [ "[host][name]" ]
  remove_field => [ "[host][id]" ]
  remove_field => [ "[host][architecture]" ]
  remove_field => [ "[host][os][platform]" ]
  remove_field => [ "[host][os][version]" ]
  remove_field => [ "[host][os][family]" ]
  remove_field => [ "[host][ip]" ]
  remove_field => [ "[host][mac]" ]
  remove_field => [ "[host][os]" ]
  remove_field => [ "[host]" ]
}
 mutate {
  add_field => {
    "host" => "%{[beat][hostname]}"
  }
}

[nxhack]

Hi @mksavic

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "metricbeat"
  }
}

filter {
  mutate {
    remove_field => [ "[host]" ]
  }
  mutate {
    add_field => {
      "host" => "%{[beat][hostname]}"
    }
  }
}