# Logstash errors after upgrading to filebeat-6.3.0

**URL:** <https://discuss.elastic.co/t/logstash-errors-after-upgrading-to-filebeat-6-3-0/135984>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 14, 2018, 6:51pm UTC](https://discuss.elastic.co/t/logstash-errors-after-upgrading-to-filebeat-6-3-0/135984 "2018-06-14T18:51:00Z")\
**Posts on this page:** 10\
**Page:** 2

<div class="post-metadata">

**Author:** ![ceekay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ceekay/32/5687_2.png) [@ceekay](https://discuss.elastic.co/u/ceekay)\
**Post date:** [June 29, 2018, 1:43pm UTC](https://discuss.elastic.co/t/logstash-errors-after-upgrading-to-filebeat-6-3-0/135984/21 "2018-06-29T13:43:47Z")

</div>

Thanks - I'll need to test the processor on Monday though. It's well outside work hours now in my timezone.

As for the `host` field, we set it directly in the Beat's config (resolved from Puppet or Ansible facts) - it's not added at the LS Beats input. This is the only reliable way we've found to get a system's FQDN, as Go doesn't appear to be able to resolve them. The `beat.hostname` field is a short hostname, so that would cause the name collisions mentioned above.

Given that we're adding `host` as part of the Beat's config, will the processor rename the new host object before our custom field is added? If not I'm not sure how this solution would work.

Here's an example config to explain:

```auto
filebeat:
  registry_file: /var/lib/filebeat/registry
  config.prospectors:
    enabled: true
    path: /etc/filebeat/conf.d/*.yml
fields:
  host: sample.host.fqdn
  timezone: Pacific/Auckland
fields_under_root: true
output:
  logstash:
    hosts: ["ingress1.xxx.xxx:10514", "ingress2.xxx.xxx:10514"]
    compression_level: 3
    ssl:
      certificate_authorities: ["/etc/ssl/certs/ingress.crt"]
      verification_mode: full
[...]

```

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [June 29, 2018, 1:48pm UTC](https://discuss.elastic.co/t/logstash-errors-after-upgrading-to-filebeat-6-3-0/135984/22 "2018-06-29T13:48:50Z")

</div>

I see, you set host directly on the Beats level. I need to check in detail in what order the processors are applied.

I also need to test your above config on what the affect is. It should probably work as from a Beats perspective it's fine to have `host` and `host.name`. To remove only 1 field you could drop just `host.name` instead of `host`. Let me know if that works.

Side note: My hope is that we can actually solve the FQDN problem with the `add_host_metadata` processor and have a config option there for making `host.name` FQDN.

---

<div class="post-metadata">

**Author:** ![ceekay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ceekay/32/5687_2.png) [@ceekay](https://discuss.elastic.co/u/ceekay)\
**Post date:** [June 29, 2018, 1:58pm UTC](https://discuss.elastic.co/t/logstash-errors-after-upgrading-to-filebeat-6-3-0/135984/23 "2018-06-29T13:58:28Z")

</div>

My initial testing showed that our custom field wasn't making it to the output - there was no FQDN at all in the Filebeat 6.3.0 messages, but the`host.name` field was present.

Reverting back to 6.2.4 with the same config returned it to what we'd normally expect to see.

If we can possibly keep the host metadata (but renamed!) that would be great as it looks pretty useful.

[edit]  
OK I just tried the rename processor (renaming `host` -\> `node`) on a dev box and here's what I got. The custom `host` field was suppressed, unfortunately.

```auto
6.3.0 output:
{
  "@timestamp": "2018-06-29T13:57:54.914Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.0"
  },
  "beat": {
    "name": "dev-mgmt1",
    "hostname": "dev-mgmt1",
    "version": "6.3.0"
  },
  "offset": 49993950,
  "node": {
    "name": "dev-mgmt1"
  },
  "prospector": {
    "type": "log"
  },
  "platform": "Development",
  "type": "latency-test",
  "input": {
    "type": "log"
  },
  "timezone": "Pacific/Auckland",
  "source": "/var/log/latency-test.log",
  "message": "1530280673.966",
  "client_id": “dev”,
}

```

```auto
6.2.4 output:
{
  "@timestamp": "2018-06-29T14:03:31.858Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "message": "1530281011.585",
  "source": "/var/log/latency-test.log",
  "offset": 49999020,
  "platform": "Development",
  "prospector": {
    "type": "log"
  },
  "timezone": "Pacific/Auckland",
  "type": "latency-test",
  "client_id": “dev”,
  "host": "dev-mgmt1.fqdn.xxx”,
  "beat": {
    "name": "dev-mgmt1",
    "hostname": "dev-mgmt1",
    "version": "6.2.4"
  }
}

```

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [June 29, 2018, 2:22pm UTC](https://discuss.elastic.co/t/logstash-errors-after-upgrading-to-filebeat-6-3-0/135984/24 "2018-06-29T14:22:02Z")

</div>

Thanks for testing. I need to test this also on my end but an other thing you could try is adding `host` field under and other name and then use the rename processor after you renamed the `host.*` fields. But not 100% sure it will work.

The general problem is that `fields: *` existed before the processor existed which makes it a bit special, meaning you can define the order it is executed. That is why I want to replace it by a processor: [https://github.com/elastic/beats/issues/7350](https://github.com/elastic/beats/issues/7350) If the above does not work, sounds like we should increase the priority for it.

---

<div class="post-metadata">

**Author:** ![ceekay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ceekay/32/5687_2.png) [@ceekay](https://discuss.elastic.co/u/ceekay)\
**Post date:** [June 29, 2018, 2:33pm UTC](https://discuss.elastic.co/t/logstash-errors-after-upgrading-to-filebeat-6-3-0/135984/25 "2018-06-29T14:33:00Z")

</div>

I'll try the `host` rename a bit later - it's after 2am here and I probably shouldn't be working.

Just a note on issue #7350: If adding fields is replaced by a processor, will there be allowances to continue adding them in external prospectors? Our prospectors are dropped in as fragments, and also contain root fields, such as `client_id` and `type`.

---

<div class="post-metadata">

**Author:** ![ceekay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ceekay/32/5687_2.png) [@ceekay](https://discuss.elastic.co/u/ceekay)\
**Post date:** [July 1, 2018, 11:21pm UTC](https://discuss.elastic.co/t/logstash-errors-after-upgrading-to-filebeat-6-3-0/135984/26 "2018-07-01T23:21:56Z")

</div>

The rename appears to work fine. I've got the results I need with this config, adding a field named fqdn instead of host:

```auto
[...]
fields:
  fqdn: dev-mgmt1.fqdn.xxx
processors:
- rename:
    fields:
     - from: "host"
       to: "node"
     - from: "fqdn"
       to: "host"
    ignore_missing: true
[...]

```

This gives the following output:

```auto
[...]
  "host": "dev-mgmt1.fqdn.xxx,
[...]
  "node": {
    "name": "dev-mgmt1"
  },
[...]

```

My only reservation here is, would this be considered a hack? Is it going to cause us problems down the line?

Thanks for all your help so far @ruflin

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [July 2, 2018, 12:16pm UTC](https://discuss.elastic.co/t/logstash-errors-after-upgrading-to-filebeat-6-3-0/135984/27 "2018-07-02T12:16:51Z")

</div>

For your first question about the processor: Processors also work on the prospector level so this feature would keep working. A replacement for `fields` config option would have to same or more functionality. The interesting part is as soon as we have a processor, we can add more config options like `overwrite_existing` etc.

I would not consider the above a hack so you should be fine also in the long run. But as you know, these things are tricky 😉

One note for the `host.*` in general: If you have the option and see a path forward where you can introduce it in your ingestion tooling in the long run, I would recommend it as we will start to use `host.*` more and more in 7.0.

Let me know if I can help further.

---

<div class="post-metadata">

**Author:** ![mksavic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mksavic/32/32920_2.png) [@mksavic](https://discuss.elastic.co/u/mksavic)\
**Post date:** [July 2, 2018, 10:55pm UTC](https://discuss.elastic.co/t/logstash-errors-after-upgrading-to-filebeat-6-3-0/135984/28 "2018-07-02T22:55:11Z")

</div>

For clarification, this would go in a Logstash `conf` file?

So the following example `conf` file...

```auto
input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "metricbeat"
  }
}

```

Would be this?

```auto
input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "metricbeat"
  }
}
mutate {
  remove_field => ["[host][name]" ]
  remove_field => ["[host][id]" ]
  remove_field => ["[host][architecture]" ]
  remove_field => ["[host][os][platform]" ]
  remove_field => ["[host][os][version]" ]
  remove_field => ["[host][os][family]" ]
  remove_field => ["[host][ip]" ]
  remove_field => ["[host][mac]" ]
  remove_field => ["[host][os]" ]
  remove_field => ["[host]" ]
}
 mutate {
  add_field => {
    "host" => "%{[beat][hostname]}"
  }
}

```

---

<div class="post-metadata">

**Author:** ![nxhack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nxhack/32/3939_2.png) [@nxhack](https://discuss.elastic.co/u/nxhack)\
**Post date:** [July 2, 2018, 11:50pm UTC](https://discuss.elastic.co/t/logstash-errors-after-upgrading-to-filebeat-6-3-0/135984/29 "2018-07-02T23:50:55Z")

</div>

Hi @mksavic

```auto
input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "metricbeat"
  }
}

filter {
  mutate {
    remove_field => ["[host]" ]
  }
  mutate {
    add_field => {
      "host" => "%{[beat][hostname]}"
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 31, 2018, 12:05am UTC](https://discuss.elastic.co/t/logstash-errors-after-upgrading-to-filebeat-6-3-0/135984/30 "2018-07-31T00:05:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/logstash-errors-after-upgrading-to-filebeat-6-3-0/135984.md?page=1)