# Unable to remove %{host} field when using kafka output plugin with codec plain

**URL:** <https://discuss.elastic.co/t/unable-to-remove-host-field-when-using-kafka-output-plugin-with-codec-plain/114684>\
**Category:** Logstash\
**Created:** [January 9, 2018, 10:44am UTC](https://discuss.elastic.co/t/unable-to-remove-host-field-when-using-kafka-output-plugin-with-codec-plain/114684 "2018-01-09T10:44:00Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![berlin](https://avatars.discourse-cdn.com/v4/letter/b/41988e/32.png) [@berlin](https://discuss.elastic.co/u/berlin)\
**Post date:** [January 9, 2018, 10:44am UTC](https://discuss.elastic.co/t/unable-to-remove-host-field-when-using-kafka-output-plugin-with-codec-plain/114684/1 "2018-01-09T10:44:01Z")

</div>

Hi,  
Please review the below configuration:

```
input {
  sqs {
    access_key_id => ""
    secret_access_key => ""
    polling_frequency => "1"
    queue => "xxx"
    region => "eu-west-1"
    threads => "1"
    codec => plain
  }
}

filter {
  mutate { rename => { "@timestamp" => "timestamp" } }
  mutate { remove_field => ["@version", "path", "host", "type"] }
}

output {
  kafka {
    bootstrap_servers=> "kafka:9092"
    topic_id=> "xxx"
    codec=> "plain"
  }
  stdout { codec => "rubydebug" }
}

```

The plugin does not remove the %{host} field, its just pass to kafka the field in this format "%{host}"

`/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic xxx --property print.key=true --property key.separator="-"`

Kafka output: `%{host} <some xml>`

The messages does not contain %{host} when using `json codec` in kafka output plugin or when using `stdout { codec => rubydebug }`, but I dont want to use the json codec, I just want to keep the xml format from sqs to kafka, and I want logstash not to add the `%{host}` field, any suggestions ?

Thanks

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 9, 2018, 12:37pm UTC](https://discuss.elastic.co/t/unable-to-remove-host-field-when-using-kafka-output-plugin-with-codec-plain/114684/2 "2018-01-09T12:37:11Z")

</div>

The plain codec does this to encode the event for transmission.  
`encoded = @format ? event.sprintf(@format) : event.to_s`

In your config, `format` is not used so `event.to_s` is invoked - but `to_s` is usually used for debugging and it adds the `%{host}` bit.

You should use the `format` setting. Presuming that the `message` field holds the xml and you don't need the timestamp, you can do:

```auto
    codec => plain { format => "%{message}" }

```

or if you want the timestamp

```auto
    codec => plain { format => "%{timestamp} -- %{message}" }

```

Use the fields as you saw them before with the rubydebug codec to experiment without the kafka output...

```auto
output {
  stdout { codec => plain { format => "%{timestamp} -- %{message}" } }
}

```

What you see in stdout is what you will get in kafka. When you are happy switch back to the kafka output.

---

<div class="post-metadata">

**Author:** ![berlin](https://avatars.discourse-cdn.com/v4/letter/b/41988e/32.png) [@berlin](https://discuss.elastic.co/u/berlin)\
**Post date:** [January 9, 2018, 1:50pm UTC](https://discuss.elastic.co/t/unable-to-remove-host-field-when-using-kafka-output-plugin-with-codec-plain/114684/3 "2018-01-09T13:50:02Z")

</div>

thanks you for your quick response!

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 9, 2018, 3:35pm UTC](https://discuss.elastic.co/t/unable-to-remove-host-field-when-using-kafka-output-plugin-with-codec-plain/114684/4 "2018-01-09T15:35:10Z")

</div>

Did it work?

---

<div class="post-metadata">

**Author:** ![berlin](https://avatars.discourse-cdn.com/v4/letter/b/41988e/32.png) [@berlin](https://discuss.elastic.co/u/berlin)\
**Post date:** [January 9, 2018, 4:15pm UTC](https://discuss.elastic.co/t/unable-to-remove-host-field-when-using-kafka-output-plugin-with-codec-plain/114684/5 "2018-01-09T16:15:33Z")

</div>

yes, thank you, may I post here one more question, how to add xml tag to a current xml message, or to open new topic question ?

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 9, 2018, 4:46pm UTC](https://discuss.elastic.co/t/unable-to-remove-host-field-when-using-kafka-output-plugin-with-codec-plain/114684/6 "2018-01-09T16:46:03Z")

</div>

Maybe this `codec => plain { format => "<xml>%{message}</xml>" }`??

---

<div class="post-metadata">

**Author:** ![berlin](https://avatars.discourse-cdn.com/v4/letter/b/41988e/32.png) [@berlin](https://discuss.elastic.co/u/berlin)\
**Post date:** [January 9, 2018, 7:29pm UTC](https://discuss.elastic.co/t/unable-to-remove-host-field-when-using-kafka-output-plugin-with-codec-plain/114684/7 "2018-01-09T19:29:19Z")

</div>

I would like to add the below tags under the `<NotificationMetaData>` element in my current message:

```
    <Timestamp>XXXXXX</Timestamp>
	<EventId>XXXX</EventId>

```

For example the current message:

```
<Notification>
<NotificationMetaData>
	<NotificationType>test</NotificationType>
</NotificationMetaData>
</Notification>

```

The result should be:

```
<Notification>
<NotificationMetaData>
	<Timestamp>XXXXXX</Timestamp>
	<EventId>XXXX</EventId>
	<NotificationType>test</NotificationType>
</NotificationMetaData>
</Notification>

```

Any suggestion ?

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 10, 2018, 9:19am UTC](https://discuss.elastic.co/t/unable-to-remove-host-field-when-using-kafka-output-plugin-with-codec-plain/114684/8 "2018-01-10T09:19:02Z")

</div>

To actually edit the XML, well that is hard because we don't have an xml codec.  
If we did, then the workflow would be `decode xml at the input -> add fields to the event -> encode xml at the output.`

I think you will have to experiment with `mutate gsub`  
e.g.

```auto
  mutate {
    gsub => [
      "message", "<NotificationType>", "<Timestamp>%{[@timestamp]}</Timestamp><EventId>%{[event_id]}</EventId><NotificationType>"
    ]
  }

```

See [docs](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-gsub)

NOTE: It will be hard to get the indentation and newlines to look "pretty" but if its important you can try but it will be valid xml if you don't.

The second argument in the gsub array is a regex so you need to follow the regex escaping rules, e.g. if you wanted to say add the new elements after the closing tag `</NotificationType>` then the regex wold need to escape the forward slash - `<\/NotificationType>`  
Like so:

```auto
  mutate {
    gsub => [
      "message", "<\/NotificationType>", "</NotificationType><Timestamp>%{[@timestamp]}</Timestamp><EventId>%{[event_id]}</EventId>"
    ]
  }

```

---

<div class="post-metadata">

**Author:** ![berlin](https://avatars.discourse-cdn.com/v4/letter/b/41988e/32.png) [@berlin](https://discuss.elastic.co/u/berlin)\
**Post date:** [January 10, 2018, 10:30pm UTC](https://discuss.elastic.co/t/unable-to-remove-host-field-when-using-kafka-output-plugin-with-codec-plain/114684/9 "2018-01-10T22:30:02Z")

</div>

Thanks for all your help, much appreciated.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 7, 2018, 10:30pm UTC](https://discuss.elastic.co/t/unable-to-remove-host-field-when-using-kafka-output-plugin-with-codec-plain/114684/10 "2018-02-07T22:30:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.