Unable to remove %{host} field when using kafka output plugin with codec plain

[berlin]

Hi,
Please review the below configuration:

input {
  sqs {
    access_key_id => ""
    secret_access_key => ""
    polling_frequency => "1"
    queue => "xxx"
    region => "eu-west-1"
    threads => "1"
    codec => plain
  }
}

filter {
  mutate { rename => { "@timestamp" => "timestamp" } }
  mutate { remove_field => [ "@version", "path", "host", "type" ] }
}

output {
  kafka {
    bootstrap_servers=> "kafka:9092"
    topic_id=> "xxx"
    codec=> "plain"
  }
  stdout { codec => "rubydebug" }
}

The plugin does not remove the %{host} field, its just pass to kafka the field in this format "%{host}"

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic xxx --property print.key=true --property key.separator="-"

Kafka output: %{host} <some xml>

The messages does not contain %{host} when using json codec in kafka output plugin or when using stdout { codec => rubydebug }, but I dont want to use the json codec, I just want to keep the xml format from sqs to kafka, and I want logstash not to add the %{host} field, any suggestions ?

Thanks

[guyboertje]

The plain codec does this to encode the event for transmission.
encoded = @format ? event.sprintf(@format) : event.to_s

In your config, format is not used so event.to_s is invoked - but to_s is usually used for debugging and it adds the %{host} bit.

You should use the format setting. Presuming that the message field holds the xml and you don't need the timestamp, you can do:

    codec => plain { format => "%{message}" }

or if you want the timestamp

    codec => plain { format => "%{timestamp} -- %{message}" }

Use the fields as you saw them before with the rubydebug codec to experiment without the kafka output...

output {
  stdout { codec => plain { format => "%{timestamp} -- %{message}" } }
}

What you see in stdout is what you will get in kafka. When you are happy switch back to the kafka output.

[berlin]

thanks you for your quick response!

[guyboertje]

Did it work?

[berlin]

yes, thank you, may I post here one more question, how to add xml tag to a current xml message, or to open new topic question ?

[guyboertje]

Maybe this codec => plain { format => "<xml>%{message}</xml>" }??

[berlin]

I would like to add the below tags under the <NotificationMetaData> element in my current message:

    <Timestamp>XXXXXX</Timestamp>
	<EventId>XXXX</EventId>

For example the current message:

<Notification>
<NotificationMetaData>
	<NotificationType>test</NotificationType>
</NotificationMetaData>
</Notification>

The result should be:

<Notification>
<NotificationMetaData>
	<Timestamp>XXXXXX</Timestamp>
	<EventId>XXXX</EventId>
	<NotificationType>test</NotificationType>
</NotificationMetaData>
</Notification>

Any suggestion ?

[guyboertje]

To actually edit the XML, well that is hard because we don't have an xml codec.
If we did, then the workflow would be decode xml at the input -> add fields to the event -> encode xml at the output.

I think you will have to experiment with mutate gsub
e.g.

  mutate {
    gsub => [
      "message", "<NotificationType>", "<Timestamp>%{[@timestamp]}</Timestamp><EventId>%{[event_id]}</EventId><NotificationType>"
    ]
  }

See docs

NOTE: It will be hard to get the indentation and newlines to look "pretty" but if its important you can try but it will be valid xml if you don't.

The second argument in the gsub array is a regex so you need to follow the regex escaping rules, e.g. if you wanted to say add the new elements after the closing tag </NotificationType> then the regex wold need to escape the forward slash - <\/NotificationType>
Like so:

  mutate {
    gsub => [
      "message", "<\/NotificationType>", "</NotificationType><Timestamp>%{[@timestamp]}</Timestamp><EventId>%{[event_id]}</EventId>"
    ]
  }

[berlin]

Thanks for all your help, much appreciated.