Prevent logstash from adding additional data on output

Hello, i am using the kafka output and input plugins in logstash. The log route is like this: logstash -> kafka -> logstash (indexing) -> elasticsearch. logstash encodes additional data (such as file path, hostname, etc)to the logs output. How do i prevent logstash from adding additional data (only the raw log sent to kafka)? or can i make the additional data follow my raw log separator "|"?

Remove unnecessary fields:

mutate{ remove_field => [ "path", "hostname" ] }

If you like to add data, use join

should i add it in filter or output?

Only filter supports mutate.

What are the logstash inputs in this first logstash, the one that is sending to kafka?

If you want to send just the source message that this logstash is collecting, you could change the codec in the kafka output.

codec => plain { format => "%{message}" }

This will make logstash outputs to kafka only the message field, this will work assuming that you are not doing any filtering or transformation in this first logstash as it is probably this logstash that is adding the fields.

1 Like

The data send to the first logstash is a raw data with a "|" seperator. i want the first logstash just sending the raw data to kafka. The one that indexes the log to elasticsearch will be at the logstash (indexing)