# Prevent logstash from adding additional data on output

**URL:** <https://discuss.elastic.co/t/prevent-logstash-from-adding-additional-data-on-output/307834>\
**Category:** Logstash\
**Created:** [June 22, 2022, 5:44am UTC](https://discuss.elastic.co/t/prevent-logstash-from-adding-additional-data-on-output/307834 "2022-06-22T05:44:49Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![mikhatanu](https://avatars.discourse-cdn.com/v4/letter/m/ba8739/32.png) [@mikhatanu](https://discuss.elastic.co/u/mikhatanu)\
**Post date:** [June 22, 2022, 5:44am UTC](https://discuss.elastic.co/t/prevent-logstash-from-adding-additional-data-on-output/307834/1 "2022-06-22T05:44:49Z")

</div>

Hello, i am using the kafka output and input plugins in logstash. The log route is like this: logstash -\> kafka -\> logstash (indexing) -\> elasticsearch. logstash encodes additional data (such as file path, hostname, etc)to the logs output. How do i prevent logstash from adding additional data (only the raw log sent to kafka)? or can i make the additional data follow my raw log separator "|"?

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [June 22, 2022, 7:27am UTC](https://discuss.elastic.co/t/prevent-logstash-from-adding-additional-data-on-output/307834/2 "2022-06-22T07:27:21Z")

</div>

Remove unnecessary fields:

`mutate{ remove_field => ["path", "hostname"] }`

If you like to add data, use `join`

---

<div class="post-metadata">

**Author:** ![mikhatanu](https://avatars.discourse-cdn.com/v4/letter/m/ba8739/32.png) [@mikhatanu](https://discuss.elastic.co/u/mikhatanu)\
**Post date:** [June 22, 2022, 10:44am UTC](https://discuss.elastic.co/t/prevent-logstash-from-adding-additional-data-on-output/307834/3 "2022-06-22T10:44:41Z")

</div>

should i add it in filter or output?

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [June 22, 2022, 12:45pm UTC](https://discuss.elastic.co/t/prevent-logstash-from-adding-additional-data-on-output/307834/4 "2022-06-22T12:45:33Z")

</div>

Only filter supports mutate.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 22, 2022, 12:55pm UTC](https://discuss.elastic.co/t/prevent-logstash-from-adding-additional-data-on-output/307834/5 "2022-06-22T12:55:16Z")

</div>

> [@mikhatanu](#):
>
> logstash -\> kafka -\> logstash (indexing) -\> elasticsearch.

What are the logstash inputs in this first logstash, the one that is sending to kafka?

If you want to send just the source message that this logstash is collecting, you could change the codec in the `kafka` output.

```auto
codec => plain { format => "%{message}" }

```

This will make logstash outputs to kafka only the message field, this will work assuming that you are not doing any filtering or transformation in this first logstash as it is probably this logstash that is adding the fields.

---

<div class="post-metadata">

**Author:** ![mikhatanu](https://avatars.discourse-cdn.com/v4/letter/m/ba8739/32.png) [@mikhatanu](https://discuss.elastic.co/u/mikhatanu)\
**Post date:** [June 23, 2022, 2:40am UTC](https://discuss.elastic.co/t/prevent-logstash-from-adding-additional-data-on-output/307834/6 "2022-06-23T02:40:47Z")

</div>

The data send to the first logstash is a raw data with a "|" seperator. i want the first logstash just sending the raw data to kafka. The one that indexes the log to elasticsearch will be at the logstash (indexing)

---

<div class="post-metadata">

**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\
**Post date:** [June 30, 2022, 6:11am UTC](https://discuss.elastic.co/t/prevent-logstash-from-adding-additional-data-on-output/307834/7 "2022-06-30T06:11:12Z")

</div>

Expanding on the answer by Rios -

In your `logstash.conf` file, you can do something like

```auto
input {
  kafka {
    bootstrap_servers => "<your_kafka_node_1:9093>,<kafka_node_2:9093>..."
    topics => ["your-topic"]
    #security_protocol => "SSL"
    #ssl_truststore_location => "path_to_truststore.jks"
    #ssl_truststore_password => "Top@Secret"
    client_id => "kafka_logstash_raw"
    group_id => "kafka_logstash_raw"
    consumer_threads => 2
    codec => "json"
  }
}

```

If `ssl` is enabled, you can `uncomment` the SSL related things.

`Consumer_threads` depends on the no of **partitions** in kafka topic.

```auto
filter {
  
  ## PARSE DATA ##
  
    ## GROK MESSAGE FIELD DATA
  grok {
    match => ["message", "(?m)%{TIMESTAMP_ISO8601:CreateDate} %{LOGLEVEL:Severity} \[%{NOTSPACE:Thread}\] %{JAVACLASS:ClassName} - \{%{GREEDYDATA:LogMsg}\}" ]
    tag_on_failure => ["_failure", "_grokparsefailure"]
  }
  
  ## IF NO FAILURES perform VALIDATIONS
  if "_failure" not in [tags] {
    date {
      # use timestamp from the log "ISO8601" "YYYY-MM-dd HH:mm:ss,SSS"
      match => ["CreateDate", "ISO8601", "YYYY-MM-dd HH:mm:ss,SSS", "YYYY-MM-dd HH:mm:ss"]
      target => "@timestamp"
      timezone => "UTC"
      tag_on_failure => ["_failure", "_timestampfailure"]
    }
	<here you can look to remove the fields>
	<and also split all your fields by delimiter "|">
 }
}

```

Hope this helps. You can modify the `grok` pattern to match your log line.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 28, 2022, 6:11am UTC](https://discuss.elastic.co/t/prevent-logstash-from-adding-additional-data-on-output/307834/8 "2022-07-28T06:11:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.