# Problem in sending data from HTTP plugin to kafka plugin in logstash

**URL:** <https://discuss.elastic.co/t/problem-in-sending-data-from-http-plugin-to-kafka-plugin-in-logstash/197110>\
**Category:** Logstash\
**Created:** [August 28, 2019, 12:17pm UTC](https://discuss.elastic.co/t/problem-in-sending-data-from-http-plugin-to-kafka-plugin-in-logstash/197110 "2019-08-28T12:17:39Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![DZ1](https://avatars.discourse-cdn.com/v4/letter/d/b487fb/32.png) [@DZ1](https://discuss.elastic.co/u/DZ1)\
**Post date:** [August 28, 2019, 12:17pm UTC](https://discuss.elastic.co/t/problem-in-sending-data-from-http-plugin-to-kafka-plugin-in-logstash/197110/1 "2019-08-28T12:17:39Z")

</div>

I am trying to send JSON data to HTTP plugin and that output to Kafka using Kafka plugin. In this HTTP input plugin shows output in the first attempt and sends only a small amount of data to Kafka. When I hit it next time there is no data in logs but that small fraction of data comes at Kafka topic.

```auto
input {
 http {
    host => "0.0.0.0"
    port => "5000"
    response_headers => {
      "Access-Control-Allow-Origin" => "*"
      "Content-Type" => "text/plain"
      "Access-Control-Allow-Headers" => "Origin, X-Requested-With, Content-Type,
       Accept"
    }
  }
}

output {
      stdout {
        codec => json
       }
      kafka {
        #codec => json
        bootstrap_servers => "kafka:9092"
        topic_id => "NotificationTopic"
     }
}

```

I am getting this in Kafka Topic:

```auto
{
      "@version" => "1",
    "@timestamp" => 2019-08-28T12:23:52.368Z,
       "message" => "2019-08-28T12:23:51.647Z 192.168.144.1 %{message}"
}

```

The HTTP input plugin should be accepting data at every hit and should be sent the same to the Kafka output plugin.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 28, 2019, 12:43pm UTC](https://discuss.elastic.co/t/problem-in-sending-data-from-http-plugin-to-kafka-plugin-in-logstash/197110/2 "2019-08-28T12:43:28Z")

</div>

If the http input knows from the content-type that the input body is json then it will automagically parse the JSON, so you will not have a [message] field. The default codec on the kafka output is plain, and what you are seeing there is the [default](https://github.com/logstash-plugins/logstash-codec-plain/blob/892c5a304e0d08985daf6d843db2dbbe566562c5/lib/logstash/codecs/plain.rb#L39) format (since you do not have a [message] field the sprintf reference to it does not get substituted).

You need to either force the codec on the input so that it does not parse the JSON, or add a codec to the output. json\_lines might work.

---

<div class="post-metadata">

**Author:** ![DZ1](https://avatars.discourse-cdn.com/v4/letter/d/b487fb/32.png) [@DZ1](https://discuss.elastic.co/u/DZ1)\
**Post date:** [August 28, 2019, 1:44pm UTC](https://discuss.elastic.co/t/problem-in-sending-data-from-http-plugin-to-kafka-plugin-in-logstash/197110/3 "2019-08-28T13:44:30Z")

</div>

@Badger Yes, you are right. I tried it with codec and it worked fine. Thanks for your suggestion.  
Here are my config that worked:

```auto
input {
 http {
    host => "0.0.0.0"
    port => "5000"
    codec => json
  }
}

filter {
  mutate {
      remove_field => ["host", "@version", "@timestamp", "headers"]
  }
}

output {
      stdout { }
      kafka {
        codec => json_lines
        bootstrap_servers => "metrics-kafka:9092"
        topic_id => "NotificationTopic"
     }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 25, 2019, 1:44pm UTC](https://discuss.elastic.co/t/problem-in-sending-data-from-http-plugin-to-kafka-plugin-in-logstash/197110/4 "2019-09-25T13:44:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.