# No connection between Kafka and Logstash

**URL:** https://discuss.elastic.co/t/no-connection-between-kafka-and-logstash/130260
**Category:** Logstash
**Created:** [May 2, 2018, 2:07pm UTC](https://discuss.elastic.co/t/no-connection-between-kafka-and-logstash/130260 "2018-05-02T14:07:20Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![TheAnonymousModeIT](https://avatars.discourse-cdn.com/v4/letter/t/bc8723/32.png) [@TheAnonymousModeIT](https://discuss.elastic.co/u/TheAnonymousModeIT)
#### Post date: [May 2, 2018, 2:07pm UTC](https://discuss.elastic.co/t/no-connection-between-kafka-and-logstash/130260/1 "2018-05-02T14:07:20Z")

</div>

Hi,

I had a working configuration:

- Logstash Standard Input -\> Output to Elasticsearch -\> Kibana  
I started Logstash and entered some text (like "test1") in the standard input. Logstash wrote it into Elasticsearch and I could see it in Kibana.

Then I changed the configuration

- Logstash Kafka Input -\> Output to Elasticsearch -\> Kibana  
This does not work!

Docker-Compose-File:

```
  zookeeper:
    container_name: dev_zookeeper
    image: 31z4/zookeeper:latest
    restart: always
    hostname: zookeeper
    ports:
      - 2181:2181
    volumes:
      - "/data/zookeeper-data/:/usr/share/elasticsearch/data"

  # ************************************************************************** #
  # 1 of 3 brookers #

  kafka0:
    container_name: dev_kafka_node0
    image: myImage
    ports:
      - 9090:9091 # External Port / Internal Port  
    environment:
      ZOOKEEPER_IP: 192.168.194.134
      ZOOKEEPER_PORT: 2181
      KAFKA_BROOKER_ID: 0 # Unique Brooker ID
      KAFKA_LISTENERS_IP: localhost # Internal IP
      KAFKA_LISTENERS_PORT: 9091 # Internal Port
      KAFKA_ADVERTISED_LISTENERS_IP: 192.168.194.134 # External IP
      KAFKA_ADVERTISED_LISTENERS_PORT: 9090 # Extenral Port
    depends_on:
      - zookeeper

  # ************************************************************************* #
  # 1 of 3 nodes #

  elasticsearch0:
    container_name: dev_es_node0
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.4
    ports:
      - 9200:9200 # REST
      - 9300:9300 # Communication between nodes
    environment:
    - ES_JAVA_OPTS=-Xms512m -Xmx512m
    - cluster.name=devcluster
    - node.name=devnode0
    - node.master=true
    - node.data=true
    - discovery.zen.ping.unicast.hosts=dev_es_node0:9300,dev_es_node1:9301,dev_es_node2:9302
    volumes:

      logstash:
        container_name: dev_logstash
        image: docker.elastic.co/logstash/logstash:6.2.4
        ports:
        - 5000:5000
        - 5001:5001
        environment:
        - XPACK_MONITORING_ENABLED=false
        - ELASTICSEARCH_HOST=192.168.194.134
        - ELASTICSEARCH_PORT=9200
        volumes:    
         - ./logstash-pipeline/logstash.conf:/usr/share/logstash/pipeline/logstash.conf

  # ************************************************************************* #

```

logstash.conf

```
input{
  kafka{
    bootstrap_servers => "192.168.194.134:9090"
    topics => ["Test1","Test2","Test3"]
    auto_offset_reset => "earliest"
    codec => "json"
  }
}

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => ["192.168.194.134:9200"]
  }
}

```

Test:  
./kafka-topics.sh --zookeeper 192.168.194.134:2181   
--create --topic "Test1"   
--partitions 1   
--replication-factor 3

=\> The topic will be created (I can also list the topics), but nothing happens in Logstash.

 ![Running%20Logastash](https://us1.discourse-cdn.com/elastic/original/3X/2/5/2594bfcfd0a982fc3cb3e1d4c8a45e9900b3cf6d.png)

 ![Test](https://us1.discourse-cdn.com/elastic/original/3X/2/0/20c00a6e89d3a40cd0ff7c133cf6df4d55f832f9.png)

Thanks a lot for your help!

---

<div class="post-metadata">

### Author: ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [May 3, 2018, 12:31am UTC](https://discuss.elastic.co/t/no-connection-between-kafka-and-logstash/130260/2 "2018-05-03T00:31:32Z")

</div>

Did you put JSON-encoded events _into_ the kafka topics, or did you merely create the topics?

---

<div class="post-metadata">

### Author: ![TheAnonymousModeIT](https://avatars.discourse-cdn.com/v4/letter/t/bc8723/32.png) [@TheAnonymousModeIT](https://discuss.elastic.co/u/TheAnonymousModeIT)
#### Post date: [May 3, 2018, 6:21am UTC](https://discuss.elastic.co/t/no-connection-between-kafka-and-logstash/130260/3 "2018-05-03T06:21:36Z")

</div>

I tried today the following command but it hat no effect in logstash and after some time I get the following error message:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1a4d86826d47498eb782f905eb8fa18607a1a3b8.png)

There is no output in logstash docker container:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/9/19068c48c319710e8fe24a3ff8d2f44f37ee3a5f.png)

Independently of the json format. Logstash should print on the standard output something, shouldnt it?

But on the other hand I can execute kafka-kopics.sh without an error (but also without any output in logstash)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/7/47ca21a77295d256a19ad595166cab3816ab55db.png)

---

<div class="post-metadata">

### Author: ![TheAnonymousModeIT](https://avatars.discourse-cdn.com/v4/letter/t/bc8723/32.png) [@TheAnonymousModeIT](https://discuss.elastic.co/u/TheAnonymousModeIT)
#### Post date: [May 4, 2018, 12:02pm UTC](https://discuss.elastic.co/t/no-connection-between-kafka-and-logstash/130260/6 "2018-05-04T12:02:22Z")

</div>

That fixed the problem: KAFKA\_LISTENERS\_IP: 0.0.0.0

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 1, 2018, 12:02pm UTC](https://discuss.elastic.co/t/no-connection-between-kafka-and-logstash/130260/7 "2018-06-01T12:02:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
