# Logstash kafka input plugin doesn't pull data

**URL:** https://discuss.elastic.co/t/logstash-kafka-input-plugin-doesnt-pull-data/88589
**Category:** Logstash
**Created:** [June 7, 2017, 1:03pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-plugin-doesnt-pull-data/88589 "2017-06-07T13:03:06Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![guy\_es](https://avatars.discourse-cdn.com/v4/letter/g/67e7ee/32.png) [@guy\_es](https://discuss.elastic.co/u/guy_es)
#### Post date: [June 7, 2017, 1:03pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-plugin-doesnt-pull-data/88589/1 "2017-06-07T13:03:06Z")

</div>

Hi all,

I have an ubuntu machine with the following containers and settings, but logstash doesn't pull events from kafka at all, using the same configuration on a machine witch the services install directly on (not docker images) works fine.

Your advice will be much appreciated!

Thanks

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES  
3c579260d187 logstash:2.4.0 "/docker-entrypoin..." 34 minutes ago Up 14 minutes logstash  
b4529a0f2a41 confluentinc/cp-kafka:3.2.1 "/etc/confluent/do..." 44 minutes ago Up 14 minutes 0.0.0.0:9092-\>9092/tcp kafka  
4bc859aaae5e confluentinc/cp-zookeeper:3.2.1 "/etc/confluent/do..." 2 hours ago Up 14 minutes 2888/tcp, 0.0.0.0:2181-\>2181/tcp, 3888/tcp zookeeper  
8e6cba89c688 elasticsearch:2.4.3 "/docker-entrypoin..." 2 hours ago Up 14 minutes 0.0.0.0:9200-\>9200/tcp, 0.0.0.0:9300-\>9300/tcp elasticsearch

* * *

docker-compose:

version: '2'

services:  
aio-elasticsearch:  
container\_name: "elasticsearch"  
image: elasticsearch:2.4.3  
ports:  
- "9200:9200"  
- "9300:9300"  
networks:  
- aionet  
restart: unless-stopped

aio-logstash:  
container\_name: "logstash"  
image: logstash:2.4.0  
volumes: ["./logstash.conf:/opt/logstash/logstash.conf:ro"]  
command: -f /opt/logstash/logstash.conf --debug  
links:  
- aio-zk  
- aio-kafka  
networks:  
- aionet  
restart: unless-stopped

aio-zk:  
image: "confluentinc/cp-zookeeper:3.2.1"  
container\_name: zookeeper  
ports:  
- "2181:2181"  
networks:  
- aionet  
restart: unless-stopped  
environment:  
ZOOKEEPER\_CLIENT\_PORT: 2181  
ZOOKEEPER\_TICK\_TIME: 2000  
ZOOKEEPER\_SYNC\_LIMIT: 2

aio-kafka:  
image: "confluentinc/cp-kafka:3.2.1"  
container\_name: kafka  
ports:  
- "9092:9092"  
networks:  
- aionet  
restart: unless-stopped  
environment:  
KAFKA\_ZOOKEEPER\_CONNECT: 'zookeeper:2181'  
KAFKA\_ADVERTISED\_LISTENERS: 'PLAINTEXT://0.0.0.0:9092'  
KAFKA\_BROKER\_ID: 2

# \>\>\> ADD MORE DOCKERIZED CORE SERVICES HERE \<\<\<

networks:

# The network over which all Docker services will communicate

aionet:  
driver: bridge

* * *

root@3c579260d187:/# /opt/logstash/bin/logstash-plugin list --verbose logstash-input-kafka  
logstash-input-kafka (5.1.6)

* * *

input {

```
kafka {
    topics => ['test']
    group_id => 'test_1'
    bootstrap_servers => 'kafka:9092'
}

```

}  
filter {  
date {  
match =\> ["timestamp", "ISO8601"]  
}  
metrics {  
meter =\> "events"  
add\_tag =\> "metric"  
}  
mutate {  
rename =\> { "\_id" =\> "doc\_id" }  
}  
}

output {  
# only emit events with the 'metric' tag  
if "metric" in [tags] {  
stdout {  
codec =\> line {  
format =\> "rate: %{[events][rate\_1m]}, count: %{[events][count]}"  
}  
}  
}

```
else {
    elasticsearch {
        hosts => ["elasticsearch"]
        index => "events-%{+YYYY-MM-dd}-v%{schema_version}"
        routing => "%{org_pk}"
        document_id => "%{doc_id}"
        flush_size => "100"
        idle_flush_time => "1"
        document_type => "c_levent"
    }

    stdout { codec => rubydebug }

}

```

}

---

<div class="post-metadata">

### Author: ![OPMism](https://avatars.discourse-cdn.com/v4/letter/o/848f3c/32.png) [@OPMism](https://discuss.elastic.co/u/OPMism)
#### Post date: [June 29, 2017, 6:46am UTC](https://discuss.elastic.co/t/logstash-kafka-input-plugin-doesnt-pull-data/88589/2 "2017-06-29T06:46:41Z")

</div>

Try setting up this property in docker-compose.yml.  
KAFKA\_ADVERTISED\_HOST\_NAME

This should point to the docker IP & not localhost.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 27, 2017, 6:47am UTC](https://discuss.elastic.co/t/logstash-kafka-input-plugin-doesnt-pull-data/88589/3 "2017-07-27T06:47:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.