# Logstash not sending data to elastic from kafka
**URL:** https://discuss.elastic.co/t/logstash-not-sending-data-to-elastic-from-kafka/78866
**Category:** Logstash
**Created:** [March 16, 2017, 1:09pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elastic-from-kafka/78866 "2017-03-16T13:09:54Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Hussain\_Kothari](https://discuss.elastic.co/u/Hussain_Kothari)
#### Post date: [March 16, 2017, 1:09pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elastic-from-kafka/78866/1 "2017-03-16T13:09:54Z")
Hi,
im trying to add data in elastic with logstash and data is input from kafka
this is the simple config im using on logstash server in conf file /etc/logstash/conf.d/1-config.conf
input{
kafka{
bootstrap\_servers =\> "dummykafka:9092"
topics =\> ["dummy.kafka.topic"]
}
}
output{
elasticsearch{
index =\> "test.winlogbeat.eventlogs-%{+YYYY.MM.dd}"
hosts =\> ["elasticdatanode:50000"]
user =\> "AdminID"
password =\> "dummypassword"
}
}
i restart the logstash server with this command sudo initctl restart logstash
after restart of logstash data should be added to elastic, but it doesnt. data is present in kafka
Can you please help? or how do i enable logging in logstash?
---
### Author:  [@Hussain\_Kothari](https://discuss.elastic.co/u/Hussain_Kothari)
#### Post date: [March 16, 2017, 2:52pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elastic-from-kafka/78866/2 "2017-03-16T14:52:16Z")
So i found the logstash logs. But i don't understand what this error means.
[2017-03-16T09:46:43,950][ERROR][logstash.agent] Pipeline aborted due to error {:exception=\>#\, :backtrace=\>["/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:440:in `check_password'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:512:in`password='", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:246:in `normalize_url'", "org/jruby/RubyArray.java:2414:in`map'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:256:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:66:in`initialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http\_client.rb:183:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:35:in`initialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http\_client\_builder.rb:57:in `build'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch.rb:196:in`build\_client'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:13:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:8:in`register'", "/usr/share/logstash/logstash-core/lib/logstash/output\_delegator.rb:37:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:229:in`start\_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:229:in`start\_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:183:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:292:in`start\_pipeline'"]}
---
### Author:  [@Hussain\_Kothari](https://discuss.elastic.co/u/Hussain_Kothari)
#### Post date: [March 16, 2017, 3:47pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elastic-from-kafka/78866/3 "2017-03-16T15:47:22Z")
Hey i found the Answer to this in another post
> [@Pipeline aborted due to error in password field logstash](https://discuss.elastic.co/t/pipeline-aborted-due-to-error-in-password-field-logstash/68975/3):
>
> Hi Nick, here is my config elasticsearch { hosts =\> ["[http://x.x.x.x:9200](http://x.x.x.x:9200)"] manage\_template =\> false template\_name =\> "network-\*" index =\> "network-%{+YYYY.MM.dd}" document\_type =\> "%{type}" document\_id =\> "%{fingerprint}" user =\> "xxxx" password =\> "xxxSys201^" }
the password needs to be encoded.
Thanks 🙂
---
### Author:  [@Hussain\_Kothari](https://discuss.elastic.co/u/Hussain_Kothari)
#### Post date: [March 16, 2017, 3:59pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elastic-from-kafka/78866/4 "2017-03-16T15:59:25Z")
But now i have another issue.
Logstash logs (below) show no error after i restart logstash, but it does not create indexes in elastic. The config is the same as above.
Is there something I'm missing? Do we have to create some Templates in elastic for indexes or is it created by default by Logstash??
[2017-03-16T10:39:49,975][WARN][logstash.runner] SIGTERM received. Shutting down the agent.
[2017-03-16T10:39:49,987][WARN][logstash.agent] stopping pipeline {:id=\>"main"}
[2017-03-16T10:40:12,925][INFO][org.apache.kafka.clients.consumer.ConsumerConfig] ConsumerConfig values:
metric.reporters = []
[metadata.max.age.ms](http://metadata.max.age.ms) = 300000
partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor]
[reconnect.backoff.ms](http://reconnect.backoff.ms) = 50
sasl.kerberos.ticket.renew.window.factor = 0.8
max.partition.fetch.bytes = 1048576
bootstrap.servers = [dummykafka:9092]
ssl.keystore.type = JKS
enable.auto.commit = true
sasl.mechanism = GSSAPI
interceptor.classes = null
exclude.internal.topics = true
ssl.truststore.password = null
[client.id](http://client.id) = logstash
ssl.endpoint.identification.algorithm = null
max.poll.records = 2147483647
check.crcs = true
[request.timeout.ms](http://request.timeout.ms) = 40000
[heartbeat.interval.ms](http://heartbeat.interval.ms) = 3000
[auto.commit.interval.ms](http://auto.commit.interval.ms) = 5000
receive.buffer.bytes = 65536
ssl.truststore.type = JKS
ssl.truststore.location = null
ssl.keystore.password = null
fetch.min.bytes = 1
send.buffer.bytes = 131072
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
[group.id](http://group.id) = logstash
[retry.backoff.ms](http://retry.backoff.ms) = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
[sasl.kerberos.service.name](http://sasl.kerberos.service.name) = null
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.trustmanager.algorithm = PKIX
ssl.key.password = null
[fetch.max.wait.ms](http://fetch.max.wait.ms) = 500
sasl.kerberos.min.time.before.relogin = 60000
[connections.max.idle.ms](http://connections.max.idle.ms) = 540000
[session.timeout.ms](http://session.timeout.ms) = 30000
metrics.num.samples = 2
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
ssl.protocol = TLS
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.keystore.location = null
ssl.cipher.suites = null
security.protocol = PLAINTEXT
ssl.keymanager.algorithm = SunX509
[metrics.sample.window.ms](http://metrics.sample.window.ms) = 30000
auto.offset.reset = latest
[2017-03-16T10:40:13,017][INFO][org.apache.kafka.common.utils.AppInfoParser] Kafka version : 0.10.0.1
[2017-03-16T10:40:13,017][INFO][org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : a7a17cere4fda6c5
[2017-03-16T10:40:13,200][INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] Discovered coordinator dummykafka:9092 (id: 2147483645 rack: null) for group logstash.
[2017-03-16T10:40:13,209][INFO][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] Revoking previously assigned partitions [] for group logstash
[2017-03-16T10:40:13,210][INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] (Re-)joining group logstash
[2017-03-16T10:40:13,226][INFO][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] Successfully joined group logstash with generation 1
[2017-03-16T10:40:13,227][INFO][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] Setting newly assigned partitions [test.kafka.topic] for group logstash
[2017-03-16T10:40:13,327][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>["http://~hidden~:~hidden~@elasticnode:50000"]}}
[2017-03-16T10:40:13,329][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:url=\>#\, :healthcheck\_path=\>"/"}
[2017-03-16T10:40:13,566][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>#\}
[2017-03-16T10:40:13,572][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil}
[2017-03-16T10:40:13,693][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>50001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "norms"=\>false}, "dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword"}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "include\_in\_all"=\>false}, "@version"=\>{"type"=\>"keyword", "include\_in\_all"=\>false}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}
[2017-03-16T10:40:13,714][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["elasticnode:50000"]}
[2017-03-16T10:40:13,721][INFO][logstash.pipeline] Starting pipeline {"id"=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>500}
[2017-03-16T10:40:13,785][INFO][logstash.pipeline] Pipeline main started
[2017-03-16T10:40:13,884][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 13, 2017, 3:59pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elastic-from-kafka/78866/5 "2017-04-13T15:59:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.