Custom index not creating at ElasticSearch cluster

Elastic Stack Logstash
1

HI All,

I am using ES 1.5.2 with logstash 2.4.0 . i have the following configurations to pull data from kafka topic and index to ES.

input{
kafka {
zk_connect => ":2181"
group_id => "logstash"
topic_id => "test"
consumer_threads => 2
}
}
output {
elasticsearch {
hosts => ["10.10.1.2:9200"]
index => "test123"
}
stdout { codec => rubydebug }
}

I given test123 as my index name but custom index not creating at ES rather it is creating default "test" index and pushing data to that index only.please suggest me where i am doing wrong in above configurations.

Thanks
phani

2

Not sure what you mean. By default Logstash doesn't write to an index named "test", so if Logstash is sending events to an index with that name it's because you have configured it that way.

3

Thank you @magnusbaeck for quick reply i am trying to write documents to custom index named as "test123" but it is not creating at elasticsearch side.

phani

4

Are you reading events from Kafka then, i.e. is it an input or an output problem?

5

able to consume data from kafka but writing to es i am getting problem it is problem with output it seems.

6

Have a look in the Logstash logs.

7

@magnusbaeck i have looked at the logs but it seems logstash not generating logs when i start logstash with kafka configuration it is showing like this.

./logstash -f kafka-topic.conf
Settings: Default pipeline workers: 2
Pipeline main started

no logs hitting to logstash server.i have used .tar for logstash rather than service

please advise me.

8

Increase the log level by starting with --verbose or even --debug.

9

Hi @magnusbaeck,

Thank you for suggesting and i have checked using --verbose there i found like mapping i created loading or not . it seems the default logstash mapping is applying with document_type logs instead of my custom one.The following is the configuration i applied to get it work.

action => "index"
hosts => ["10.10.1.36:9200"]
index => "logs_test"
document_type => "lv2"
template => "/home/test_bkp.json"
template_name => "kafka_template"
template_overwrite => 'true'

Explicit document type , template_overwrite => 'true' did the trick for me and i am loading custom index template from template.

Now indexing data from kafka topic to es is working and how can we create monthly index using the same configurations please suggest me.

Thanks
phani