Config not working for logstash 2

Elastic Stack Logstash
1

Hi,

Can someone show me whats wrong with the config here. I am trying to convert my logstast 1.5.4 config to logstash 2. When I run it with --configtest flag, I get message 'Configuration is Ok'. and when I run the logstash agent with this config, index is not created. Any help will be greatly appreciated.

input {
couchdb_changes {
db => "roles"
hosts => ["127.0.0.1:5984"]
sequence_path => "seq_files\roles_couchdb_seq"
tags => ["roles"]
}
}
output {
if "roles" in [tags] {
elasticsearch {
document_id => "%{[@metadata][_id]}"
hosts => ["127.0.0.1:9200"]
index => "roles_index"
}
}
}

Thanks,
Imad.

2

Is there anything in the Logstash logs? What if you turn up the logging verbosity? What if you replace the elasticsearch output with a stdout output?

3

Here is the log file: https://gist.github.com/imadulhaque/f2cc28f7b0f6c46d2c78#file-1122015logstash_log-txt.

PS: I have updated log file on github.

4

Partial quote of log:

{:timestamp=>"2015-11-02T09:38:52.904000-0500", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::CouchDBChanges db=>"roles", host=>"127.0.0.1:5984", sequence_path=>"seq_files\\roles_couchdb_seq", tags=>["roles"], codec=><LogStash::Codecs::Plain charset=>"UTF-8">, port=>5984, secure=>false, password=>, heartbeat=>1000, keep_revision=>false, ignore_attachments=>true, always_reconnect=>true, reconnect_delay=>10>\n Error: bad component(expected host component): 127.0.0.1:5984\n Exception: URI::InvalidComponentError\n

Did the acceptable values for the host parameter change between 1.5.x and 2.0? The documentation doesn't mention support for specifying the port number.

5

Please see the updated log on github. I am still getting the error with hosts => ["127.0.0.1:5984"].

6

AFAICT the parameter is named host and not hosts in Logstash 2.0. What if you use host and drop the port number?

7

Do you mean my config should be like this:
input {
couchdb_changes {
db => "roles"
host => "127.0.0.1"
sequence_path => "seq_files\roles_couchdb_seq"
tags => ["roles"]
}
}
output {
if "roles" in [tags] {
elasticsearch {
document_id => "%{[@metadata][_id]}"
hosts => ["127.0.0.1:9200"]
index => "roles_index"
}
}
}
how would it get data from couchdb if port number is dropped?

8

Do you mean my config should be like this:

Yes.

how would it get data from couchdb if port number is dropped?

There's a port option that defaults to 5984.

9

See https://www.elastic.co/guide/en/logstash/current/plugins-inputs-couchdb_changes.html#plugins-inputs-couchdb_changes-port

10

Ok that worked, many thanks for this.

I am seeing deleted documents as well in my indices. Would these be coming from couchdb Do I need to filter those in logstash config file. How can I do that?

11

"I am seeing deleted documents as well in my indices. Would these be coming from couchdb Do I need to filter those in logstash config file. How can I do that?"
will really appreciate any help, thanks.

12

There is/was an issue where the document source is being sent with a delete command (this should not be so). This causes deletes to not work correctly.

Please check to see the status of this at https://github.com/logstash-plugins/logstash-output-elasticsearch

(I'm at an airport and don't have good wifi, or I'd do the digging myself).