Deleting documents from couchdb doesn't delete it from ES index

Elastic Stack Logstash
1

I wondering if there is anything extra that I need do to delete a document from ES index when its deleted from my couchdb.

Here is how my config file looks like:

input {
couchdb_changes {
db => "users"
host => "127.0.0.1"
port => 5984
sequence_path => "seq_files\users_couchdb_seq"
tags => ["users"]
}
}
output {
if "users" in [tags] {
elasticsearch {
document_id => "%{[@metadata][_id]}"
host => "127.0.0.1"
index => "users_index"
protocol => "http"
host => '127.0.0.1'
port => 9200
}
}
}

Many thanks.

2

What version of Logstash are you using?

3

I am using 1.5.4 version.

4

I would run logstash with the --debug flag and capture the logs. I'd like to see them, if I could.

5

Actually whats happening is if data gets deleted by couchdb's normal delete call. Data seems to get deleted from index but the following entry still remains in the index corresponding to the item that got deleted in couchdb. How can I also have this deleted. Is there some sort of config setting to delete this as well.
{
"_index": "user_index",
"_type": "logs",
"_id": "b5a30ba6bcf908c1a3db884e68136d9e",
"_score": 1,
"_source": {
"@version": "1",
"@timestamp": "2015-09-23T19:40:01.172Z",
"tags": [
"sites"
]
}
}

But the problem is when the whole couchDB is deleted, data in its respective index in ES is not deleted. I uploaded part of log at: https://gist.github.com/imadulhaque/cd4cce8ea1356150ee75. Please help.

6

That is not the couchdb_changes plugin then, but probably this issue: https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/195

7

@theuntergeek any idea when this fix will be in? We are close to release now, this can really put us in difficult situation if we don't have this fixed anytime soon.

8

I can't say for sure. You could ask at that issue, and put some pressure there.

9

As far as the data you're showing is concerned:

{
  "index": "userindex",
  "_type": "logs",
  "_id": "b5a30ba6bcf908c1a3db884e68136d9e",
  "_score": 1,
  "_source": {
    "@version": "1",
    "@timestamp": "2015-09-23T19:40:01.172Z",
    "tags": [
      "sites"
    ]
  }
}

The couchdb_changes plugin creates an empty event on delete, and populates only the @metadata field with an action and an _id. Logstash requires that events have the @timestamp and @version fields, so the run method in the code has decorate(event), which applies those fields. Having tags => ['sites'] in your input puts that field there. I am making a presumption that these events are becoming documents in Elasticsearch because the output plugin is sending this data, instead of just the DELETE action and the associated document ID.