Where is my _rev field

Elastic Stack Elasticsearch
1

Hi,

I am using logstash to put my data from couchdb into elasticsearch. I just noticed that my _rev field in not ported to elasticsearch. Do I need to do something special to have that in elasticsearch. Prompt help will be appreciated.

Thanks,
imad.

2

Can someone please help me with this?

3

ES will simply store whatever is passed, if it's not there then chances are it's not being extracted from couchbase.

4

Yes I know that. Is there anything in logstash config file that I can set so that _rev field also goes to my ES indices. I only have input and output sections in my logstash config file.
Here is my input in logstash config file:

couchdb_changes {
    db => "users"
    sequence_path => "seq_files\users_couchdb_seq"
    tags => ["users"]
}

and here is my output:

if "users" in [tags] {
   elasticsearch { 
    document_id => "%{[@metadata][_id]}"
    index => "users_index"
    hosts => ["127.0.0.1:9200"]
}
5

As far as I recall, user defined fields are not allowed to start with an underscore, as this indicates an Elasticsearch internal field. Can you try renaming it in Logstash and see if that makes a difference?

6

@Christian_Dahlqvist, _id from couchdb documents was also put in my ES indices as id in _source field of ES index
automatically then why _rev is omitted.

Anyways I added this to my logstash config file:
filter{
mutate {
convert => { "_rev" => "rev" }
}
}
but after that none of the indices were getting created at all.

PS: _rev is a default/intrinsic field for every document in couchdb like _id.

7

Now tried this:

filter{
mutate {
   rename => { "_rev" => "rev" }
}
}

but that also didn't put rev field in elasticserch indices.
Could it be that logstash coucdb plugin is not handing over _rev field to logstash to put it in ES indices?
It should be a very basic question for logstash/elasticsearch guys. I'll really appreciate a prompt solution as I am close to release, thanks!

8

Yes, so that is what you need to check, maybe with a stdout.

9

But why would it do that since _id is also coming through.
What would be the solution of this?

10

Ok I will spell it out.

If it's not in LS then either it's not coming from CB, or it is and LS is not carrying it over for some reason, or something else. You need to get visibility into the pipeline and you should start by adding {stdout {codec => rubydebug}} to the output section of your LS config, then see what is actually coming through.
From there, you can make your next move.

11

In stdout, I didn't see _rev field coming through. Any idea what could be the problem then?

can you please clarify if this is the default behavior of logstash/logstash couchdb plugin to omit _rev since I am not doing anything non regular to get data from couchdb.

12

Righto then, start super basic. Have the CB input, no filters, then stdout and see if it's there

13

did that, here is my config file:

 input { 
     couchdb_changes {
     db => "folder"
     tags => ["folder"]
   }
}
output {
    if "folder" in [tags] {
     stdout { codec => rubydebug }
  }
}

But still didn't see _rev field.

14

You aren't using the keep_revision flag in your input. Sending _rev is not always desired, so it is omitted by default, unless this flat is set to true.

15

@theuntergeek thanks for the help.
Just want to confirm that keep_revision only includes _rev field in index or does it maintain all revisions in ES index. For example, if a document A is inserted and then updated in couchDB, would ES index has only one copy of document A with correct/latest _rev, or would it save two copies of document one with insert _rev and the 2nd with update _rev?
In other words is there any side effect of setting keep_revision to true, since its false by default?

16

Elasticsearch won't keep revisions of a document, only the most recent you indexed.

Using the keep_revision flag will simply keep the _rev field in the indexed document so you can verify which version Elasticsearch has.

17

@theuntergeek you are awesome, thanks for the help!

18

if you don't mind can you also suggest a solution for another problem I am stuck with. You can find it here: New java.exe process starts everytime I stop and run logstash

Thanks alot!