# Couchdb\_changes index only doc field from event
**URL:** https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223
**Category:** Logstash
**Created:** [January 14, 2016, 11:09am UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223 "2016-01-14T11:09:31Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@\_alexey\_](https://discuss.elastic.co/u/_alexey_)
#### Post date: [January 14, 2016, 11:09am UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/1 "2016-01-14T11:09:32Z")
Hello Logstash Community!
I want to upgrade elasticsearch to newew version and replace couchdb\_river with couchdb\_changes plugin
while working this input plugin create events with folowing fields ["doc","doc\_as\_upsert","@version","@timestamp"] and elasticsearch create documents with this fields, is there any possibility to replace whole event with doc field, or treat doc field as event root?
Thanks in advance!
---
### Author:  [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [January 14, 2016, 12:20pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/2 "2016-01-14T12:20:02Z")
Unfortunately, there is no way to remove those fields when sending through Logstash.
Logstash will always send `@timestamp` and `@version`. And without `doc_as_upsert`, elasticsearch won't create any "new" docs, as the plugin only sends updates.
---
### Author:  [@\_alexey\_](https://discuss.elastic.co/u/_alexey_)
#### Post date: [January 14, 2016, 12:23pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/3 "2016-01-14T12:23:03Z")
when i query elasticsearch \_source:{doc:{},doc\_as\_upsert:{}}. with couchdb\_river in source i only get doc field contents
---
### Author:  [@\_alexey\_](https://discuss.elastic.co/u/_alexey_)
#### Post date: [January 14, 2016, 12:25pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/4 "2016-01-14T12:25:15Z")
the problem is that i don`t need those fields and I want elasticsearch output to index only doc contents
---
### Author:  [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [January 14, 2016, 12:53pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/5 "2016-01-14T12:53:09Z")
That's correct. `doc_as_upsert` exits Logstash, but is only a flag to Elasticsearch. It wouldn't be in the final, indexed document.
You'll have to roll your own solution if you want something different from what Logstash offers. Logstash cannot _not_ send `@timestamp` and `@version`.
---
### Author:  [@\_alexey\_](https://discuss.elastic.co/u/_alexey_)
#### Post date: [January 14, 2016, 1:05pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/6 "2016-01-14T13:05:58Z")
For what elasticsearch?
for logstash output plugin or for ES?
logstash-elasticsearch-output receives event =\> {doc,doc\_as\_upsert,@timestamp,@version} and pass it **as is** to ES for indexing! So in ES index there are documents with \_source: {doc:{},doc\_as\_upsert:true,@version:"",@timestamp:""}
The question is: Is this a correct behaviour or i am doing something wrong?
If this is correct behaviour why it is so?
---
### Author:  [@\_alexey\_](https://discuss.elastic.co/u/_alexey_)
#### Post date: [January 14, 2016, 2:44pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/7 "2016-01-14T14:44:52Z")
All I was trying to do is this
filter {
ruby {
code =\> "event['doc'].to\_hash.each{|k,v| event[k] = v}"
remove\_field =\> ["doc"]
}
}
Thanks can close this
---
### Author:  [@chrishb](https://discuss.elastic.co/u/chrishb)
#### Post date: [January 14, 2016, 3:20pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/8 "2016-01-14T15:20:58Z")
I am about to make the same change after updating to Elasticsearch 2.1.1 (moving from using the CouchDB river to using Logstash to ship data from CouchDB to Elasticsearch). Is there really no way (excluding the above filter) to omit the @version, @timestamp, and doc\_as\_upsert fields and index what is contained in the 'doc' field into Elasticsearch? The goal is to have the document in Elasticsearch exactly reflect its CouchDB counterpart.
---
### Author:  [@\_alexey\_](https://discuss.elastic.co/u/_alexey_)
#### Post date: [January 14, 2016, 3:39pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/9 "2016-01-14T15:39:41Z")
I was struggling with such behavour for two days. Searching around internet didn't help, all i can implement is this.... Speaking off @version @timestamp and doc\_as\_upsert i think elasticsearch output plugin should omit them while indexing document or you could possibly think of using elasticsearch scripting to remove those fields
---
### Author:  [@\_alexey\_](https://discuss.elastic.co/u/_alexey_)
#### Post date: [January 14, 2016, 3:52pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/10 "2016-01-14T15:52:19Z")
[couchdb+logstash+elastic](https://www.hoekstra.nu/wordpress/blog/2015/08/12/logstash-couchdb-elasticsearch-bending-your-rivers/)
---
### Author:  [@chrishb](https://discuss.elastic.co/u/chrishb)
#### Post date: [January 14, 2016, 4:04pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/11 "2016-01-14T16:04:55Z")
@theuntergeek , do you have any suggestions on where I can go from here?
---
### Author:  [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [January 14, 2016, 4:18pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/12 "2016-01-14T16:18:08Z")
This is what I get for commenting at 5am when my head is still fuzzy.
I am traveling right now. I'll take a look this evening after I land.
---
### Author:  [@chrishb](https://discuss.elastic.co/u/chrishb)
#### Post date: [January 14, 2016, 4:20pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/13 "2016-01-14T16:20:00Z")
Looking forward to it. Safe travels.
---
### Author:  [@\_alexey\_](https://discuss.elastic.co/u/_alexey_)
#### Post date: [January 14, 2016, 5:20pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/14 "2016-01-14T17:20:55Z")
@theuntergeek sorry if i offend you. Maybe we didn't understand each other. I really appreciate your help! And i am looking forward for your expertise.
---
### Author:  [@chrishb](https://discuss.elastic.co/u/chrishb)
#### Post date: [January 15, 2016, 2:30pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/15 "2016-01-15T14:30:54Z")
@_alexey_, I took a (very) quick look into this last night. The below block from 'logstash-input-couchdb\_changes' builds the event, assigns the document to the doc field and deletes the \_id and \_rev field from the document. This could be easily changed to achieve the desired result - you could add a flag to toggle on/off.
```
if line['doc']['_deleted']
hash['@metadata']['action'] = 'delete'
else
hash['doc'] = line['doc']
hash['@metadata']['action'] = 'update'
hash['doc'].delete('_id')
hash['doc_as_upsert'] = true
hash['doc'].delete('_rev') unless @keep_revision
end
```
Alternatively, building on your example above, you could alter the event to resemble the couch document with the addition of `add_field => { "_id" => "%{[@metadata][_id]}"}` to add the `_id` field back and use the keep\_revision on the input plugin. I.e:
```
input {
couchdb_changes {
...
keep_revision => true
}
}
filter{
ruby {
code => "event['doc'].to_hash.each{|k,v| event[k] = v}"
remove_field => ["doc"]
remove_field => ["doc_as_upsert"]
remove_field => ["@version"]
remove_field => ["@timestamp"]
add_field => { "_id" => "%{[@metadata][_id]}"}
}
}
```
---
### Author:  [@\_alexey\_](https://discuss.elastic.co/u/_alexey_)
#### Post date: [January 15, 2016, 2:37pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/16 "2016-01-15T14:37:07Z")
As I understand you will use this solution to migrate to es 2.1?
---
### Author:  [@chrishb](https://discuss.elastic.co/u/chrishb)
#### Post date: [January 15, 2016, 2:48pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/17 "2016-01-15T14:48:35Z")
Possibly. I'm just investigating what's required in updating to ES 2.x from 1.7.3. If there aren't any big gains from updating we will most likely stick with our current configuration (ES1.7.3 using river-couchdb).
---
### Author:  [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [January 15, 2016, 2:49pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/18 "2016-01-15T14:49:32Z")
Sorry for not responding last night. I think this is actually a regression in the Elasticsearch output since Logstash 1.5.0. It's not using the "doc" field when doing the upsert. More to follow when I'm out of training and can chase it down.
---
### Author:  [@\_alexey\_](https://discuss.elastic.co/u/_alexey_)
#### Post date: [January 15, 2016, 7:02pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/19 "2016-01-15T19:02:10Z")
I do not think there is regression within elasticsearch output plugin, it indexes what you pass to it, why should it bother that event has **doc** field **doc\_as\_upsert** field and everything else. In my opinion the problem is within couchdb\_changes input plugin. From developer(e.g. my) point of view event is couchdb document everything else should goes to @metadata field so from this build event definition
```auto
private
def build_event(line)
# In lieu of a codec, build the event here
line = LogStash::Json.load(line)
return nil if line.has_key?("last_seq")
hash = Hash.new
hash['@metadata'] = { '_id' => line['doc']['_id'] }
if line['doc']['_deleted']
hash['@metadata']['action'] = 'delete'
else
hash['doc'] = line['doc']
hash['@metadata']['action'] = 'update'
hash['doc'].delete('_id')
hash['doc_as_upsert'] = true
hash['doc'].delete('_rev') unless @keep_revision
end
hash['@metadata']['seq'] = line['seq']
event = LogStash::Event.new(hash)
@logger.debug("event", :event => event.to_hash_with_metadata) if @logger.debug?
event
end
```
we should go here
```auto
private
def build_event(line)
# In lieu of a codec, build the event here
line = LogStash::Json.load(line)
return nil if line.has_key?("last_seq")
hash['@metadata'] = { '_id' => line['doc']['_id'] }
if line['doc']['_deleted']
line['doc']['@metadata']{'action' => 'delete', '_id'=>line['doc']['_id'],'seq'=>line['seq']}
else
line['doc']['@metadata']{'action' => 'update', '_id'=>line['doc']['_id'],'doc_as_upsert'=>'true','seq'=>line['seq']}
end
event = LogStash::Event.new(line['doc'])
@logger.debug("event", :event => event.to_hash_with_metadata) if @logger.debug?
event
end
```
Changing build\_event like this we save couchdb document **as is** in elasticsearch index and we don`t even need doc\_as\_upsert and \_id fields in @metadata cause if action is not delete we should set elasticsearch output plugin config doc\_as\_upsert to true anyway and \_id we could use from event[\_id] cause every couchdb document has **\_id** and with this changes output section of logstash config could be like this:
```auto
output {
if "delete" in [@metadata][action] {
elasticsearch {
...
document_id => [_id]
action => "delete"
}
} else {
elasticsearch {
...
document_id=>[_id]
action => "update"
doc_as_upsert => 'true'
}
}
}
```
---
### Author:  [@\_alexey\_](https://discuss.elastic.co/u/_alexey_)
#### Post date: [January 15, 2016, 7:13pm UTC](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223/20 "2016-01-15T19:13:21Z")
even more build event could be like this
```auto
private
def build_event(line)
# In lieu of a codec, build the event here
line = LogStash::Json.load(line)
return nil if line.has_key?("last_seq")
event = LogStash::Event.new(line['doc'])
@logger.debug("event", :event => event.to_hash_with_metadata) if @logger.debug?
event
end
```
and config
```auto
output {
if [_deleted] {
elasticsearch {
...
document_id => [_id]
action => "delete"
}
} else {
elasticsearch {
...
document_id=>[_id]
action => "update"
doc_as_upsert => 'true'
}
}
}
```
[Next page](https://discuss.elastic.co/t/couchdb-changes-index-only-doc-field-from-event/39223.md?page=2)