# AmazonLogstashPlugin to AmazonElasticsearch documentId question

**URL:** <https://discuss.elastic.co/t/amazonlogstashplugin-to-amazonelasticsearch-documentid-question/34550>\
**Category:** Logstash\
**Created:** [November 13, 2015, 6:42pm UTC](https://discuss.elastic.co/t/amazonlogstashplugin-to-amazonelasticsearch-documentid-question/34550 "2015-11-13T18:42:33Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![skanjila](https://avatars.discourse-cdn.com/v4/letter/s/e19b73/32.png) [@skanjila](https://discuss.elastic.co/u/skanjila)\
**Post date:** [November 13, 2015, 6:42pm UTC](https://discuss.elastic.co/t/amazonlogstashplugin-to-amazonelasticsearch-documentid-question/34550/1 "2015-11-13T18:42:33Z")

</div>

Hello Folks,  
First time trying to do this logstash workflow so please bear with me, I am trying to setup the logstash dynamodb plugin ([http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.DynamoDBLogstash.html](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.DynamoDBLogstash.html)) to connect to an amazon hosted elasticsearch instance, however the thing I need to do on the output side is to use one of the fields in the source data set to create the document\_id for elasticsearch's index. So lets say the field in the input dynamodb stream is named foo and resides in table Bar, I have tried the following:

1)bin/logstash -e 'input {  
dynamodb{endpoint =\> "validendpoint"  
streams\_endpoint =\> "validstream"  
view\_type =\> "new\_and\_old\_images"  
table\_name =\> "Bar"} }  
output {  
amazon\_es {  
hosts =\> ["somehost"]  
region =\> "someregion"  
index =\> "valid index"  
document\_id =\> "%message[foo]"  
}  
stdout { } }

The above didn't work, I also tried message[0][foo] and that didn't work either, so 2 questions: 1) is it possible to do this without a filter in between the input and output 2) how do I access the foo field and use that as the document\_id for my amazon es index?

I have read many a site on this but the docs seem to be not great, any help would be much appreciated.

---

<div class="post-metadata">

**Author:** ![skanjila](https://avatars.discourse-cdn.com/v4/letter/s/e19b73/32.png) [@skanjila](https://discuss.elastic.co/u/skanjila)\
**Post date:** [November 15, 2015, 3:38pm UTC](https://discuss.elastic.co/t/amazonlogstashplugin-to-amazonelasticsearch-documentid-question/34550/2 "2015-11-15T15:38:59Z")

</div>

Ok, so I tried the following:

bin/logstash -e 'input {  
dynamodb{endpoint =\> "endpoint"  
streams\_endpoint =\> "stream"  
view\_type =\> "new\_and\_old\_images"  
table\_name =\> "MyTable"} }  
filter {  
ruby {  
code =\> "event['computed\_id']=event['fieldname']"  
}  
}  
output {  
amazon\_es {  
hosts =\> ["testhost"]  
region =\> "myregion"  
index =\> "myindex"  
document\_id =\> "%{computed\_id}"  
}  
stdout { } }'

Incidentally I am using logstash version 1.5.4

I get the following ruby exception:

Ruby exception occurred: undefined local variable or method `computed\_id' for #LogStash::Filters::Ruby:0x4612127f {:level=\>:error}

Is event not a valid array in this case, I noticed that dynamodb sends a string called message which contains the actual json message, I am wondering whether I need to do a JSON.parse on this and then extract the value of the field I need?

Can someone share with me a successful logstash config that sets the document\_id based on a value of particular field of the input coming in if you have it working, I'd just like to see some working example(s) that I can try to mimic and tweak?

Your input/help is much appreciated.  
Thanks

---

<div class="post-metadata">

**Author:** ![skanjila](https://avatars.discourse-cdn.com/v4/letter/s/e19b73/32.png) [@skanjila](https://discuss.elastic.co/u/skanjila)\
**Post date:** [November 16, 2015, 4:57pm UTC](https://discuss.elastic.co/t/amazonlogstashplugin-to-amazonelasticsearch-documentid-question/34550/3 "2015-11-16T16:57:23Z")

</div>

After many hours of hacking I finally figured it out, showing the logstash config that worked for me to help others:

bin/logstash -e 'input {  
dynamodb{endpoint =\> "someendpoint"  
streams\_endpoint =\> "somestream"  
view\_type =\> "new\_and\_old\_images"  
table\_name =\> "SomeTable"} }  
filter {  
json {  
source =\> "message"  
target =\> "doc"  
}  
}  
output {  
amazon\_es {  
hosts =\> ["somehost"]  
region =\> "someregion"  
index =\> "someindex"  
document\_id =\> "%{[doc][dynamodb][keys][fieldname][S]}"  
}  
stdout { } }'

The field that I was interested in is buried deep within the json hierarchy in the message and it seems that the only thing needed that is critical to this whole thing is the filter with the source and target storing a local temporary variable called doc which is then parsed in the output.

Regards

---

<div class="post-metadata">

**Author:** ![huytv593](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huytv593/32/13445_2.png) [@huytv593](https://discuss.elastic.co/u/huytv593)\
**Post date:** [November 28, 2016, 4:26am UTC](https://discuss.elastic.co/t/amazonlogstashplugin-to-amazonelasticsearch-documentid-question/34550/4 "2016-11-28T04:26:12Z")

</div>

Thanks for your suggestion. I have questions:

1. Can I set DynamoDB key as document\_id?
2. How about DELETE event, using document\_id can help us clear items have been deleted in DynamoDB on ES?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:30am UTC](https://discuss.elastic.co/t/amazonlogstashplugin-to-amazonelasticsearch-documentid-question/34550/5 "2017-07-06T04:30:13Z")

</div>