# Logstash (ELK) not streaming data real-time from dynamodb, duplicates upon restart & fetching in some random order

**URL:** <https://discuss.elastic.co/t/logstash-elk-not-streaming-data-real-time-from-dynamodb-duplicates-upon-restart-fetching-in-some-random-order/66022>\
**Category:** Logstash\
**Created:** [November 14, 2016, 8:43pm UTC](https://discuss.elastic.co/t/logstash-elk-not-streaming-data-real-time-from-dynamodb-duplicates-upon-restart-fetching-in-some-random-order/66022 "2016-11-14T20:43:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ameya.aloni](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@ameya.aloni](https://discuss.elastic.co/u/ameya.aloni)\
**Post date:** [November 14, 2016, 8:43pm UTC](https://discuss.elastic.co/t/logstash-elk-not-streaming-data-real-time-from-dynamodb-duplicates-upon-restart-fetching-in-some-random-order/66022/1 "2016-11-14T20:43:07Z")

</div>

We're facing multiple issues while using ELK stack. We suspect they're Logstash Configuration issues. Issues are as follows:

1. Logstash connected to Dynamodb streams isn't showing real-time changes. We even have an explicit `perform_stream=>true` in our Logstash configuration. Note: We do get the latest data if we restart the logstash (which is running in a docker container). Could this be cross-region issue? Dynamodb (in us-east-1) while Logstash & Elasticsearch (in us-west-1)?

2. Upon restarting Logstash the entire Dynamodb table data is presumably duplicated in Elasticsearch. Dynamodb has around 70K+ Item Count while Elasticsearch has more than double Searchable Documents. Could it be because we have `perform_stream=>true` config?

3. Intermittently the latest data can be seen but it is sandwiched between older records; some kind of random data fetch order. Could it be due to multiple workers trying to log at the same time?

4. We need the json message contents from Dynamodb as is. However, we noticed that when we run Logstash the output shows the data in "Stream Records". When we use `log_format=>"json_binary_as_text"`, we can see the json message as we require. Is this sufficient?

Following is our Logstash Configuration:

> ```
> input { 
> dynamodb {
> endpoint => "dynamodb.us-east-1.amazonaws.com"
> streams_endpoint => "streams.dynamodb.us-east-1.amazonaws.com"
> view_type => "new_image"
> perform_scan => true
> perform_stream => true
> publish_metrics => true
> table_name => "here-we-have-dynamodb-table-name"
> log_format => "json_binary_as_text"
> }
> }
> output {
> elasticsearch {
> hosts => "here-we-have-our-elasticsearch-endpoint-which-is-in-us-west-1"
> } 
> }
> 
> ```

NOTE: There are _no_ errors in the logs (`docker logs --follow container-name`).  
Any help on these issues is really appreciated.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 15, 2016, 3:46am UTC](https://discuss.elastic.co/t/logstash-elk-not-streaming-data-real-time-from-dynamodb-duplicates-upon-restart-fetching-in-some-random-order/66022/2 "2016-11-15T03:46:39Z")

</div>

I have never used the dynamodb plugin, so can not help with that. As it is not one of the official plugins you may want to reach out to the creator.

The reason you are getting duplicates is that you are not specifying any [document\_id](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id) for the document in the Elasticsearch output. Elasticsearch therefore automatically generates a new id every time the document is processed and inserts it multiple times. If you have a primary key, use this as document\_id as this will cause Elasticsearch to update the record if you reprocess.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 13, 2016, 3:46am UTC](https://discuss.elastic.co/t/logstash-elk-not-streaming-data-real-time-from-dynamodb-duplicates-upon-restart-fetching-in-some-random-order/66022/3 "2016-12-13T03:46:45Z")

</div>

Closed on December 13, 2016, 3:46am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.