# Unable to push incremental data from Cassandra DB to daily index using logstash

**URL:** <https://discuss.elastic.co/t/unable-to-push-incremental-data-from-cassandra-db-to-daily-index-using-logstash/166324>\
**Category:** Logstash\
**Created:** [January 30, 2019, 9:36am UTC](https://discuss.elastic.co/t/unable-to-push-incremental-data-from-cassandra-db-to-daily-index-using-logstash/166324 "2019-01-30T09:36:21Z")\
**Posts on this page:** 6\
**Page:** 2

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [February 6, 2019, 1:38pm UTC](https://discuss.elastic.co/t/unable-to-push-incremental-data-from-cassandra-db-to-daily-index-using-logstash/166324/22 "2019-02-06T13:38:02Z")

</div>

Nice!

It looks like it is working to me.

The time\_stamp value from the previous run is being insert into the next statement by substitution.

Now you can increase the limit to 5 `LIMIT 5` and let us verify that the fifth time\_stamp is used in the statement on the next scheduled run.

Please verify that the contents of `/opt/elk/.logstash_jdbc_last_run` is being updated after each scheduled run.

At this point you should check to ensure that the results are ordered by `time_stamp` ascending, this is important to guarantee a continually increasing `time_stamp` value (the assumption of the `>` condition).

---

<div class="post-metadata">

**Author:** ![Sapna](https://avatars.discourse-cdn.com/v4/letter/s/ee59a6/32.png) [@Sapna](https://discuss.elastic.co/u/Sapna)\
**Post date:** [February 7, 2019, 2:10pm UTC](https://discuss.elastic.co/t/unable-to-push-incremental-data-from-cassandra-db-to-daily-index-using-logstash/166324/23 "2019-02-07T14:10:16Z")

</div>

Hello,

Do we have any option to sort date time in same logstash conf file??  
Cassandra sorting does not work for me with error order by and IN can not work together:  
select \* from table where host\_name IN ('_ **','** _\*') AND time\_stamp \> : sql\_last\_value ORDER BY time\_stamp ASC LIMIT 5 Allow filtering;

If I dont use host\_name where condition , it throws an error :  
Order By only supported when partition key is restricted by EQ or IN.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [February 9, 2019, 11:28pm UTC](https://discuss.elastic.co/t/unable-to-push-incremental-data-from-cassandra-db-to-daily-index-using-logstash/166324/24 "2019-02-09T23:28:57Z")

</div>

Ordering within Logstash is not possible for a few reasons:

1. By design, each event is treated as autonomous. The pipeline does not care whether is sees 1 event or 600 billion events and especially does not attempt to reason (hold state) about what happened before any single event.
2. The pipeline is multi-threaded, out of order execution is the norm and one worker (thread) is assumed do its work sooner or later than another worker starting at about the same time.

---

<div class="post-metadata">

**Author:** ![Sapna](https://avatars.discourse-cdn.com/v4/letter/s/ee59a6/32.png) [@Sapna](https://discuss.elastic.co/u/Sapna)\
**Post date:** [February 11, 2019, 5:31am UTC](https://discuss.elastic.co/t/unable-to-push-incremental-data-from-cassandra-db-to-daily-index-using-logstash/166324/25 "2019-02-11T05:31:39Z")

</div>

Thanks.

In case my ordering by issue doesnt fix, will it push same records again which are already indexed?  
or it will remember about the records are already processed.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [February 12, 2019, 10:42am UTC](https://discuss.elastic.co/t/unable-to-push-incremental-data-from-cassandra-db-to-daily-index-using-logstash/166324/26 "2019-02-12T10:42:23Z")

</div>

There is no inherent remembering of ids.

This blog post explains your options regarding duplicates.

> **[Little Logstash Lessons: Handling Duplicates](https://www.elastic.co/blog/logstash-lessons-handling-duplicates)**
>
> Approaches for de-duplicating data in Elasticsearch using Logstash. We also go into examples of how you can use IDs in Elasticsearch Output.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 12, 2019, 10:42am UTC](https://discuss.elastic.co/t/unable-to-push-incremental-data-from-cassandra-db-to-daily-index-using-logstash/166324/27 "2019-03-12T10:42:24Z")

</div>

Closed on March 12, 2019, 10:42am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/unable-to-push-incremental-data-from-cassandra-db-to-daily-index-using-logstash/166324.md?page=1)
