# Data not getting synced properly from SQL to elastic

**URL:** <https://discuss.elastic.co/t/data-not-getting-synced-properly-from-sql-to-elastic/351742>\
**Category:** Logstash\
**Created:** [January 24, 2024, 4:53pm UTC](https://discuss.elastic.co/t/data-not-getting-synced-properly-from-sql-to-elastic/351742 "2024-01-24T16:53:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![abhishek\_agarwal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhishek_agarwal/32/131089_2.png) [@abhishek\_agarwal](https://discuss.elastic.co/u/abhishek_agarwal)\
**Post date:** [January 24, 2024, 4:53pm UTC](https://discuss.elastic.co/t/data-not-getting-synced-properly-from-sql-to-elastic/351742/1 "2024-01-24T16:53:19Z")

</div>

I am syncing data from sql to elastic and in filter in the code block I am appending to map eg features but what is happening is that when I am running the bulk sql query ,not all features are getting appended to each index. eg if index 1 has 10 features then 2 or 3 are getting ingested to elastic but when I am directly specifying the id in = in sql query then all 10 features are getting synced. But the bulk query if I am running separating then also it gives the correct count.

What could be the potential issue? Tried multple things but my map is not getting updated to full features for each hotelIds.

```auto
filter {
    aggregate {
      task_id => "%{HotelCode}"
      code => "
        map['HotelCode'] ||= event.get('HotelCode')

        map['Features']||= []  

        map['Features'] << {

            'FeatureId' => event.get('FeatureId'),
            'FeatureIdName' => event.get('FeatureIdName'),
            'Group' => event.get('GroupName'),
            'Rank' => event.get('Rank')
        }

        event.cancel()
       "
      push_previous_map_as_event => true
      timeout => 5
    }
  }

```

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [January 24, 2024, 4:58pm UTC](https://discuss.elastic.co/t/data-not-getting-synced-properly-from-sql-to-elastic/351742/2 "2024-01-24T16:58:22Z")

</div>

Could this be because of your `timeout => 5`?

> #### `timeout` [edit](https://github.com/logstash-plugins/logstash-filter-aggregate/edit/main/docs/index.asciidoc)
> 
> - Value type is [number](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#number)
> - Default value is `1800`
> 
> The amount of seconds (since the first event) after which a task is considered as expired.
> 
> When timeout occurs for a task, its aggregate map is evicted.
> 
> If _push\_map\_as\_event\_on\_timeout_ or _push\_previous\_map\_as\_event_ is set to true, the task aggregation map is pushed as a new Logstash event.
> 
> Timeout can be defined for each "task\_id" pattern.

If your SQL query is returning a large number of results, it will only hold open each record in memory for aggregation for 5 seconds with timeout set to 5.

Specifying a specific ID would result in fewer results that are less likely to hit the timeout.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 24, 2024, 5:10pm UTC](https://discuss.elastic.co/t/data-not-getting-synced-properly-from-sql-to-elastic/351742/3 "2024-01-24T17:10:09Z")

</div>

Are you running this pipeline with `pipeline.workers` set to `1` as well?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 21, 2024, 5:10pm UTC](https://discuss.elastic.co/t/data-not-getting-synced-properly-from-sql-to-elastic/351742/4 "2024-02-21T17:10:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.