Data not getting synced properly from SQL to elastic

Elastic Stack Elasticsearch
1

I am syncing data from sql to elastic and in filter in the code block I am appending to map eg features but what is happening is that when I am running the bulk sql query ,not all features are getting appended to each index. eg if index 1 has 10 features then 2 or 3 are getting ingested to elastic but when I am directly specifying the id in = in sql query then all 10 features are getting synced. But the bulk query if I am running separating then also it gives the correct count.

filter {
    aggregate {
      task_id => "%{HotelCode}"
      code => "
        map['HotelCode'] ||= event.get('HotelCode')

        map['Features']||= []  

        map['Features'] << {

            'FeatureId' => event.get('FeatureId'),
            'FeatureIdName' => event.get('FeatureIdName'),
            'Group' => event.get('GroupName'),
            'Rank' => event.get('Rank')
        }

        event.cancel()
       "
      push_previous_map_as_event => true
      timeout => 5
    }
  }
2

@abhishek_agarwal if this is about configuring Logstash, you might want to choose that category for your question to have it reach the right groups, or at least add the logstash tag.
elastic-stack-sql is about using Elastic's SQL implementation once your data is ingested into Elasticsearch.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.