Hi everyone,
I'm fetching data from Elasticsearch and putting into Bigquery through Logstash pipeline.
When I'm fetching 15 days of data then I'm getting same number of records as source. No issues with 15 days of data.
However, when I'm fetching data for 1.5 months, my data is getting double and triple in the target BQ table.
I'm using below query:
"query":
{
"range" : {
"timestamp" : {
"gte" : "2019-04-16",
"lte" : "2019-05-31"
}
}
}
I'm not sure whether my query is correct to pull data from ES and put into BQ or not.
In kibana, above query is working fine. But when I'm pulling data through Logstash with same query, then data is getting double.
Could anyone advice me on this issue.
Thanks in advance!