Unable to restrict duplicate documents using http_poller

Hi All,

I have been trying to ingest some, time series data into elasticsearch using logstash http poller plugin.
my data is from a monitoring tool which capture the parameters like hostname, memory usage, cpu load, system status and time when it captured the data

hostname : server1
memory load : 95%
cpu load : 80%
system status : Up
time: 01-08-2021 12:35:44  

When i run logstash using the provided rest api, for every poll it ingests the complete data into elasticsearch, but i want logstash to read the files based on the latest time recorded, so that i can get rid of duplicate data.

Any suggestions please.


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.