Only write the newest data into ES

Elastic Stack Logstash
1

I'm pulling a datafile into elasticsearch
I'm using http_poller with an API and I'm including the metadata where I get a last-modified timestamp for my datafile.

What I want to do is compare the http_poller timestamp with the timestamp in the ES database and only write the new version of my datafile into ES.
The end goal would be that logstash checks the API ones a second, but only grab the datafile when there is an update. But how do I do that?

My guess is that in the output{} should make a lookup in the ES database, but I can't figure out how. Any suggestions?

input {
  http_poller {
    urls => {...}
    metadata_target => "http_poller_metadata"
  }
}

filter {
}

output {
	if API_timestamp > ES_timestamp
		elasticsearch { ...  }
  stdout {
    codec => rubydebug
  }
}
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.