Only insert json objects that are not in elasticsearch

Hello. Very new to elasticsearch. I am trying to use the ELK stack to provide a tool for viewing records obtain from an external API. The setup is:

  • Get new records from external API
  • Add them to elasticsearch if they do not exist in elastic search
  • View using Kibana

Every time I get new records from the external API, I save them as a list of dictionaries -- each record is a dictionary, there is a list of such records. I am looking for a query to get all documents from elasticsearch that match the any item in that list.

For example if json_dump is a list of json objects or dictionaries, how would you query elasticsearch such that only the documents that match json objects in json_dump are returned. The idea here is to then push documents_from_external_API - documents_from_es_query into elasticsearch via logstash.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.