# Separate documents from API request

**URL:** <https://discuss.elastic.co/t/separate-documents-from-api-request/215902>\
**Category:** Elasticsearch\
**Created:** [January 21, 2020, 3:06pm UTC](https://discuss.elastic.co/t/separate-documents-from-api-request/215902 "2020-01-21T15:06:10Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![victor.nilsson](https://avatars.discourse-cdn.com/v4/letter/v/eb8c5e/32.png) [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\
**Post date:** [January 21, 2020, 3:06pm UTC](https://discuss.elastic.co/t/separate-documents-from-api-request/215902/1 "2020-01-21T15:06:10Z")

</div>

Hi,

We're using logstash with the HTTP\_input plugin, to index events from a ReST API. The problem is that for every API request, we effectively index duplicates. If we run a GET request the output looks something like this:

```
{
  "_index": "randomindex",
  "_type": "_doc",
  "_id": "XTqUyG8BFEo1Fcv79Ns-",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-01-21T14:50:02.087Z",
    "@version": "1",
    "requests": [
      {
        "internalIp": "127.0.0.1",
        "categories": [
          "Ecommerce/Shopping",
          "Business Services",
          "Financial Institutions",
          "Phishing",
          "Infrastructure",
          "Application"
        ],
        "datetime": "2020-01-21T12:26:36.000Z",
        "originType": "Roaming Computers",
        "originId": 1231231,
        "tags": [],
        "externalIp": "127.0.0.1",
      },
      {
        "internalIp": "127.0.0.1",
        "categories": [
          "Ecommerce/Shopping",
          "Business Services",
          "Financial Institutions",
          "Phishing",
          "Infrastructure",
          "Application"
        ],
        "datetime": "2020-01-21T06:02:42.000Z",
        "originType": "Roaming Computers",
        "originId": 1231231,
        "tags": [],
        "externalIp": "127.0.0.1",
      },
      {
        "internalIp": "127.0.0.1",
        "categories": [
          "Sports",
          "Malware"
        ],
        "datetime": "2020-01-21T10:52:54.000Z",
        "originType": "Roaming Computers",
        "originId": 123123,
        "tags": [],
      },

```

The next time we GET from the API, we get the same logs back, effectively indexing duplicates.

Could we somehow use some field in the event to sort out duplicates? Maybe using the "datetime" field?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 18, 2020, 3:06pm UTC](https://discuss.elastic.co/t/separate-documents-from-api-request/215902/2 "2020-02-18T15:06:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.