# Some documents missing in Elasticsearch

**URL:** <https://discuss.elastic.co/t/some-documents-missing-in-elasticsearch/125787>\
**Category:** Logstash\
**Created:** [March 27, 2018, 4:26pm UTC](https://discuss.elastic.co/t/some-documents-missing-in-elasticsearch/125787 "2018-03-27T16:26:52Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ben.zeghers](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@ben.zeghers](https://discuss.elastic.co/u/ben.zeghers)\
**Post date:** [March 27, 2018, 4:26pm UTC](https://discuss.elastic.co/t/some-documents-missing-in-elasticsearch/125787/1 "2018-03-27T16:26:52Z")

</div>

I'm using Logstash (hosted in AWS Fargate) to import json files from my S3 bucket to Elasticsearch (hosted by Elastic Cloud). It works perfectly file for most of the files but some (~1%) never show up when I query for them using Kibana. If I re-upload the same file to S3, which causes the document to be re-indexed, it then shows up as expected. I've enabled debug logging for Logstash and can see that it's processing the file (see below).

I'm not sure how to debug this problem any further or if it's an issue with Logstash or Elasticsearch. Any help would be much appreciated.

Logstash debug logs:

```
[2018-03-21T19:48:11,679][DEBUG][logstash.inputs.s3] S3 input processing {:bucket=>"my-bucket", :key=>"142439.json"}
[2018-03-21T19:48:11,680][DEBUG][logstash.inputs.s3] S3 input: Download remote file {:remote_key=>"142439.json", :local_filename=>"/tmp/logstash/142439.json"}
[2018-03-21T19:48:11,691][DEBUG][logstash.inputs.s3] Processing file {:filename=>"/tmp/logstash/142439.json"}
[2018-03-21T19:48:11,768][DEBUG][logstash.pipeline] filter received {"event"=>{"sale_id"=>142439, snip..., "@version"=>"1", "@timestamp"=>2018-03-21T19:48:11.691Z}}
[2018-03-21T19:48:11,768][DEBUG][logstash.util.decorators] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"[@metadata][action]", "value"=>["index"]}
[2018-03-21T19:48:11,768][DEBUG][logstash.pipeline] output received {"event"=>{"sale_id"=>142439, snip..., "@version"=>"1", "@timestamp"=>2018-03-21T19:48:11.691Z}}

```

My Logstash configuration file is:

```
input {
    s3 {
        bucket => "${BUCKET}"
        codec => "json"
    }
}

filter {
    if [remove] {
        mutate {
            add_field => {
                "[@metadata][action]" => "delete"
            }
        }
    } else {
        mutate {
            add_field => {
                "[@metadata][action]" => "index"
            }
        }
    }
}

output {
    elasticsearch {
        hosts => ["${ELASTICSEARCH_HOST}"]
        user => "${ELASTICSEARCH_USER}"
        password => "${ELASTICSEARCH_PASSWORD}"
        index => "used-retail-sales"
        document_id => "%{sale_id}"
        action => "%{[@metadata][action]}"
    }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 24, 2018, 4:27pm UTC](https://discuss.elastic.co/t/some-documents-missing-in-elasticsearch/125787/2 "2018-04-24T16:27:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.