# Logstash-s3-plugin not importing data or creating sincedb file

**URL:** <https://discuss.elastic.co/t/logstash-s3-plugin-not-importing-data-or-creating-sincedb-file/155245>\
**Category:** Logstash\
**Created:** [November 3, 2018, 7:02am UTC](https://discuss.elastic.co/t/logstash-s3-plugin-not-importing-data-or-creating-sincedb-file/155245 "2018-11-03T07:02:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![tom.isaacson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom.isaacson/32/37257_2.png) [@tom.isaacson](https://discuss.elastic.co/u/tom.isaacson)\
**Post date:** [November 3, 2018, 7:02am UTC](https://discuss.elastic.co/t/logstash-s3-plugin-not-importing-data-or-creating-sincedb-file/155245/1 "2018-11-03T07:02:56Z")

</div>

I have an S3 bucket containing logs from other S3 buckets showing downloads, etc. I pulled [Docker](https://hub.docker.com/r/sebp/elk/) then installed [S3 input plugin | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-s3.html). I created a logstast.conf file:

```
input {
    s3 {
      access_key_id => "XXXX"
      secret_access_key => "XXXX"
      bucket => "mybucket"
      region => "us-east-1"
      prefix => "logs/"
      type => "s3"
    }
}

output { 
  elasticsearch { hosts => ["localhost:9200"] }
}

```

then run this with "/opt/logstash/bin/logstash --path.data /tmp/logstash/data -f config/logstash.conf". The S3 input seems to start correctly and if I turn on --debug I can see it parsing logs:

> [2018-11-03T06:03:06,273][DEBUG][logstash.inputs.s3] objects length is: {:length=\>26999}  
> [2018-11-03T06:03:06,273][DEBUG][logstash.inputs.s3] S3 input: Found key {:key=\>"logs/2016-05-21-13-32-20-1F8C0D40595F3975"}  
> [2018-11-03T06:03:06,273][DEBUG][logstash.inputs.s3] S3 input: Adding to objects {:key=\>"logs/2016-05-21-13-32-20-1F8C0D405

but I'm never seeing any data in Elasticsearch. I added:  
` sincedb_path => "/etc/s3backup/sincedb"`  
to my .conf file but I'm not seeing anything in this directory.

Have I made an obvious mistake? What else can I do to debug this?

---

<div class="post-metadata">

**Author:** ![jordansissel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jordansissel/32/44957_2.png) [@jordansissel](https://discuss.elastic.co/u/jordansissel)\
**Post date:** [November 14, 2018, 4:16am UTC](https://discuss.elastic.co/t/logstash-s3-plugin-not-importing-data-or-creating-sincedb-file/155245/2 "2018-11-14T04:16:13Z")

</div>

> [@tom.isaacson](#):
>
> [2018-11-03T06:03:06,273][DEBUG][logstash.inputs.s3] objects length is: {:length=\>26999}  
> [2018-11-03T06:03:06,273][DEBUG][logstash.inputs.s3] S3 input: Found key {:key=\>"logs/2016-05-21-13-32-20-1F8C0D40595F3975"}  
> [2018-11-03T06:03:06,273][DEBUG][logstash.inputs.s3] S3 input: Adding to objects {:key=\>"logs/2016-05-21-13-32-20-1F8C0D405

For these logs, how many objects is it finding?

The reason I ask is this: There's a problem with S3 that it is very slow to list objects. If your bucket has thousands (or millions) of objects, it can take many minutes or even hours to finish listing.

The way the S3 input plugin works today is basically this:

1. List all objects in the given prefix
2. Process any objects found by that listing.
3. Update sincedb each time an object completes processing.
4. Go back to step 1, ignoring any already-processed objects.

If step 1 would list millions of objects, then the act of listing could take an hour (or more), and Logstash would wait for the listing to complete before processing (step 2) any log files found in your S3 bucket. This is an annoying issue, I admit, though I dont' know for sure if this is what impacts you.

I wrote an issue [documenting this phenomenon](https://github.com/logstash-plugins/logstash-input-s3/issues/128) and a [proposal for improving](https://github.com/logstash-plugins/logstash-input-s3/issues/129) that might interest you.

Knowing how many objects your `logs/` prefix has would help determine whether or not your issue matches what I am describing.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 12, 2018, 4:16am UTC](https://discuss.elastic.co/t/logstash-s3-plugin-not-importing-data-or-creating-sincedb-file/155245/3 "2018-12-12T04:16:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.