# Logstash missing logs from s3 bucket

**URL:** https://discuss.elastic.co/t/logstash-missing-logs-from-s3-bucket/90333
**Category:** Logstash
**Created:** [June 21, 2017, 4:59pm UTC](https://discuss.elastic.co/t/logstash-missing-logs-from-s3-bucket/90333 "2017-06-21T16:59:51Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![sarthak2016](https://avatars.discourse-cdn.com/v4/letter/s/d26b3c/32.png) [@sarthak2016](https://discuss.elastic.co/u/sarthak2016)
#### Post date: [June 21, 2017, 4:59pm UTC](https://discuss.elastic.co/t/logstash-missing-logs-from-s3-bucket/90333/1 "2017-06-21T16:59:51Z")

</div>

Hello All,

I am importing around 500 CSV's into "test" bucket every hourly, But when i verify data/hits into Kibana, Half of logs are missing, When i checked CSV's into S3, All CSV's are exits, Also Logstash script is running correctly without any error, I also reviewed sincedb\_path =\> 'C:/Test/logstash/test.sincedb' and its saves last process CSV time,

For Example,  
T1\_File 1\_Datetime.CSV  
T1\_File 2\_Datetime.CSV  
T2\_File 3\_Datetime.CSV  
T2\_File 4\_Datetime.CSV  
T3\_File 5\_Datetime.CSV  
.....  
.....  
.....  
.....  
File 500\_Datetime.CSV,

Logstash import File 1 to File 90 CSV's, Then Gap File 91 to 100 CSV's,

Please find the Logstash input code and Kibana visualization,

Logstash input section as below,

input  
{  
s3  
{  
access\_key\_id =\> "XXXXXXXXXXXXXXXXXX"  
secret\_access\_key =\> "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"  
region =\> "us-east-1"  
bucket =\> "test"  
prefix =\> "Logstashfolder/test/"  
delete =\> false  
interval =\> 10 # seconds  
sincedb\_path =\> 'C:/Test/logstash/test.sincedb'  
}

}

Kibana View for T1 file Last 24 hours

 ![](https://us1.discourse-cdn.com/elastic/original/3X/7/5/75d9db1696d5a610dcf24d98dc9f25f7fbc0f3d9.png)

Note: CSV's are exits into S3 between 5 AM to 12 PM, Logstash missed this CSV's to import into ES, If i manually upload this files/CSV's between 5 AM to 12 PM into S3 bucket then Logsatsh will import it,

Kibana View for T2 file Last 24 hours

![](https://us1.discourse-cdn.com/elastic/original/3X/2/9/290538a61f184dacc86921fe1f26bca197008214.png)

Can someone please let me know whats the actual issue and how to resolve it ? Or Its Logstash performance issue ?

Thanks  
SAM

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 19, 2017, 5:00pm UTC](https://discuss.elastic.co/t/logstash-missing-logs-from-s3-bucket/90333/2 "2017-07-19T17:00:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.