Logstash stops streaming/uploading csv files to elastic search after some time

Elastic Stack Logstash
1

ElasticSearch Configuration
1 zone - 4gb ram - 120GB storage

Logstash Configuration
4vcpu - 16gb ram - t2.xlarge instance - logstash 7.7.0

Workflow

  1. One tool gets data from different sources and creates csv files in different folders.
  2. Logstash is configured to process those csv files and send it to elastic search.
  3. In 24hrs, we create about 5 gb of total files. There are multiple csv files. Each file size is around 1-2mb.
  4. Logstash uploads about 4000 entries per min to elastic search by processing those files. As soon as a new file is created it will upload to elastic search.

Logstash conf file

input {
file {
path => "/home/ubuntu/output/**/*.csv"
start_position => "beginning"
sincedb_write_interval => 15
mode => "read"
file_completed_action => "delete"
}
}
filter {
csv {
separator => "$"
columns => ["url", "index", "hash", "all", "subject", "test1", "test12", "test13", "test14", "test5", "test56", "test57", "date1", "date2"]
}
}
output {
elasticsearch
{
hosts => ["[https://hash.us-east-1.aws.found.io:9243"]
user => "user"
password => "pass"
index => "testindex"
}
}

Logstash.yml /etc/logstash/logstash.yml

path.data: /var/lib/logstash
pipeline.ordered: auto
queue.type: memory
queue.type: persisted
queue.max_bytes: 8gb
path.logs: /var/log/logstash
cloud.id: "BB_user:pass"
cloud.auth: "user:pass"

Note: I have removed all commented lines from logstash.yml while posting here.

Problem we are are facing ...

  1. Logstash will work perfectly for few hours then it starts producing delay of few mins then afterwards it will produce delay of few hours.
  2. When we restart, it starts working correctly. i.e. it will work perfectly for few hours then again start producing delay.
  3. Now, yesterday, it completely stopped working. When we restarted it, it never uploads anything to elasticsearch. It had 35k csv files pending and it didn't upload anything.

Can you please take a look at this and help us with it?

2

The bottleneck could be on Logstash/Elasticsearch side or a combination of both.

For Elasticsearch: How many primary shards do you have for your index testindex? Are you saturating your IOPS/network on that Elasticsearch instance?

For Logstash : You could try using a stdout plugin and use the Dots Codec to find out Logstash's throughput and see if the lag is happening on the Logstash side to rule out the indexing throughput on the Elasticsearch as the reason.

3

Thanks Rahul for replying.

For logstash

  1. I ran this plugin.
  stdout { codec => dots }
  1. Currently, we have 200k+ csv files with 1000 entries each.
  2. Logstash starts and doesn't upload anything.
  3. When I tried with single file, it works perfectly.

For Elasticsearch

  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  }

Not sure what's happening.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.