# Logstash 8.6 low performance
**URL:** https://discuss.elastic.co/t/logstash-8-6-low-performance/344661
**Category:** Logstash
**Created:** [October 9, 2023, 1:33pm UTC](https://discuss.elastic.co/t/logstash-8-6-low-performance/344661 "2023-10-09T13:33:02Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@RobertC1](https://discuss.elastic.co/u/RobertC1)
#### Post date: [October 9, 2023, 1:33pm UTC](https://discuss.elastic.co/t/logstash-8-6-low-performance/344661/1 "2023-10-09T13:33:02Z")
Hi there
I ha a server with Linux Ubuntu 20.04 and ELK 8.6
I noticed that the ingestion proccess became slow and I have not change any parameters.
This is the conf file for theindex.
```auto
input {
file {
path => "/opt/trabaja/csv/trx_hours*.csv"
start_position => "beginning"
sincedb_path=> "NULL"
mode => "read"
file_completed_action => "delete"
file_sort_by => "path"
exit_after_read => true
}
}
filter {
csv { separator => ";"
columns => ["nodo","fecha","usuario","serviceid","sesid","base","sp","trn","ssn","tiempo","tiempo_sp","observacion"]}
mutate { remove_field => ["message", "@version","host","[log][file][path]" ] }
mutate { convert => ["fecha", "string"] }
mutate { convert => ["usuario", "string"] }
mutate { convert => ["serviceid", "string"] }
mutate { convert => ["sesid", "string"] }
mutate { convert => ["base", "string"] }
mutate { convert => ["sp", "string"] }
mutate { convert => ["trn", "float"] }
mutate { convert => ["ssn", "float"] }
mutate { convert => ["tiempo", "float"] }
mutate { convert => ["tiempo_sp", "float"] }
mutate { convert => ["observacion", "string"] }
mutate { add_field => { "fecha_dia" => "%{fecha}" } }
date {
match => ["fecha_dia", "yyyy-MM-dd HH:mm:ss.SSS" ,"ISO8601"]
timezone => "America/Argentina/Buenos_Aires"
target => "@timestamp"
}
mutate { remove_field => ["fecha_dia"]}
}
output{
elasticsearch { hosts => ["localhost:9200"]
index => "trx_hours_new"
user => "elastic"
password => "Accusys123*"
retry_on_conflict => 0 }
stdout { }
}
```
logstash jvm.options 8gb
Any help please.
Thanks in advance.
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [October 11, 2023, 11:21am UTC](https://discuss.elastic.co/t/logstash-8-6-low-performance/344661/2 "2023-10-11T11:21:08Z")
Since you are using Linux, set `sincedb_path => "/dev/null"`. The param: sincedb\_path=\> "NUL" is for Windows. Your .conf looks simple for processing should be fast.
Can you provide more details:
- how slow is, how many messages/event process per minute?
- does the /opt/trabaja/csv/ directory have a lot of files?
- what is file size?
- you are deleting file after read, am I right?
- why do you need stdout { }? It's debug and consume resources
- any particular reason for retry\_on\_conflict =\> 0?
- have you check with the [API](http://localhost:9600/_node/stats/pipelines?pretty) or in Kibana Stack monitoring which plugin consume the most resources
---
### Author:  [@RobertC1](https://discuss.elastic.co/u/RobertC1)
#### Post date: [October 16, 2023, 6:43pm UTC](https://discuss.elastic.co/t/logstash-8-6-low-performance/344661/3 "2023-10-16T18:43:18Z")
- how slow is, how many messages/event process per minute?
**60k per minute**
- does the /opt/trabaja/csv/ directory have a lot of files?
**Yes, with different sizes, the less value 8MB the biggest 1.7GB**
- you are deleting file after read, am I right?
**Yes the files are deteted.**
- why do you need stdout { }? It's debug and consume resources
**I will take it out.**
- any particular reason for retry\_on\_conflict =\> 0?
**None, I will delete the line**
- have you check with the [API](http://localhost:9600/_node/stats/pipelines?pretty) or in Kibana Stack monitoring which plugin consume the most resources
**Could you give more hints about the kibana, which plugin do I have to use for monitoring**
Thanks for your time and recommendations
---
### Author:  [@RobertC1](https://discuss.elastic.co/u/RobertC1)
#### Post date: [October 16, 2023, 8:49pm UTC](https://discuss.elastic.co/t/logstash-8-6-low-performance/344661/4 "2023-10-16T20:49:23Z")
Following your recommendatios the file.conf
```auto
input {
file {
path => "/opt/trabaja/csv/trx_hours*.csv"
start_position => "beginning"
sincedb_path=> "NULL"
mode => "read"
file_completed_action => "delete"
file_sort_by => "path"
exit_after_read => true
}
}
filter {
csv { separator => ";"
columns => ["nodo","fecha","usuario","serviceid","sesid","base","sp","trn","ssn","tiempo","tiempo_sp","observacion"]}
mutate { remove_field => ["message", "@version","host","[log][file][path]" ] }
mutate { convert => ["fecha", "string"] }
mutate { convert => ["usuario", "string"] }
mutate { convert => ["serviceid", "string"] }
mutate { convert => ["sesid", "string"] }
mutate { convert => ["base", "string"] }
mutate { convert => ["sp", "string"] }
mutate { convert => ["trn", "float"] }
mutate { convert => ["ssn", "float"] }
mutate { convert => ["tiempo", "float"] }
mutate { convert => ["tiempo_sp", "float"] }
mutate { convert => ["observacion", "string"] }
mutate { add_field => { "fecha_dia" => "%{fecha}" } }
date {
match => ["fecha_dia", "yyyy-MM-dd HH:mm:ss.SSS" ,"ISO8601"]
timezone => "America/Argentina/Buenos_Aires"
target => "@timestamp"
}
mutate { remove_field => ["fecha_dia"]}
}
output{
elasticsearch { hosts => ["localhost:9200"]
index => "trx_hours_new"
user => "elastic"
password => "Accusys123*"
action =>"index" }
}
```
Now is indexing 388.000 lines per minute.
I really appreciate you sharing your great experience.
Thanks
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [October 17, 2023, 8:18am UTC](https://discuss.elastic.co/t/logstash-8-6-low-performance/344661/5 "2023-10-17T08:18:21Z")
For more then hundred replays, this is for the first time someone clearly responded to my questions.
Dear Elastic Team Members, can you please provide a gift to him? A t-shirt, mug, pen, anything?
Back to the topic. The main cause of slow processing is debug. Your .conf is not complex. I have removed lines with conversion to string - default is string, no need AFAIK, and removed few lines and `fecha_dia` if you need only for date convers. Try also with the dissect plugin, maaaaaybe you will get a little bit more on performances. If you don't need the `event` field, you can removed.
In the file plugin, you should use:
- Windows: sincedb\_path =\> "NUL"
- Linux: sincedb\_path =\> "/dev/null", in your case
You can set real path and file in sincedb\_path to track which files are processes. It's up to you.
```auto
input {
file {
path => "/opt/trabaja/csv/trx_hours*.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
mode => "read"
file_completed_action => "delete"
file_sort_by => "path"
exit_after_read => true
}
}
filter {
csv { separator => ";"
columns => ["nodo","fecha","usuario","serviceid","sesid","base","sp","trn","ssn","tiempo","tiempo_sp","observacion"]
}
# dissect {
# mapping => { "message" => "%{nodo};"%{fecha}";"%{usuario}";"%{serviceid}";"%{sesid}";"%{base}";"%{sp}";"%{trn}";"%{ssn}";"%{tiempo}";"%{tiempo_sp}";"%{observacion}" }
# }
mutate { remove_field => ["message", "host","log", "event"] }
mutate { convert => ["trn", "float"] }
mutate { convert => ["ssn", "float"] }
mutate { convert => ["tiempo", "float"] }
mutate { convert => ["tiempo_sp", "float"] }
date {
match => ["fecha", "yyyy-MM-dd HH:mm:ss.SSS" ,"ISO8601"]
timezone => "America/Argentina/Buenos_Aires"
# target => "@timestamp" # no need, @timestamp it's default destination field.
}
}
```
If you like to use only the CSV plugin, you can do conversion inside csv.
```auto
csv {
separator => ";"
convert => {
"trn" => "float"
"ssn" => "float"
"tiempo" => "float"
"tiempo_sp" => "float"
}
columns => [...
}
```
LS can provide processing info. [Here](https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#pipeline-stats) you can find more info.
For LS\<8.x you can use monitoring integrated inside, add in logstash.yml:
```auto
xpack.monitoring.enabled: true
xpack.monitoring.collection.interval: 5s
xpack.monitoring.collection.pipeline.details.enabled: true
xpack.monitoring.elasticsearch.hosts: "localhost:9200"
```
For LS 8.x you should use Metricbeat and the logstash module to see info in Kibana. With Stack Monitoring enabled, even without Metricbeat you should see some [metrics in Kibana](https://www.elastic.co/guide/en/kibana/current/logstash-page.html). Also when LS is running you can use curl [http://localhost:9600/\_node/stats/pipelines?pretty](http://localhost:9600/_node/stats/pipelines?pretty) to see perfomances in JSON format.
---
### Author:  [@RobertC1](https://discuss.elastic.co/u/RobertC1)
#### Post date: [October 18, 2023, 9:04pm UTC](https://discuss.elastic.co/t/logstash-8-6-low-performance/344661/6 "2023-10-18T21:04:00Z")
@Rios
Finally this is the .conf file.
```auto
input {
file {
path => "/opt/trabaja/csv/trx_hours*.csv"
start_position => "beginning"
sincedb_path => "NULL"
mode => "read"
file_completed_action => "delete"
file_sort_by => "last_modified"
exit_after_read => true
}
}
filter {
csv { separator => ";"
convert => {
"trn" => "float"
"ssn" => "float"
"tiempo" => "float"
"tiempo_sp" => "float"
}
columns => ["nodo","fecha","usuario","serviceid","sesid","base","sp","trn","ssn","tiempo","tiempo_sp","observacion"]
}
mutate { remove_field => ["message", "@version","host","[log][file][path]" ] }
date {
match => ["fecha_dia", "yyyy-MM-dd HH:mm:ss.SSS" ,"ISO8601"]
timezone => "America/Argentina/Buenos_Aires"
target => "@timestamp"
}
mutate { remove_field => ["fecha_dia"]}
}
output{
elasticsearch { hosts => ["localhost:9200"]
index => "trx_hours_new"
user => "elastic"
password => "Accusys123*"
action =>"index" }
}
```
Now is indexing 800K per minute.
I could not not take away this section
```auto
date {
match => ["fecha_dia", "yyyy-MM-dd HH:mm:ss.SSS" ,"ISO8601"]
timezone => "America/Argentina/Buenos_Aires"
target => "@timestamp"
}
mutate { remove_field => ["fecha_dia"]}
}
```
Because @timestamp takes the actual date when is running.
Nevertheless, it was a very good help
Thanks a lot!!
P.S. I am waiting for my gift.
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [October 20, 2023, 7:15am UTC](https://discuss.elastic.co/t/logstash-8-6-low-performance/344661/7 "2023-10-20T07:15:15Z")
I cannot see where was copied `fecha` to `fecha_dia` in this .conf
Also, you can set without coping:
```auto
date {
match => ["fecha", "yyyy-MM-dd HH:mm:ss.SSS" ,"ISO8601"]
timezone => "America/Argentina/Buenos_Aires"
target => "@timestamp"
}
```
---
### Author:  [@RobertC1](https://discuss.elastic.co/u/RobertC1)
#### Post date: [October 20, 2023, 10:03am UTC](https://discuss.elastic.co/t/logstash-8-6-low-performance/344661/8 "2023-10-20T10:03:24Z")
@Rios
Yes I missed that line in th code; but was there.
Thank you very much for all the suggestions.
---
### Author:  [@RobertC1](https://discuss.elastic.co/u/RobertC1)
#### Post date: [October 25, 2023, 2:24pm UTC](https://discuss.elastic.co/t/logstash-8-6-low-performance/344661/9 "2023-10-25T14:24:09Z")
@Rios
You were right, there was not need to create the field fecha\_dia to get the right datetime.
```auto
date {
match => ["fecha", "yyyy-MM-dd HH:mm:ss.SSS" ,"ISO8601"]
timezone => "America/Argentina/Buenos_Aires"
target => "@timestamp"
}
```
I already changed all the .conf.
Best regards.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 22, 2023, 2:24pm UTC](https://discuss.elastic.co/t/logstash-8-6-low-performance/344661/10 "2023-11-22T14:24:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.