Improve performance in a single-node deployment


#1

Hi, I hope you can help me.

I'm currently trying to load a lot of logs that are almost xml. The problem comes because in a minute 20 logs files of 5mb are generated and logstash takes approximately 12 minutes to load them. Ideally logstash upload 20 files per minute.

I've been practicing all month to improve performance according to logstash and elasticsearch documentation

my configuration file

input {
  file {
    path => "/elastic/input/example/allarchives.log"
    start_position => "beginning"
   sincedb_path => "/dev/null"
     codec => multiline {
                         charset => "ISO-8859-1"
                         pattern => '\<\/GTD.*\>'
                         negate => true
                           what => "next"
                           max_lines => 2000
 }
   
  }
}
filter {
xml {
		source => "message"
    target => "doc"
    remove_field => [  "message" ]
}
}
output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "example"
  }
stdout {}
}

I have logstash 6.4, elasticsearch 6.4 and kibana 6.4 running on a single-node, which has these specifications:

6 x Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz
RAM 24GB
OS: Red Hat Enterprise linux 7

sorry for my bad english, I am attentive to your suggestions

I leave some screenshots of the configuration of the index and monitoring with kibana

rendimiento1



(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.