Logstash and elasticsearch performing


(passat) #1

hey ,

my configuration 40 G Ram, 2 CPU 4 core , elk and nginx in the same server

il give 15g -Xmx for logstash and elactic

input {
file {
start_position => "beginning"
path => "/var/log/StatVM/*"
sincedb_path => "/dev/null"
}
}

filter {
grok {
break_on_match => "false"
match => {
"message" => '%{DATA}%{DATE_EU:Date};%{TIME:Time};%{NUMBER:Nombre}%{DATA}'
}
add_field => {
"timestamp" => "%{Date} %{Time}"
}
}

date {
match => [ "timestamp", "dd MM YYYY HH:mm:ss" ]
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
}
}

the size of the file is of 5G and when i do TOP i was


(Xavier Facq) #2

What is the question ?


(passat) #3

oh sorry ,
you can see my cpu 750% i don't undestand , in the documentation we can see up memory for up performance but java don't use this memory ... only CPU , so i must use many node ? what i have to do for up my performance ?


(Xavier Facq) #4

Your logstash use all the CPU, maybe there is a lot of logs to parse from the beginning ? Else, ask the question on the logstash forum.


(passat) #5

up !


(Magnus Bäck) #6

What's the message rate through Logstash?


(passat) #7

That is to say, can you develop, i begining


(Magnus Bäck) #8

How many messages is Logstash processing per second? If it's processing thousands of messages every second it's not surprising if it uses a lot of CPU.


(passat) #9

yes, Hundred thousand and I can have losses ? between logstash and ES ? does he exist a cluster for logstash ?


(Magnus Bäck) #10

Hundred thousand

Per second?

and I can have losses ? between logstash and ES ?

There are cases when that can occur.

does he exist a cluster for logstash ?

Not in the same sense as Elasticsearch, but you can distribute load to multiple Logstash instances either by sending the events to a message broker than any number of Logstash instances can read from, or you can use a load balancer in front of the Logstash instances.


(passat) #11

no sorry just thousand

in which case

can you give me some solution or explication


(Magnus Bäck) #12

can you give me some solution or explication

https://www.elastic.co/guide/en/logstash/5.1/deploying-and-scaling.html#deploying-minimal-install


(system) #13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.