Logstash and elasticsearch performing

Elastic Stack Logstash
1

hey ,

my configuration 40 G Ram, 2 CPU 4 core , elk and nginx in the same server

il give 15g -Xmx for logstash and elactic

input {
file {
start_position => "beginning"
path => "/var/log/StatVM/*"
sincedb_path => "/dev/null"
}
}

filter {
grok {
break_on_match => "false"
match => {
"message" => '%{DATA}%{DATE_EU:Date};%{TIME:Time};%{NUMBER:Nombre}%{DATA}'
}
add_field => {
"timestamp" => "%{Date} %{Time}"
}
}

date {
match => [ "timestamp", "dd MM YYYY HH:mm:ss" ]
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
}
}

the size of the file is of 5G and when i do TOP i was

2

What is the question ?

3

oh sorry ,
you can see my cpu 750% i don't undestand , in the documentation we can see up memory for up performance but java don't use this memory ... only CPU , so i must use many node ? what i have to do for up my performance ?

4

Your logstash use all the CPU, maybe there is a lot of logs to parse from the beginning ? Else, ask the question on the logstash forum.

5

up !

6

What's the message rate through Logstash?

7

That is to say, can you develop, i begining

8

How many messages is Logstash processing per second? If it's processing thousands of messages every second it's not surprising if it uses a lot of CPU.

9

yes, Hundred thousand and I can have losses ? between logstash and ES ? does he exist a cluster for logstash ?

10

Hundred thousand

Per second?

and I can have losses ? between logstash and ES ?

There are cases when that can occur.

does he exist a cluster for logstash ?

Not in the same sense as Elasticsearch, but you can distribute load to multiple Logstash instances either by sending the events to a message broker than any number of Logstash instances can read from, or you can use a load balancer in front of the Logstash instances.

11

no sorry just thousand

in which case

can you give me some solution or explication

12

can you give me some solution or explication

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.