Logstash is picking messages quiet slowly

[rohitarorait82 (Rohit arora)]

Hi All,

We have traffic around 100 million records per day and it was working fine till yesterday , but today I can see 50 million records are piled up at the the source system . i have tried increasing pipeline.workers as well but no luck. Any suggestions how to make it work

[warkolm (Mark Walkom)]

What version are you on?
What do you mean by "piled up"?
What is your config?

[rohitarorait82 (Rohit arora)]

Hi @warkolm : Thanks for your reply.

I am using version 7.7. We are using IBM MQ as source system and queues are getting piled over.

PFB cofig files.

input {
jms {
yaml_file => "/etc/logstash/input_mq_DPW1.yml"
yaml_section => "wmq"
include_header => false
include_properties => false
include_body => true
use_jms_timestamp => false
destination => "Input.Q"
interval => 3
runner => async
}
}
filter {
xml {
store_xml => false
source => "message"
remove_field => ["message"]
xpath =>[ "/inmsg/ConsumerToESB_request/text()", "REQUEST"]
xpath =>[ "/inmsg/ESBToConsumer_response/text()", "RESPONSE"]
xpath =>[ "/inmsg/CosumerToProvider_request/text()", "CosumerToProvider_request"]
xpath =>[ "/inmsg/ProviderToConsumer_response/text()", "ProviderToConsumer_response"]
xpath =>[ "/indetails/area/text()", "AREA_ID"]
xpath =>[ "/indetails/apiname/text()", "SERVICE_NAME"]
xpath =>[ "/indetails/cons/text()", "CONS_NAME"]

}
mutate{
gsub => ["REQUEST","<","<"]
gsub => ["REQUEST",">",">"]
gsub => ["RESPONSE",">",">"]
gsub => ["RESPONSE","<","<"]
gsub => ["CosumerToProvider_request","<","<"]
gsub => ["CosumerToProvider_request",">",">"]
gsub => ["ProviderToConsumer_response","<","<"]
gsub => ["ProviderToConsumer_response",">",">"]

}
}
output{
if "service1" in [SERVICE_NAME]{
elasticsearch {
hosts => ["http://ip1:9200","http://ip2:9200","http://ip3:9200","http://ip4:9200"]
index => "index1-%{+YYYY.ww}"
user => "elastic"
password => "password"
}
}
if "service2" in [SERVICE_NAME]{
elasticsearch {
hosts => ["http://ip1:9200","http://ip2:9200","http://ip3:9200","http://ip4:9200"]
index => "index2-%{+YYYY.ww}"
user => "elastic"
password => "password"
}
}

else {
elasticsearch {
hosts => ["http://ip1:9200","http://ip2:9200","http://ip3:9200","http://ip4:9200"]
index => "index3-%{+YYYY.MM}"
user => "elastic"
password => "password"
}
}
}