@guyboertje thanks for the prompt response. I was just reading your response at _node/stats - duration_in_millis vs queue_push_duration_in_millis
I have the pipeline stats for my last run. So, I am posting that for now.
"pipeline" : {
"events" : {
"duration_in_millis" : 142914507,
"in" : 57297971,
"out" : 57297971,
"filtered" : 57297971,
"queue_push_duration_in_millis" : 272075
},
"plugins" : {
"inputs" : [ {
"id" : "fab0b744ca08b9cc7117f287f8d87e4dddfcadeb-2",
"events" : {
"out" : 57297971,
"queue_push_duration_in_millis" : 272075
},
"current_connections" : 0,
"name" : "beats",
"peak_connections" : 2
} ],
"filters" : [ {
"id" : "fab0b744ca08b9cc7117f287f8d87e4dddfcadeb-4",
"events" : {
"duration_in_millis" : 179249,
"in" : 57297971,
"out" : 57297971
},
"name" : "geoip"
}, {
"id" : "fab0b744ca08b9cc7117f287f8d87e4dddfcadeb-3",
"events" : {
"duration_in_millis" : 1827832,
"in" : 57297971,
"out" : 57297971
},
"matches" : 57295891,
"failures" : 2080,
"patterns_per_field" : {
"message" : 1
},
"name" : "grok"
}, {
"id" : "fab0b744ca08b9cc7117f287f8d87e4dddfcadeb-5",
"events" : {
"duration_in_millis" : 499858,
"in" : 57297971,
"out" : 57297971
},
"matches" : 57295891,
"name" : "date"
} ],
"outputs" : [ {
"id" : "fab0b744ca08b9cc7117f287f8d87e4dddfcadeb-6",
"events" : {
"duration_in_millis" : 4794278,
"in" : 57297971,
"out" : 57297971
},
"name" : "elasticsearch"
} ]
}
It seems grok is the plugin that is taking time [Correct me if I am wrong].
I will continue the process of elimination that you suggested. Meanwhile, I would like to have your suggestion on how this pipeline can be sped up. As the RAM and CPU usage is not much, I can add more logstash processes on the same box. Will that work?