I have an logstash pipeline with http poller urls. Each urls will send JSON data for a given server name.
Below is my conf -
input {
http_poller {
urls => {
servsb => "http://servsb:8003/servmetric/json/"
servsd => "http://servsd:8003/servmetric/json/"
servsf => "http://servsf:8003/servmetric/json/"
servsr => "http://servsr:8003/servmetric/json/"
}
automatic_retries => 3
schedule => { cron => "*/5 * * * *"}
"codec" => "json_lines"
}
}
filter {
date {
match => ["metric_starttime","YYYY-MM-dd HH:mm:ss"]
remove_field => ["metric_starttime"]
target => "metric_starttime_ts"
}
}
output {
stdout {
"codec" => "rubydebug"
}
}
So I am expecting that every 5 min, I should get 4 outputs. However, the output seems random in sequence of number of json output as below -
1st interval data
{
"metric_starttime_ts" => 2018-10-04T11:21:00.000Z,
"@version" => "1",
"servername" => "servsr ",
"server_online_gp" => 8,
"@timestamp" => 2018-10-04T11:30:00.629Z
}
{
"metric_starttime_ts" => 2018-10-04T11:21:00.000Z,
"@version" => "1",
"servername" => "servsd ",
"server_online_gp" => 9,
"@timestamp" => 2018-10-04T11:30:00.631Z
}
{
"metric_starttime_ts" => 2018-10-04T11:21:00.000Z,
"@version" => "1",
"servername" => "servsf ",
"server_online_gp" => 10,
"@timestamp" => 2018-10-04T11:30:00.631Z
}
2nd interval data
{
"metric_starttime_ts" => 2018-10-04T11:21:00.000Z,
"@version" => "1",
"servername" => "servsb ",
"server_online_gp" => 9,
"@timestamp" => 2018-10-04T11:35:00.140Z,
}
{
"metric_starttime_ts" => 2018-10-04T11:26:00.000Z,
"@version" => "1",
"servername" => "servsb ",
"server_online_gp" => 9,
"@timestamp" => 2018-10-04T11:35:00.161Z,
}
{
"metric_starttime_ts" => 2018-10-04T11:26:00.000Z,
"@version" => "1",
"servername" => "servsf ",
"server_online_gp" => 10,
"@timestamp" => 2018-10-04T11:35:00.173Z,
}
{
"metric_starttime_ts" => 2018-10-04T11:26:00.000Z,
"@version" => "1",
"servername" => "servsd ",
"server_online_gp" => 9,
"@timestamp" => 2018-10-04T11:35:00.175Z
}
3rd interval data
{
"metric_starttime_ts" => 2018-10-04T11:31:00.000Z,
"@version" => "1",
"servername" => "servsd ",
"server_online_gp" => 9,
"@timestamp" => 2018-10-04T11:40:00.135Z
}
{
"metric_starttime_ts" => 2018-10-04T11:26:00.000Z,
"@version" => "1",
"servername" => "servsr ",
"server_online_gp" => 8,
"@timestamp" => 2018-10-04T11:40:00.135Z
}
{
"metric_starttime_ts" => 2018-10-04T11:31:00.000Z,
"@version" => "1",
"servername" => "servsf ",
"server_online_gp" => 10,
"@timestamp" => 2018-10-04T11:40:00.139Z
}
{
"metric_starttime_ts" => 2018-10-04T11:31:00.000Z,
"@version" => "1",
"servername" => "servsr ",
"server_online_gp" => 8,
"@timestamp" => 2018-10-04T11:40:00.145Z
}
So basically in
1st interval - got data for servsr, servsd and servsf (so servsb missing)
2nd interval - got data for servsb (of 1st interval), servsb (current interval), servsf and servsd (servsr missing)
3rd interval - got data for servsd, servsr (2nd interval), servsf and servsr (for current interval) so servsb missing
So data gets collected and is not missed, however it's not coming in the correct server.
Can you please help.
Regards,
Harish