I'm hitting this error logs randomly while using "logstash filters elasticsearch"
[2017-04-29T16:13:16,826][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-*", :query=>"Event:\"Start Event\"" AND ID:\"af099bf8-54be-4eaf-9ca5-3900c4e2c355\"", :event=>2017-04-29T09:21:46.000Z %{host} %{message}, :error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>}
[2017-04-29T16:13:30,162][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-*", :query=>"Event:\"Start Event\"" AND ID:\"73a20922-9d23-4a13-b142-f6ebd02384c0\"", :event=>2017-04-29T06:31:20.000Z %{host} %{message}, :error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>}
[2017-04-29T16:13:30,163][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-*", :query=>"Event:\"Start Event\"" AND ID:\"f0b6492e-48ce-4e44-9800-efb67c207195\"", :event=>2017-04-29T04:33:57.000Z %{host} %{message}, :error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>}
[2017-04-29T16:13:30,900][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-*", :query=>"Event:\"Start Event\"" AND ID:\"18261d24-9f71-4c73-98ec-a418a2eb526f\"", :event=>2017-04-29T09:42:12.000Z %{host} %{message}, :error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>}
[2017-04-29T16:14:50,905][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-*", :query=>"Event:\"Start Event\"" AND ID:\"48e949ca-718b-40e2-b03a-f9a0b0fec829\"", :event=>2017-04-29T14:06:51.000Z %{host} %{message}, :error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>}
[2017-04-29T16:15:16,893][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-*", :query=>"Event:\"Start Event\"" AND ID:\"3969820231\"", :event=>2017-04-29T12:04:52.000Z %{host} %{message}, :error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>}
[2017-04-29T16:15:28,004][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-*", :query=>"Event:\"Start Event\"" AND ID:\"8ddf65b4-e332-4a97-8fc4-bcf00cd8e2a2\"", :event=>2017-04-29T12:04:47.000Z %{host} %{message}, :error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>}
[2017-04-29T16:15:34,540][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-*", :query=>"Event:\"Start Event\"" AND ID:\"9ea3c4e4-4aa8-471a-80a1-c38460bb2dcf\"", :event=>2017-04-29T13:49:45.000Z %{host} %{message}, :error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>}
[2017-04-29T16:17:04,317][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-*", :query=>"Event:\"Start Event\"" AND ID:\"b88aadb7-6568-4e10-a5b5-15d2aa6eb805\"", :event=>2017-04-29T07:47:39.000Z %{host} %{message}, :error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>}
[2017-04-29T16:17:15,725][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-*", :query=>"Event:\"Start Event\"" AND ID:\"62fbf3df-9d1f-4746-966f-e31cf5857874\"", :event=>2017-04-29T03:34:51.000Z %{host} %{message}, :error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>}
[2017-04-29T16:18:52,755][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-*", :query=>"Event:\"Start Event\"" AND ID:\"f77a6fe3-d636-4f09-aa5a-6b50fef22c24\"", :event=>2017-04-29T11:31:43.000Z %{host} %{message}, :error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>}
the query works fine & many other events could be fetched... but it sometimes fails & emit such error logs.
Ubuntu 16.04.2
logstash Version: 1:5.3.2-1 (the latest from deb repository)
logstash-filter-elasticsearch (3.1.1)
elasticsearch Version: 5.3.2
filter{
elasticsearch {
hosts => ["localhost:9200"]
query => 'Event:"Start Event" AND ID:"%{ID}"'
fields => { "@timestamp" => "starttime" }
enable_sort => false
index => "logstash-*"
tag_on_failure => [ "omitme" ]
add_tag => [ "reindex2_catchstarttime" , "%{starttime}" ]
}
}
I'm using 1 second sleep before the filter, & this actually degrades the performance. & can't increase it more.
please help