Logstash filter elasticsearch : "Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool"


Hello everyone, i've create this topic because i got a problem with the utilisation of the filter plugin elasticsearch.

Here is my situation :

I use elasticsearch, logstash and kibana in version 5.5.0 on a Windows Server 2012 R2.

I have a logstash instance that retrieves data from a postgresql database every minute (about 4000 input).
Then I use the plugin filter elasticsearch to retrieve data from an index (let's call it "index-2") in order to aggregate them with the source of my logstash instance.

Here is my config for logstash :

        jdbc_connection_string => "CONNECTION_STRING"
        jdbc_user => "POSTGRESQL_USER"
	jdbc_password => "POSTRGRESQL_PASSWORD"
        jdbc_driver_library => "PATH_to_the_driver"
        jdbc_driver_class => "org.postgresql.Driver"
	schedule => "* * * * *"
        statement => "SELECT * FROM public.vue ;"

		hosts => ["IP_SRV_ELASTIC"]
		index => "INDEX-2"
		query => "id:%{id}"
		tag_on_failure => [ "erreur_elastic_filter_plugin" ]
		fields => { "field_1" => "field_1"
			    "field_2" => "field_2"
			    "field_3" => "field_3"}

			index => "INDEX-1"
			hosts => "IP_SRV_ELASTIC"
			document_id => "month_%{id}_%{+YYYY.MM.dd.HH.mm}"
			template => "PATH_TO_TEMPLATE"
			template_overwrite => true
			template_name => "TEMPLATE_NAME_IN_ELASTIC"

The problem is that for some entries the elasticsearch filter plugin has an error (about 20 errors per minute). The error is :

17:18:02.079 [[main]>worker2] WARN logstash.filters.elasticsearch - Failed to query elasticsearch for previous event {:index=>"INDEX-2", :query=>"id:VALEUR_ID", :event=>2017-07-24T15:18:00.564Z %{host} %{message},error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>}

I've created a visualization based on the data from INDEX-1 that shows the problem is not from a specific value (see below).
Moreover i've checked that the data i want is present in INDEX-2.

So can you help me to resolve my problem ?

(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.