Hello everyone, i've create this topic because i got a problem with the utilisation of the filter plugin elasticsearch.
Here is my situation :
I use elasticsearch, logstash and kibana in version 5.5.0 on a Windows Server 2012 R2.
I have a logstash instance that retrieves data from a postgresql database every minute (about 4000 input).
Then I use the plugin filter elasticsearch to retrieve data from an index (let's call it "index-2") in order to aggregate them with the source of my logstash instance.
Here is my config for logstash :
input
{
jdbc
{
jdbc_connection_string => "CONNECTION_STRING"
jdbc_user => "POSTGRESQL_USER"
jdbc_password => "POSTRGRESQL_PASSWORD"
jdbc_driver_library => "PATH_to_the_driver"
jdbc_driver_class => "org.postgresql.Driver"
schedule => "* * * * *"
statement => "SELECT * FROM public.vue ;"
}
}
filter
{
elasticsearch
{
hosts => ["IP_SRV_ELASTIC"]
index => "INDEX-2"
query => "id:%{id}"
tag_on_failure => [ "erreur_elastic_filter_plugin" ]
fields => { "field_1" => "field_1"
"field_2" => "field_2"
"field_3" => "field_3"}
}
}
output
{
elasticsearch
{
index => "INDEX-1"
hosts => "IP_SRV_ELASTIC"
document_id => "month_%{id}_%{+YYYY.MM.dd.HH.mm}"
template => "PATH_TO_TEMPLATE"
template_overwrite => true
template_name => "TEMPLATE_NAME_IN_ELASTIC"
}
}
The problem is that for some entries the elasticsearch filter plugin has an error (about 20 errors per minute). The error is :
17:18:02.079 [[main]>worker2] WARN logstash.filters.elasticsearch - Failed to query elasticsearch for previous event {:index=>"INDEX-2", :query=>"id:VALEUR_ID", :event=>2017-07-24T15:18:00.564Z %{host} %{message},error=>#<Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.>}
I've created a visualization based on the data from INDEX-1 that shows the problem is not from a specific value (see below).
Moreover i've checked that the data i want is present in INDEX-2.
So can you help me to resolve my problem ?