ERR Failed to connect: Connection marked as failed because the onConnect callback failed

[alexus]

My environment:

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.4 (Maipo)
# rpm -q elasticsearch filebeat 
elasticsearch-6.0.0-1.noarch
filebeat-6.0.0-1.x86_64
# 

I keep getting following error inside of my /var/log/filebeat/filebeat:

2017-11-28T15:18:06-05:00 ERR Failed to connect: Connection marked as failed because the onConnect callback failed: Error loading pipeline for fileset auditd/log: This module requires the following Elasticsearch plugins: ingest-geoip. You can install them by running the following commands on all the Elasticsearch nodes:
sudo bin/elasticsearch-plugin install ingest-geoip

even though, I already have ingest-geoip plugin installed:

# ./bin/elasticsearch-plugin list
ingest-geoip
# 

Please advise.

[andrewkroh]

Can you also share the config you are using for Filebeat (filebeat export config). And test the outputs (filebeat test output).

[alexus]

filebeat export config:

# filebeat export config
filebeat:
  config:
    modules:
      path: /etc/filebeat/modules.d/*.yml
      reload:
        enabled: false
  prospectors:
  - enabled: false
    paths:
    - /var/log/*.log
    - /var/log/messages
    type: log
output:
  elasticsearch:
    hosts:
    - elasticsearch.xxxxx.local:9200
path:
  config: /etc/filebeat
  data: /var/lib/filebeat
  home: /usr/share/filebeat
  logs: /var/log/filebeat
setup:
  kibana:
    host: kibana.xxxxx.local:5601
  template:
    settings:
      index:
        number_of_shards: 3
# 

filebeat test output:

# filebeat test output
elasticsearch: http://elasticsearch.xxxxx.local:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: xx.xx.xxx.xxx
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 6.0.0
#

[andrewkroh]

That looks normal.

Is there more than one node in the cluster? Do all nodes have the plugin installed?

Can you query GET /_nodes/ingest and post the output.

[alexus]

ahh, maybe that's why) I just went ahead and installed it on 2nd node:

# ./bin/elasticsearch-plugin install ingest-geoip
-> Downloading ingest-geoip from elastic
[=================================================] 100%   
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed ingest-geoip
# 

GET /_nodes/ingest:

# curl --request GET localhost:9200/_nodes/ingest?pretty
{
  "_nodes" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "cluster_name" : "xxxxx",
  "nodes" : {
    "P3GVaQ1zRMCcbQr5yYnVtA" : {
      "name" : "7pvelk02.xxxxx.local",
      "transport_address" : "xx.xx.xxx.xxx:9300",
      "host" : "xx.xx.xxx.xxx",
      "ip" : "xx.xx.xxx.xxx",
      "version" : "6.0.0",
      "build_hash" : "8f0685b",
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "ingest" : {
        "processors" : [
          {
            "type" : "append"
          },
          {
            "type" : "convert"
          },
          {
            "type" : "date"
          },
          {
            "type" : "date_index_name"
          },
          {
            "type" : "dot_expander"
          },
          {
            "type" : "fail"
          },
          {
            "type" : "foreach"
          },
          {
            "type" : "grok"
          },
          {
            "type" : "gsub"
          },
          {
            "type" : "join"
          },
          {
            "type" : "json"
          },
          {
            "type" : "kv"
          },
          {
            "type" : "lowercase"
          },
          {
            "type" : "remove"
          },
          {
            "type" : "rename"
          },
          {
            "type" : "script"
          },
          {
            "type" : "set"
          },
          {
            "type" : "sort"
          },
          {
            "type" : "split"
          },
          {
            "type" : "trim"
          },
          {
            "type" : "uppercase"
          }
        ]
      }
    },
    "liqXQP8OTA6H6kamVxmtyw" : {
      "name" : "7pvelk01.xxxxx.local",
      "transport_address" : "xx.xx.xxx.xxx:9300",
      "host" : "xx.xx.xxx.xxx",
      "ip" : "xx.xx.xxx.xxx",
      "version" : "6.0.0",
      "build_hash" : "8f0685b",
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "ingest" : {
        "processors" : [
          {
            "type" : "append"
          },
          {
            "type" : "convert"
          },
          {
            "type" : "date"
          },
          {
            "type" : "date_index_name"
          },
          {
            "type" : "dot_expander"
          },
          {
            "type" : "fail"
          },
          {
            "type" : "foreach"
          },
          {
            "type" : "geoip"
          },
          {
            "type" : "grok"
          },
          {
            "type" : "gsub"
          },
          {
            "type" : "join"
          },
          {
            "type" : "json"
          },
          {
            "type" : "kv"
          },
          {
            "type" : "lowercase"
          },
          {
            "type" : "remove"
          },
          {
            "type" : "rename"
          },
          {
            "type" : "script"
          },
          {
            "type" : "set"
          },
          {
            "type" : "sort"
          },
          {
            "type" : "split"
          },
          {
            "type" : "trim"
          },
          {
            "type" : "uppercase"
          }
        ]
      }
    }
  }
}
#

[andrewkroh]

Restart that node because geoip isn't showing up on both nodes yet (according to that output).

[alexus]

there you go:

# curl --silent --request GET localhost:9200/_nodes/ingest?pretty | grep geoip
            "type" : "geoip"
            "type" : "geoip"
#