Filebeat OSS 7.7.0 not able to connect AWS Elasticsearch Service 7.7.0

Elastic Stack Beats
1

I am using EC2 VM to send logs to AWS Elasticsearch Service 7.7.0. EC2 instance and Elastic Service can communicate via same VPC. But, filebeat OSS 7.7.0 not able to connect to Elasticsearch service 7.7.0.
Any suggesting, what's wrong here?

curl https://vpc-jdev-adfasdfadsfadsf.us-west-2.es.amazonaws.com
{
  "name" : "3cdbd1b8903b670a529aa22dfa62f1d6",
  "cluster_name" : "6asdf18",
  "cluster_uuid" : "vC1-adf89Kg",
  "version" : {
    "number" : "7.7.0",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2020-08-18T20:35:37.721611Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Filebeat config -

cat filebeat.yml
filebeat.inputs:

- type: log
  enabled: true
  paths:
    - /var/log/*.log
    - /var/log/httpd/*_log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"
setup.template.fields: "fields.yml"
setup.template.overwrite: true

setup.template.settings:
  index.number_of_shards: 5
  index.number_of_replicas: 1

output.elasticsearch:
  hosts: ["https://vpc-jdev-adfasdfadsfadsf.us-west-2.es.amazonaws.com"]
  index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

I am getting connection error

2020-10-15T21:06:29.149Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":100,"time":{"ms":1}},"total":{"ticks":490,"time":{"ms":45},"value":490},"user":{"ticks":390,"time":{"ms":44}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":12},"info":{"ephemeral_id":"4a5dbb9f-c6ac-492b-aa0f-9e1c892d11e4","uptime":{"ms":270211}},"memstats":{"gc_next":41953184,"memory_alloc":21046160,"memory_total":46031072},"runtime":{"goroutines":48}},"filebeat":{"harvester":{"open_files":5,"running":5}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":3714}}},"registrar":{"states":{"current":8}},"system":{"load":{"1":0,"15":0.11,"5":0.12,"norm":{"1":0,"15":0.055,"5":0.06}}}}}}
2020-10-15T21:06:42.160Z        ERROR   [publisher_pipeline_output]     pipeline/output.go:106  Failed to connect to backoff(elasticsearch(https://vpc-jdev-adfasdfadsfadsf.us-west-2.es.amazonaws.com:9200)): Get https://vpc-jdev-adfasdfadsfadsf.us-west-2.es.amazonaws.com:9200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-10-15T21:06:42.160Z        INFO    [publisher_pipeline_output]     pipeline/output.go:99   Attempting to reconnect to backoff(elasticsearch(https://vpc-jdev-adfasdfadsfadsf.us-west-2.es.amazonaws.com:9200)) with 3 reconnect attempt(s)
2

It looks rather like a network issue. Did you try to telnet to the Elasticsearch endpoint? Also, please reverify security groups.

3

yes, telenet and curl both works.
any idea, why port '9200' appends after host name even if it is 'https' call? I don't have any inbound rule for port 9200, do I need to add it?
Also, do I need to use "cloud.id" instead of "output.elasticsearch"?

telnet vpc-jdev-adfasdfadsfadsf.us-west-2.es.amazonaws.com 443
Trying 10.10.10.83...
Connected vpc-jdev-adfasdfadsfadsf.us-west-2.es.amazonaws.com.
Escape character is '^]'.
^]
4

@mtojek - any comments on my questions?

5

Found out the missing config and need to run 'filebeat setup'

setup.pack.security.enabled: false
setup.xpack.graph.enabled: false
setup.xpack.watcher.enabled: false
setup.xpack.monitoring.enabled: false
setup.xpack.reporting.enabled: false

filebeat setup --pipelines --template --dashboards -c /etc/filebeat/filebeat.yml

Below are the references for the solution -


6

Not related but did you look at Cloud by Elastic, also available if needed from AWS Marketplace ?

Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, Maps UI, Alerting and built-in solutions named Observability, Security, Enterprise Search and what is coming next :slight_smile: ...

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.