# Filebeat OSS 7.7.0 not able to connect AWS Elasticsearch Service 7.7.0 **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [October 15, 2020, 9:27pm UTC](https://discuss.elastic.co/t/filebeat-oss-7-7-0-not-able-to-connect-aws-elasticsearch-service-7-7-0/252275 "2020-10-15T21:27:21Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Jigar\_Patel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jigar_patel1/32/47985_2.png) [@Jigar\_Patel1](https://discuss.elastic.co/u/Jigar_Patel1)\ **Post date:** [October 15, 2020, 9:27pm UTC](https://discuss.elastic.co/t/filebeat-oss-7-7-0-not-able-to-connect-aws-elasticsearch-service-7-7-0/252275/1 "2020-10-15T21:27:21Z")
I am using EC2 VM to send logs to AWS Elasticsearch Service 7.7.0. EC2 instance and Elastic Service can communicate via same VPC. But, filebeat OSS 7.7.0 not able to connect to Elasticsearch service 7.7.0. Any suggesting, what's wrong here? ```auto curl https://vpc-jdev-adfasdfadsfadsf.us-west-2.es.amazonaws.com { "name" : "3cdbd1b8903b670a529aa22dfa62f1d6", "cluster_name" : "6asdf18", "cluster_uuid" : "vC1-adf89Kg", "version" : { "number" : "7.7.0", "build_flavor" : "oss", "build_type" : "tar", "build_hash" : "unknown", "build_date" : "2020-08-18T20:35:37.721611Z", "build_snapshot" : false, "lucene_version" : "8.5.1", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" } ``` Filebeat config - ```auto cat filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/*.log - /var/log/httpd/*_log filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.name: "filebeat" setup.template.pattern: "filebeat-*" setup.template.fields: "fields.yml" setup.template.overwrite: true setup.template.settings: index.number_of_shards: 5 index.number_of_replicas: 1 output.elasticsearch: hosts: ["https://vpc-jdev-adfasdfadsfadsf.us-west-2.es.amazonaws.com"] index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}" processors: - add_host_metadata: ~ - add_cloud_metadata: ~ ``` I am getting connection error ```auto 2020-10-15T21:06:29.149Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":100,"time":{"ms":1}},"total":{"ticks":490,"time":{"ms":45},"value":490},"user":{"ticks":390,"time":{"ms":44}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":12},"info":{"ephemeral_id":"4a5dbb9f-c6ac-492b-aa0f-9e1c892d11e4","uptime":{"ms":270211}},"memstats":{"gc_next":41953184,"memory_alloc":21046160,"memory_total":46031072},"runtime":{"goroutines":48}},"filebeat":{"harvester":{"open_files":5,"running":5}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":3714}}},"registrar":{"states":{"current":8}},"system":{"load":{"1":0,"15":0.11,"5":0.12,"norm":{"1":0,"15":0.055,"5":0.06}}}}}} 2020-10-15T21:06:42.160Z ERROR [publisher_pipeline_output] pipeline/output.go:106 Failed to connect to backoff(elasticsearch(https://vpc-jdev-adfasdfadsfadsf.us-west-2.es.amazonaws.com:9200)): Get https://vpc-jdev-adfasdfadsfadsf.us-west-2.es.amazonaws.com:9200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2020-10-15T21:06:42.160Z INFO [publisher_pipeline_output] pipeline/output.go:99 Attempting to reconnect to backoff(elasticsearch(https://vpc-jdev-adfasdfadsfadsf.us-west-2.es.amazonaws.com:9200)) with 3 reconnect attempt(s) ``` ---
**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\ **Post date:** [October 16, 2020, 9:15am UTC](https://discuss.elastic.co/t/filebeat-oss-7-7-0-not-able-to-connect-aws-elasticsearch-service-7-7-0/252275/2 "2020-10-16T09:15:26Z")
It looks rather like a network issue. Did you try to telnet to the Elasticsearch endpoint? Also, please reverify security groups. ---
**Author:** ![Jigar\_Patel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jigar_patel1/32/47985_2.png) [@Jigar\_Patel1](https://discuss.elastic.co/u/Jigar_Patel1)\ **Post date:** [October 17, 2020, 12:16am UTC](https://discuss.elastic.co/t/filebeat-oss-7-7-0-not-able-to-connect-aws-elasticsearch-service-7-7-0/252275/3 "2020-10-17T00:16:13Z")
yes, telenet and curl both works. any idea, why port '9200' appends after host name even if it is 'https' call? I don't have any inbound rule for port 9200, do I need to add it? Also, do I need to use "cloud.id" instead of "output.elasticsearch"? ```auto telnet vpc-jdev-adfasdfadsfadsf.us-west-2.es.amazonaws.com 443 Trying 10.10.10.83... Connected vpc-jdev-adfasdfadsfadsf.us-west-2.es.amazonaws.com. Escape character is '^]'. ^] ``` ---
**Author:** ![Jigar\_Patel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jigar_patel1/32/47985_2.png) [@Jigar\_Patel1](https://discuss.elastic.co/u/Jigar_Patel1)\ **Post date:** [October 27, 2020, 4:51pm UTC](https://discuss.elastic.co/t/filebeat-oss-7-7-0-not-able-to-connect-aws-elasticsearch-service-7-7-0/252275/4 "2020-10-27T16:51:43Z")
@mtojek - any comments on my questions? ---
**Author:** ![Jigar\_Patel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jigar_patel1/32/47985_2.png) [@Jigar\_Patel1](https://discuss.elastic.co/u/Jigar_Patel1)\ **Post date:** [October 28, 2020, 4:41pm UTC](https://discuss.elastic.co/t/filebeat-oss-7-7-0-not-able-to-connect-aws-elasticsearch-service-7-7-0/252275/5 "2020-10-28T16:41:03Z")
Found out the missing config and need to run 'filebeat setup' ```auto setup.pack.security.enabled: false setup.xpack.graph.enabled: false setup.xpack.watcher.enabled: false setup.xpack.monitoring.enabled: false setup.xpack.reporting.enabled: false filebeat setup --pipelines --template --dashboards -c /etc/filebeat/filebeat.yml ``` Below are the references for the solution - > > > hello, I am a newbie to use ES and Kibana in Amazon Elasticsearch Service, which do not support xpack plugin. > I use... > **[Filebeat and AWS Elasticsearch](https://partiallydisassembled.net/posts/filebeat-kibana-aws.html)** > > Elasticsearch, Logstash and Kibana (or ELK) are standard tools for aggregating and monitoring server logs. This post details the steps I took to integrate Filebeat (the Elasticsearch log scraper) with an AWS-managed Elasticsearch instance operating... ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 18, 2020, 6:30am UTC](https://discuss.elastic.co/t/filebeat-oss-7-7-0-not-able-to-connect-aws-elasticsearch-service-7-7-0/252275/6 "2020-11-18T06:30:54Z")
Not related but did you look at [Cloud by Elastic](https://www.elastic.co/cloud), also available if needed from [AWS Marketplace](https://aws.amazon.com/marketplace/pp/B01N6YCISK) ? Cloud by elastic is one way to have access to **all features** , all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, Maps UI, Alerting and built-in solutions named [Observability](https://www.elastic.co/observability), [Security](https://www.elastic.co/security), [Enterprise Search](https://www.elastic.co/enterprise-search) and what is coming next 🙂 ... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 16, 2020, 6:31am UTC](https://discuss.elastic.co/t/filebeat-oss-7-7-0-not-able-to-connect-aws-elasticsearch-service-7-7-0/252275/7 "2020-12-16T06:31:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.