# How to integrate from EKS to ELK **URL:** \ **Category:** Kibana\ **Tags:** docker\ **Created:** [December 16, 2022, 2:31pm UTC](https://discuss.elastic.co/t/how-to-integrate-from-eks-to-elk/321414 "2022-12-16T14:31:55Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![yasar](https://avatars.discourse-cdn.com/v4/letter/y/dc4da7/32.png) [@yasar](https://discuss.elastic.co/u/yasar)\ **Post date:** [December 16, 2022, 2:31pm UTC](https://discuss.elastic.co/t/how-to-integrate-from-eks-to-elk/321414/1 "2022-12-16T14:31:55Z")
Hi team, We would like to explore about the integrate from EKS to ELK. Normally, we have getting the logs from below path. Logs Flow - FileBeat/MetricBeat -\> AWS Kafka -\> Logstash -\> ES -\> Kibana So, we would like to integrate the EKS to ELK. So, we have already getting the docker pushing logs to ELK. The same manner, we need to connect the log flow between EKS to AWS Kafka. Could someone please guide, how could we integrate EKS to ELK that would be helpful for us to explore more on this. Thanks & Regards, Yasar Arafaath A. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 16, 2022, 11:50pm UTC](https://discuss.elastic.co/t/how-to-integrate-from-eks-to-elk/321414/2 "2022-12-16T23:50:27Z")
Perhaps Look At Elastic Agent Fleet Managed > **[Run Elastic Agent on Kubernetes managed by Fleet | Fleet and Elastic Agent...](https://www.elastic.co/guide/en/fleet/current/running-on-kubernetes-managed-by-fleet.html)** Or Stand Alone since you probably want to send to Kafka > **[Run Elastic Agent standalone on Kubernetes | Fleet and Elastic Agent Guide...](https://www.elastic.co/guide/en/fleet/current/running-on-kubernetes-standalone.html)** You can still use Beats there are docs on how to deploy on Kubernetes > **[Run Metricbeat on Kubernetes | Metricbeat Reference \[8.5\] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html)** Or if you are shipping logs in another manner to S3 you could probably forward with > **[Elastic and AWS Serverless Application Repository (SAR): Speed time to...](https://www.elastic.co/blog/elastic-and-aws-serverless-application-repository-speed-time-to-actionable-insights-with-frictionless-log-ingestion-from-amazon-s3)** > > Ingest logs frictionlessly from Amazon S3 with Elastic’s new serverless forwarder available in the AWS Serverless Application Repository (SAR). Accelerate actionable insights with a unified view across your multi-cloud environments. ---
**Author:** ![yasar](https://avatars.discourse-cdn.com/v4/letter/y/dc4da7/32.png) [@yasar](https://discuss.elastic.co/u/yasar)\ **Post date:** [December 20, 2022, 2:31pm UTC](https://discuss.elastic.co/t/how-to-integrate-from-eks-to-elk/321414/3 "2022-12-20T14:31:28Z")
Hi Stephen, I have read all those your documents. I am bit confused, could you please tell me, which is the document working with my requirements. we have already getting the docker pushing logs to ELK from one of the application team. But we don't have the much knowledge about docker. So, The same manner, we need to connect the log flow between EKS to AWS Kafka. My queries are below [1] How could we get the docker logs from ELK integration. [2] How could we integrate the docker logs from EKS to Kafka. [3] For EKS, we need to install the Kubernetes.. if yes, where we need to do that ? and final [4] How could we integrate from EKS to ELK. Regards, Yasar Arafaath A ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 20, 2022, 6:11pm UTC](https://discuss.elastic.co/t/how-to-integrate-from-eks-to-elk/321414/4 "2022-12-20T18:11:16Z")
> [@yasar](#): > > [1] How could we get the docker logs from ELK integration. > [@yasar](#): > > So, we would like to integrate the EKS to ELK. So, we have already getting the docker pushing logs to ELK. The same manner, we need to connect the log flow between EKS to AWS Kafka. I am confused those 2 sentences seems to contradict each other. So If you already have docker logs pushing to Elastic then just changes the output to Kafka... what are you using to ship the logs... > [@yasar](#): > > [2] How could we integrate the docker logs from EKS to Kafka. If you follow [this](https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html) and just changes the output to Kafka that will get your logs from EKS to Kafka EKS -\> Filebeat (log) -\> Kafka Here is the manifest: `https://raw.githubusercontent.com/elastic/beats/8.5/deploy/kubernetes/filebeat-kubernetes.yaml` Change this part along with the ```auto output.elasticsearch: hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}'] username: ${ELASTICSEARCH_USERNAME} password: ${ELASTICSEARCH_PASSWORD} ``` to [Kafka Output.](https://www.elastic.co/guide/en/beats/metricbeat/current/kafka-output.html) Example ```auto output.kafka: # initial brokers for reading cluster metadata hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092"] # message topic selection + partitioning topic: '%{[fields.log_topic]}' partition.round_robin: reachable_only: false required_acks: 1 compression: gzip max_message_bytes: 1000000 ``` > [@yasar](#): > > [4] How could we integrate from EKS to ELK. Your flow is quite complex for just getting started... and if you do not have experience with the components it will be quite hard. You are going to need to "Dig in and Test / Try / Fix" it is the only way... When I see this `EKS Logs Flow -> FileBeat/MetricBeat -> AWS Kafka -> Logstash -> ES -> Kibana` I suggest this as a progression.. ```auto EKS Logs Flow -> FileBeat/MetricBeat -> ES -> Kibana EKS Logs Flow -> FileBeat/MetricBeat -> Logstash -> ES -> Kibana EKS Logs Flow -> FileBeat/MetricBeat -> AWS Kafka -> Logstash -> ES -> Kibana ``` > [@yasar](#): > > [3] For EKS, we need to install the Kubernetes.. if yes, where we need to do that ? EKS is Kubernetes... I am confused... Good Luck! come back with specific questions on your journey. ---
**Author:** ![yasar](https://avatars.discourse-cdn.com/v4/letter/y/dc4da7/32.png) [@yasar](https://discuss.elastic.co/u/yasar)\ **Post date:** [December 23, 2022, 2:12pm UTC](https://discuss.elastic.co/t/how-to-integrate-from-eks-to-elk/321414/5 "2022-12-23T14:12:27Z")
Hi Stephen, Thanks for the helpful. Please find my below query. Could you please guide us, how can we install the Filebeat for Docker/Docker images. So, that we can accomplish the first step right.. ?? Please help us some Filebeat config that would be helpful for us Thanks, Yasar Arafaath ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 23, 2022, 3:24pm UTC](https://discuss.elastic.co/t/how-to-integrate-from-eks-to-elk/321414/6 "2022-12-23T15:24:49Z")
> [@yasar](#): > > Could you please guide us, how can we install the Filebeat for Docker/Docker images. So, that we can accomplish the first step right.. ?? I don't know what that means. Do you want install filebeat on kubernetes? If so, here are the directions. You need to actually try and then ask questions. I'm not going to try to guess and create a step-by-step instructions for you. That's what we have documentation for. > **[Run Filebeat on Kubernetes | Filebeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html)** You will see the people that get good answers. Provide specific questions and then specific configurations and then specific results, logs, errors, etc. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 20, 2023, 3:25pm UTC](https://discuss.elastic.co/t/how-to-integrate-from-eks-to-elk/321414/7 "2023-01-20T15:25:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.