We are running an ECK instance to gather logs from multiple K8s Clusters (not only the one it's running on), Winlogbeats agent on Windows VMs, and would like to send logs from network devices (syslog) to Elastic as well.
To achieve our current scenario Elastic is published with a LB service.
ECK is a way for us to host elastic service but we do not want to limit its usage to K8s workloads logs.
We need a syslog (UDP) endpoint and be able to parse this format.
We have tested different solutions but lack expertise to make an appropriate choice:
We also managed to run Beats with the example of Palo Alto. The specification is simple but ECK lacks example because it focuses on deploying filebeats for K8s logs, and not inputs that would be external to the cluster.
Also, a Filebeat pod created with ECK can be exposed externally with an appropriate service, an issue is open to make this service spec part of the beat CRD: Beat add spec for external service · Issue #3866 · elastic/cloud-on-k8s (github.com)
# CRD to create beats with ECK (Pod(s))
# Unique name for a beat
# Same namespace as ECK
# type filebeat to use panos module https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-panw.html
# Name of ECK' s elastic cluster: it will configure every setting/credentials inside the Beats to connect to ECK
# Same as above
# Actual configuration of the beat. Will generate into a secret that will be mounted inside the Pod
# Enable Palo Alto module that sets up appropriate log parsing from syslog udp input (default, udp/9001 on localhost)
- module: panw
# Make the UDP server listen on pod's IP
# Automatic configuration (elasticsearchRef) will import default CA of ECK. However we have our own certificate deployed for external access to elastic. Only workaround is to disable ssl verification. https://github.com/elastic/beats/issues/8164#issuecomment-597845134
# Choice of Deployment or DaemonSet to deploy the beat. We choose deployment and according specs
# Configuration is mounted in /etc/beat.yaml, needs this to access the file
# Firewall sends log in its configured TZ without putting the TZ information in the log. We have to make the pod work in the same TZ.
- name: filebeat
- name: tz-config
- name: tz-config
# LoadBalancer Service to expose PanOS Filebeat Syslog UDP input externally and not supported YET in ECK's beat api
# https://github.com/elastic/cloud-on-k8s/issues/3866 Beat add spec for external service
- port: 9001
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.