Syslog to ECK

[msmill]

Hi!

We are running an ECK instance to gather logs from multiple K8s Clusters (not only the one it's running on), Winlogbeats agent on Windows VMs, and would like to send logs from network devices (syslog) to Elastic as well.

To achieve our current scenario Elastic is published with a LB service.

ECK is a way for us to host elastic service but we do not want to limit its usage to K8s workloads logs.

We need a syslog (UDP) endpoint and be able to parse this format.

We have tested different solutions but lack expertise to make an appropriate choice:

• syslog-ng to ECK (run in K8s, with LB Service to expose UDP port)
• logstash to ECK (same)
• Beats ? We have no idea if such module Palo Alto Networks module | Filebeat Reference [7.12] | Elastic could be used along with ECK

Thanks !

[msmill]

We also managed to run Beats with the example of Palo Alto. The specification is simple but ECK lacks example because it focuses on deploying filebeats for K8s logs, and not inputs that would be external to the cluster.

Also, a Filebeat pod created with ECK can be exposed externally with an appropriate service, an issue is open to make this service spec part of the beat CRD: Beat add spec for external service · Issue #3866 · elastic/cloud-on-k8s (github.com)

# CRD to create beats with ECK (Pod(s))
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
   # Unique name for a beat
   name: panos-filebeat
   # Same namespace as ECK
   namespace: eck-elasticsearch
spec:
   # type filebeat to use panos module https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-panw.html
   type: filebeat
   version: 7.12.0
   # Name of ECK' s elastic cluster: it will configure every setting/credentials inside the Beats to connect to ECK
   elasticsearchRef:
   name: elasticsearch
   # Same as above
   kibanaRef:
   name: kibana
   # Actual configuration of the beat. Will generate into a secret that will be mounted inside the Pod
   config:
 filebeat.modules:
   # Enable Palo Alto module that sets up appropriate log parsing from syslog udp input (default, udp/9001 on localhost)
 - module: panw
   panos:
    enabled: true
    # Make the UDP server listen on pod's IP
    var.syslog_host: 0.0.0.0
 output.elasticsearch:
    # Automatic configuration (elasticsearchRef) will import default CA of ECK. However we have our own certificate deployed for external access to elastic. Only workaround is to disable ssl verification. https://github.com/elastic/beats/issues/8164#issuecomment-597845134
    ssl.verification_mode: none
  # Choice of Deployment or DaemonSet to deploy the beat. We choose deployment and according specs
   deployment:
podTemplate:
  spec:
    # Configuration is mounted in /etc/beat.yaml, needs this to access the file
    securityContext:
      runAsUser: 0
    # Firewall sends log in its configured TZ without putting the TZ information in the log. We have to make the pod work in the same TZ.
    containers:
      - name: filebeat
        volumeMounts:
        - name: tz-config
          mountPath: /etc/localtime
    volumes:
    - name: tz-config
      hostPath:
       path: /usr/share/zoneinfo/Europe/Zurich
       type: File

# LoadBalancer Service to expose PanOS Filebeat Syslog UDP input externally and not supported YET in ECK's beat api
# https://github.com/elastic/cloud-on-k8s/issues/3866 Beat add spec for external service
apiVersion: v1
kind: Service
metadata:
  name: panos-beat-svc
spec:
  ports:
  - port: 9001
    protocol: UDP
    targetPort: 9001
  selector:
    beat.k8s.elastic.co/name: panos-filebeat
  type: LoadBalancer