Logstash to aggregate logs

I am trying to aggregate raspberry pi logs into Logstash/ElasticSearch running in EKS.

This is my manifest file

apiVersion: v1
kind: ServiceAccount
metadata:
  name: logstash
  namespace: kube-logging
  labels:
    app: logstash

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-config
  namespace: kube-logging
  labels:
    app: logstash
data:
  logstash.conf: |-
    input {
      udp {
        port => 514
        type => syslog
      }
    }

    filter {
      if [type] == "syslog" {
        grok {
          match => { "message" => "%{SYSLOGLINE}" }
        }
        date {
          match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
      }
    }

    output {
      elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "logstash-%{+YYYY.MM.dd}"
        document_type => "system_logs"
      }
      stdout { codec => rubydebug }
    }

---

kind: Deployment
apiVersion: apps/v1beta1
metadata:
  name: logstash
  namespace: kube-logging
  labels:
    app: logstash
spec:
  template:
    metadata:
      labels:
        app: logstash
    spec:
      serviceAccountName: logstash
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: logstash
        image: docker.elastic.co/logstash/logstash:7.2.0
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: changeme
        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:
        ports:
        - name: logstash
          containerPort: 514
          protocol: UDP
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 800Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/logstash/conf.d/logstash.conf
          readOnly: true
          subPath: logstash.conf
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: logstash-config

---

kind: Service
apiVersion: v1
metadata:
  name: logstash
  namespace: kube-logging
  labels:
    app: logstash
spec:
  selector:
    app: logstash
  clusterIP: None
  ports:
    - port: 514

Application running on Raspberry Pi will be writing logs to the ingress endpoint of the logstash.

logstash container is running without any errors. but I am not able to see the indices in Kibana.

I am able to do telnet <endpoint> 514 but don't see anything for tcpdump -A -i any dst port 514 in logstash pod

What I am missing here ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.