I am trying to aggregate raspberry pi logs into Logstash/ElasticSearch running in EKS.
This is my manifest file
apiVersion: v1
kind: ServiceAccount
metadata:
name: logstash
namespace: kube-logging
labels:
app: logstash
---
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-config
namespace: kube-logging
labels:
app: logstash
data:
logstash.conf: |-
input {
udp {
port => 514
type => syslog
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGLINE}" }
}
date {
match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "logstash-%{+YYYY.MM.dd}"
document_type => "system_logs"
}
stdout { codec => rubydebug }
}
---
kind: Deployment
apiVersion: apps/v1beta1
metadata:
name: logstash
namespace: kube-logging
labels:
app: logstash
spec:
template:
metadata:
labels:
app: logstash
spec:
serviceAccountName: logstash
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: logstash
image: docker.elastic.co/logstash/logstash:7.2.0
env:
- name: ELASTICSEARCH_HOST
value: elasticsearch
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: changeme
- name: ELASTIC_CLOUD_ID
value:
- name: ELASTIC_CLOUD_AUTH
value:
ports:
- name: logstash
containerPort: 514
protocol: UDP
securityContext:
runAsUser: 0
resources:
limits:
memory: 800Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/logstash/conf.d/logstash.conf
readOnly: true
subPath: logstash.conf
volumes:
- name: config
configMap:
defaultMode: 0600
name: logstash-config
---
kind: Service
apiVersion: v1
metadata:
name: logstash
namespace: kube-logging
labels:
app: logstash
spec:
selector:
app: logstash
clusterIP: None
ports:
- port: 514
Application running on Raspberry Pi will be writing logs to the ingress endpoint of the logstash
.
logstash container is running without any errors. but I am not able to see the indices in Kibana.
I am able to do telnet <endpoint> 514
but don't see anything for tcpdump -A -i any dst port 514
in logstash pod
What I am missing here ?