Hi,
I am using 7.1.1 version elastic stack setup. Filebeat running on kubernetes cluster as pod and sending logs to ELK.
I could see all pod logs are in JSON when opening in vi editor.
example
sudo vi /datavg/docker/containers/7c012dae82f0d9bb8a36b84e861eeaa9f6601fd3e642c7a2e0b34fab0722dc65/7c012dae82f0d9bb8a36b84e861eeaa9f6601fd3e642c7a2e0b34fab0722dc65-json.log
But Kibana shows logs in plaintext.
Filebeat yml file
`> data:
kubernetes.yml: |-
- type: docker
containers.ids:
- "*"
processors:
- add_kubernetes_metadata:
in_cluster: true
fields:
cluster-name: pes
`
Thanks
Tried by adding
json.keys_under_root: true
json.add_error_key: true
but not working
kvch
(Noémi Ványi)
August 9, 2019, 3:00pm
3
Could you please share you configuration and format it using </>
?
Hi
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.registry.file_permissions: 0600
filebeat.registry.path: /datavg/filebeat/registry
filebeat.config:
inputs:
# Mounted `filebeat-inputs` configmap:
path: ${path.config}/inputs.d/*.yml
# Reload inputs configs as they change:
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
# To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this:
# Note the annotations allowed on a pod for customizing filebeat behavior:
# https://www.elastic.co/guide/en/beats/filebeat/master/configuration-autodiscover-hints.html
#filebeat.autodiscover:
# providers:
# - type: kubernetes
# hints.enabled: true
processors:
- add_cloud_metadata:
output.logstash:
hosts: ['${LOGSTASH_HOST}:${LOGSTASH_PORT:5044}']
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-inputs
namespace: kube-system
labels:
k8s-app: filebeat
data:
kubernetes.yml: |-
- type: docker
containers.ids:
- "*"
processors:
- add_kubernetes_metadata:
in_cluster: true
fields:
cluster-name: pet
---
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-host
namespace: kube-system
labels:
k8s-app: filebeat
data:
logstash-host: "XXXXXXXXXXXXXXXXXXX"
#logstash-host: "logstash.default"
logstash-port: "5044"
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
spec:
template:
metadata:
labels:
k8s-app: filebeat
spec:
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
imagePullSecrets:
- name: dockerregcredkub
containers:
- name: filebeat
image: XXXXXXXXXXXXX/filebeat:7.1.1
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: LOGSTASH_HOST
valueFrom:
configMapKeyRef:
name: logstash-host
key: logstash-host
- name: LOGSTASH_PORT
valueFrom:
configMapKeyRef:
name: logstash-host
key: logstash-port
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: inputs
mountPath: /usr/share/filebeat/inputs.d
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: datavgdockercontainers
mountPath: /datavg/docker/containers/
readOnly: true
- name: datavgdockerfilebeat
mountPath: /datavg/filebeat/
readOnly: false
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /datavg/docker/containers/
- name: datavgdockercontainers
hostPath:
path: /datavg/docker/containers/
- name: datavgdockerfilebeat
hostPath:
path: /datavg/filebeat/
- name: inputs
configMap:
defaultMode: 0600
name: filebeat-inputs
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
imagePullSecrets:
- name: dockerregcred
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: kube-system
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
---
Thanks for checking this.
(I have removed json from yml.)
system
(system)
Closed
September 7, 2019, 12:00pm
5
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.