We have deployed filebeat as kubernetes daemon set and are using hint based configuration to send logs to logstash. Here is the filebeat configuration
filebeat.autodiscover:
providers:
- type: kubernetes
host: ${HOSTNAME}
hints.enabled: true
hints.default_config.enabled: false
labels.dedot: true
annotations.dedot: true
add_resource_metadata:
namespace:
enabled: true
processors:
- add_cloud_metadata:
output.logstash:
hosts: ["xxx:5044"]
Our nginx-deployments tag the pods like this
co.elastic.logs/enabled: "true"
co.elastic.logs/fileset.stderr: error
co.elastic.logs/fileset.stdout: ingress_controller
For the stdout strem everything is working fine, but with the stderr stream multiple separate lines get combined.
Example:
Container logs
{"log":"I0821 14:25:19.736405 6 flags.go:204] Watching for Ingress class: nginx-private\n","stream":"stderr","time":"2020-08-21T14:25:19.73649997Z"}
{"log":"W0821 14:25:19.736474 6 flags.go:207] Only Ingresses with class \"nginx-private\" will be processed by this Ingress controller\n","stream":"stderr","time":"2020-08-21T14:25:19.736514707Z"}
{"log":"W0821 14:25:19.737100 6 flags.go:249] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)\n","stream":"stderr","time":"2020-08-21T14:25:19.737170774Z"}
{"log":"W0821 14:25:19.737189 6 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.\n","stream":"stderr","time":"2020-08-21T14:25:19.737225145Z"}
{"log":"I0821 14:25:19.737410 6 main.go:220] Creating API client for https://10.127.32.1:443\n","stream":"stderr","time":"2020-08-21T14:25:19.737448109Z"}
{"log":"I0821 14:25:19.749068 6 main.go:264] Running in Kubernetes cluster version v1.16 (v1.16.10) - git (clean) commit f3add640dbcd4f3c33a7749f38baaac0b3fe810d - platform linux/amd64\n","stream":"stderr","time":"2020-08-21T14:25:19.749788766Z"}
{"log":"I0821 14:25:22.167797 6 status.go:86] new leader elected: ngix-ingress-private-nginx-ingress-controller-57f7bf5b8b-pms4w\n","stream":"stderr","time":"2020-08-21T14:25:22.167865205Z"}
Appear in Logstash as one long line with newline characters in between
"message" => "I0821 14:25:19.736405 6 flags.go:204] Watching for Ingress class: nginx-private\nW0821 14:25:19.736474 6 flags.go:207] Only Ingresses with class \"nginx-private\" will be processed by this Ingress controller\nW0821 14:25:19.737100 6 flags.go:249] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)\nW0821 14:25:19.737189 6 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.\nI0821 14:25:19.737410 6 main.go:220] Creating API client for https://10.127.32.1:443\nI0821 14:25:19.749068 6 main.go:264] Running in Kubernetes cluster version v1.16 (v1.16.10) - git (clean) commit f3add640dbcd4f3c33a7749f38baaac0b3fe810d - platform linux/amd64\nI0821 14:25:22.167797 6 status.go:86] new leader elected: ngix-ingress-private-nginx-ingress-controller-57f7bf5b8b-pms4w",
Any ideas/hints would be appreciated