Hi!
In my kubernetes cluster I receive logs of some applications that have this format (specifically ambassador).
But I receive these logs inside the "message " field in my Elastic and I cannot easily filter later the messages that are of type 2xx, 3xx, 4xx, etc.
I understand that the correct way would be to parse the log before sending it, right? but how can I do it having this format?
ACCESS [2020-05-11T10:45:03.098Z] "GET /scregistry/sc-instances/xxx-contract-xxx-e940-4103-xxx-5012a8db3147 HTTP/2" 404 - 0 182 61 56 "xx.xx.173.127" "okhttp/3.9.1" "xxxx-a4e6-4402-xxx-b1845d7a9f17" "api-xxx.xxx-test.com " "xx.xx.20.156:80"
Can you help me?
Thank you
jsoriano
(Jaime Soriano)
May 11, 2020, 3:02pm
2
Hey @David_Oceans ,
Are you using autodiscover to collect the logs from your cluster? If that is the case you can setup different configurations per container to parse their logs.
For many services there are predefined modules that can be used, there are no module yet for ambassador, but maybe other modules for web servers as the one for apache or nginx are able to parse these logs.
If not, you can also try to parse it using processors, for example the dissect one could be able to parse these logs.
If filebeat processors are not enough, you can also use ingest pipelines .
In any case, feel free to open a new issue requesting support for ambassador, describing your use case.
Thanks!
Hi,
I was investigating and try differents ways to parser my nginx application, but didn't works me.
I'm using docker.elastic.co/beats/filebeat:7.3.2
This is my filebeat configmap
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: logging
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
multiline.negate: false
multiline.match: after
processors:
- add_kubernetes_metadata:
in_cluster: true
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
processors:
- add_cloud_metadata:
- add_host_metadata:
- decode_json_fields:
fields: ["message"]
target: "test"
overwrite_keys: true
- add_fields:
target: ''
fields:
gkeclustername: xxx-apps-stage
cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}
setup.ilm.rollover_alias: "${INDEX_NAME_CLUSTER}"
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
This is my deployment configuration (helm template)
# Source: APP_NAME/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: APP_NAME
labels:
app.kubernetes.io/name: APP_NAME
helm.sh/chart: APP_NAME-0.0.1
app.kubernetes.io/instance: release-name
app.kubernetes.io/version: "1.0"
app.kubernetes.io/managed-by: Tiller
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: APP_NAME
app.kubernetes.io/instance: release-name
template:
metadata:
labels:
app.kubernetes.io/name: APP_NAME
app.kubernetes.io/instance: release-name
annotations:
co.elastic.logs/fileset.stderr: error
co.elastic.logs/fileset.stdout: access
co.elastic.logs/module: nginx
spec:
containers:
- name: APP_NAME
image: ":"
imagePullPolicy: Always
ports:
- name: http
containerPort: 80
protocol: TCP
envFrom:
- configMapRef:
name: global-config-env
- configMapRef:
name: release-name-configmap
- secretRef:
name: release-name-secrets
livenessProbe:
httpGet:
path: /status/health
port: http
initialDelaySeconds: 90
periodSeconds: 20
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: /status/health
port: http
initialDelaySeconds: 30
periodSeconds: 20
timeoutSeconds: 10
successThreshold: 1
failureThreshold: 5
resources:
requests:
cpu: 40m
memory: 128Mi
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.google.com/gke-nodepool
operator: In
values:
- preemptible-node-pool
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/instance: APP_NAME
topologyKey: kubernetes.io/hostname
tolerations:
- effect: NoSchedule
key: preemptible
operator: Equal
value: "true"
But when I see the logs in kibana I still viewing the message in text plain
10.31.0.5 - - [13/May/2020:14:03:47 +0000] "GET /status/health HTTP/1.1" 200 7473 "-" "kube-probe/1.14+" "-"
I have to do something in kibana to refresh index or something? the filebeat and annotations are correct?
Thank you very much
New advances.
If I have this configured
filebeat.autodiscover:
But also this
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
multiline.negate: false
multiline.match: after
processors:
- add_kubernetes_metadata:
in_cluster: true
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
In fact when you download the latest version you see a comment
curl -L -O https://raw.githubusercontent.com/elastic/beats/master/deploy/kubernetes/filebeat-kubernetes.yaml
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
#filebeat.autodiscover:
# providers:
# - type: kubernetes
# node: ${NODE_NAME}
# hints.enabled: true
# hints.default_config:
# type: container
# paths:
# - /var/log/containers/*${data.kubernetes.container.id}.log
So, I get the merge, and this should be the right way, no?
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
hints.default_config:
type: container
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
multiline.negate: false
multiline.match: after
Thank you
jsoriano
(Jaime Soriano)
May 14, 2020, 3:16pm
5
Does you last configuration work?
Multipart message I'm not sure.
Looks good the configuration with indentation?
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: logging
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
templates:
- condition:
equals:
kubernetes.labels.app: nginx-ingress
config:
- module: nginx
ingress_controller:
enabled: true
input:
type: container
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
hints.default_config:
type: container
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
multiline.negate: false
multiline.match: after
processors:
- add_cloud_metadata:
- add_host_metadata:
- decode_json_fields:
fields: ["message"]
target: "custom"
overwrite_keys: true
- add_fields:
target: ''
fields:
gkeclustername: xxx-apps-stage
Thank you
jsoriano
(Jaime Soriano)
May 15, 2020, 12:06pm
7
Indentation looks good, yes.
I recommend you to use hints OR templates, but not both things. They work independently and they can create then multiple configurations for the same container. For example with this configuration, a pod with the app: nginx-ingress
label would have two configurations, one with the nginx module, and another one with the container input. This can lead to unexpected results.
If you use hints, you can add these annotations into the ingress controller pod so its logs are collected with the nginx module:
co.elastic.logs/module: nginx
co.elastic.logs/fileset.stdout: ingress_controller
co.elastic.logs/fileset.stderr: error
system
(system)
Closed
June 12, 2020, 12:06pm
8
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.