Hi Team,
Can you please let us know the difference between filebeat version 7.9.3 and 8.2.0. Will this create issue in Available fields in Elastic search.
we have issue in Available fields not showing in PROD those required, but they are showing in UAT. Even we have same configurations in PROD and UAT.
Below Fields are coming in UAT (FileBeat agent version 8.2.0) not coming in PROD (Filebeat Agent version 7.9.3).
kubernetes.namespace_labels.kubernetes_io/metadata_name
kubernetes.namespace_uid
kubernetes.node.hostname
kubernetes.node.labels.beta_kubernetes_io/arch
kubernetes.node.labels.beta_kubernetes_io/instance-type
kubernetes.node.labels.beta_kubernetes_io/os
kubernetes.node.labels.eks_amazonaws_com/capacityType
kubernetes.node.labels.eks_amazonaws_com/nodegroup
kubernetes.node.labels.eks_amazonaws_com/nodegroup-image
kubernetes.node.labels.eks_amazonaws_com/sourceLaunchTemplateId
kubernetes.node.labels.eks_amazonaws_com/sourceLaunchTemplateVersion
kubernetes.node.labels.failure-domain_beta_kubernetes_io/region
kubernetes.node.labels.failure-domain_beta_kubernetes_io/zone
kubernetes.node.labels.k8s_io/cloud-provider-aws
kubernetes.node.labels.kubernetes_io/arch
kubernetes.node.labels.kubernetes_io/hostname
kubernetes.node.labels.kubernetes_io/os
kubernetes.node.labels.node_kubernetes_io/instance-type
kubernetes.node.labels.topology_kubernetes_io/region
kubernetes.node.labels.topology_kubernetes_io/zone
kubernetes.node.labels.vpc_amazonaws_com/eniConfig
kubernetes.node.labels.vpc_amazonaws_com/has-trunk-attached
kubernetes.node.uid
kubernetes.pod.ip
we have below filter in Logstash.
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
target => "timestamp"
}
if ([message] =~ /nested exception/) {
mutate {
add_field => ["nested_exception", "nested exception"]
}
}
}
else if (([kubernetes][container][name] == "controller") and [kubernetes][namespace] == "Prod") {
grok {
match => [ "message" , '%{IPORHOST:clientip} - %{USER:user_name} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response_code:int} (?:-|%{NUMBER:bytes:int}) "(?:%{GREEDYDATA:referrer}|-)" "(?:%{DATA:useragent}|-)" %{NUMBER:request_length:int} %{NUMBER:request_time:float} \[%{GREEDYDATA:proxy_upstream_name}\] \[%{GREEDYDATA:proxy_alternative_upstream_name}\] %{IPORHOST:upstream_addr}:%{NUMBER:port} %{NUMBER:response_length:int} %{NUMBER:response_time:float} %{NUMBER:upstream_status:int} %{GREEDYDATA:req_id}' ]
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
target => "timestamp"
}
}