# Nginx Access Logs Parsing Using Filebeat On Kubernetes Cluster **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [September 5, 2018, 5:08am UTC](https://discuss.elastic.co/t/nginx-access-logs-parsing-using-filebeat-on-kubernetes-cluster/147303 "2018-09-05T05:08:06Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Arjun\_Sharma](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@Arjun\_Sharma](https://discuss.elastic.co/u/Arjun_Sharma)\ **Post date:** [September 5, 2018, 5:08am UTC](https://discuss.elastic.co/t/nginx-access-logs-parsing-using-filebeat-on-kubernetes-cluster/147303/1 "2018-09-05T05:08:07Z")
- We are using elasticsearch for centralized logging in our application. - There are many components in our application which generate logs. - All components running in kubernetes cluster. we are using filebeat as deamonset container on every node in the cluster. - There is an nginx ingress controller which sit in front of all service running in our cluster. nginx access logs with all othe components logs are shipped by filebeat into elasticsearch as a string. - **I want to transform nginx access logs using filebeat**. I tried many examples but none of them works as expected for me. Please help us to configure this setup. # Sample for existing document in eleasticsearch: ## { - ## "\_source": { "@timestamp": "2018-08-22T08:55:18.697Z", "message": "203.88.135.122 - [203.88.135.122] - - [22/Aug/2018:08:55:18 +0000] "GET /blue/rest/organizations/jenkins/pipelines/Backend/branches/customization-icaseboard-dev/runs/7/nodes/20/steps/25/log/?start=25682 HTTP/1.1" 200 0 "[https://jenkins.orderhive.plus/blue/organizations/jenkins/Backend/detail/customization-icaseboard-dev/7/pipeline](https://jenkins.orderhive.plus/blue/organizations/jenkins/Backend/detail/customization-icaseboard-dev/7/pipeline)" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" 944 0.003 [default-jenkins-80] 100.96.7.27:8080 0 0.004 200 f5ad0866dcd0f057939be292dfe6a7ac", - } # Sample for expexted document : ## { - ## "\_source": { "@timestamp": "2018-06-01T15:25:47.000Z", "source": "/var/log/nginx/pawel-blog-access-1.log", "input\_type": "log", "os\_name": "Ubuntu", "request": "/profitable-slack-bot-rails", "agent": ""Mozilla/5.0 (X11; Ubuntu; Linux x86\_64; rv:61.0) Gecko/20100101 Firefox/61.0"", "message": "157.234.132.47 - - [01/Jun/2018:17:25:47 +0200] "GET /profitable-slack-bot-rails HTTP/1.1" 200 12104 "[https://news.ycombinator.com/](https://news.ycombinator.com/)" "Mozilla/5.0 (X11; Ubuntu; Linux x86\_64; rv:61.0) Gecko/20100101 Firefox/61.0"", "device": "Other", "clientip": "157.234.132.47", "response": 200, "type": "log", "httpversion": "1.1", "host": "pawel-blog", "referrer": ""[https://news.ycombinator.com/](https://news.ycombinator.com/)"", "build": "", "tags": ["beats\_input\_codec\_plain\_applied", "nginx-geoip"], "os": "Ubuntu", "@version": "1", "offset": 571753, "geoip": { "latitude": -26.2309, "longitude": 28.0583, "country\_code3": "ZA", "timezone": "Africa/Johannesburg", "region\_name": "Gauteng", "ip": "157.234.132.47", "postal\_code": "2000", "continent\_code": "AF", "city\_name": "Johannesburg", "country\_name": "South Africa", "region\_code": "GT", "country\_code2": "ZA", "location": { "lon": 28.0583, "lat": -26.2309 } }, - } # filebeat.yaml * * * ## apiVersion: v1 kind: ConfigMap metadata: name: filebeat-config namespace: kube-system labels: k8s-app: filebeat [kubernetes.io/cluster-service:](http://kubernetes.io/cluster-service:) "true" data: filebeat.yml: |- filebeat.config: prospectors: # Mounted `filebeat-prospectors` configmap: path: ${path.config}/prospectors.d/_.yml # Reload prospectors configs as they change: reload.enabled: false modules: path: ${path.config}/modules.d/_.yml # Reload module configs as they change: reload.enabled: false filebeat.modules: - module: nginx filebeat.autodiscover: providers: - type: docker condition: contains: docker.container.image: "nginx-ingress-controller" config: - module: nginx access: prospector: type: docker containers.stream: stdout containers.ids: - "${data.docker.container.id}" processors: - add\_kubernetes\_metadata: in\_cluster: true error: prospector: type: docker containers.stream: stderr containers.ids: - "${data.docker.container.id}" processors: - add\_kubernetes\_metadata: in\_cluster: true cloud.id: ${ELASTIC\_CLOUD\_ID} cloud.auth: ${ELASTIC\_CLOUD\_AUTH} output.elasticsearch: hosts: ['${ELASTICSEARCH\_HOST:elasticsearch}:${ELASTICSEARCH\_PORT:9200}'] username: ${ELASTICSEARCH\_USERNAME} password: ${ELASTICSEARCH\_PASSWORD} apiVersion: v1 kind: ConfigMap metadata: name: filebeat-prospectors namespace: kube-system labels: k8s-app: filebeat [kubernetes.io/cluster-service:](http://kubernetes.io/cluster-service:) "true" data: kubernetes.yml: |- - type: docker json.message\_key: log json.keys\_under\_root: true containers.ids: - "_" processors: - add\_kubernetes\_metadata: in\_cluster: true - drop\_event: when: or: - regexp: logger\_name: 'org.apache.curator._' - equals: kubernetes.container.name: filebeat - add\_cloud\_metadata: ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [September 5, 2018, 6:53am UTC](https://discuss.elastic.co/t/nginx-access-logs-parsing-using-filebeat-on-kubernetes-cluster/147303/2 "2018-09-05T06:53:57Z")
It is a known bug you are facing. This is the issue where you can track its progress: [https://github.com/elastic/beats/issues/7914](https://github.com/elastic/beats/issues/7914) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 3, 2018, 6:54am UTC](https://discuss.elastic.co/t/nginx-access-logs-parsing-using-filebeat-on-kubernetes-cluster/147303/3 "2018-10-03T06:54:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.