# Filebeat monitor nginx 1 record, send ２ records data to elasticsearch, one json format good, the other bad **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [August 28, 2018, 2:58pm UTC](https://discuss.elastic.co/t/filebeat-monitor-nginx-1-record-send-records-data-to-elasticsearch-one-json-format-good-the-other-bad/146383 "2018-08-28T14:58:04Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![tomoncle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomoncle/32/34890_2.png) [@tomoncle](https://discuss.elastic.co/u/tomoncle)\ **Post date:** [August 28, 2018, 2:58pm UTC](https://discuss.elastic.co/t/filebeat-monitor-nginx-1-record-send-records-data-to-elasticsearch-one-json-format-good-the-other-bad/146383/1 "2018-08-28T14:58:04Z")
- filebeat Version: `filebeat version 6.4.0 (amd64), libbeat 6.4.0` - Operating System: `centos 7` - filebeat.yaml ```auto filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/*.log filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.settings: index.number_of_shards: 3 setup.kibana: host: "192.168.84.14:5601" output.elasticsearch: hosts: ["192.168.84.25:9200"] ``` - filebeat-pipline: [filebeat deafult.json](https://github.com/elastic/beats/blob/master/filebeat/module/nginx/access/ingest/default.json). - elasticsearch: `Version: 6.4.0, Build: default/tar/595516e/2018-08-17T23:18:47.308994Z, JVM: 1.8.0_181` - elasticsearch-plugins: `ingest-geoip` & `ingest-user-agent` - elasticsearch.yaml : ```auto network.host: 192.168.84.25 ``` - nginx.conf: ```auto user nginx; worker_processes auto; error_log /var/log/nginx/error.log; pid /run/nginx.pid; include /usr/share/nginx/modules/*.conf; events { worker_connections 1024; } http { sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; include /etc/nginx/mime.types; default_type application/octet-stream; log_format timed_combined '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '$request_time $upstream_response_time $pipe'; access_log /var/log/nginx/access.log timed_combined; include /etc/nginx/conf.d/*.conf; server { listen 80 default_server; listen [::]:80 default_server; server_name _; root /usr/share/nginx/html; include /etc/nginx/default.d/*.conf; location / { proxy_pass http://192.168.1.23:8000; } error_page 404 /404.html; location = /40x.html { } error_page 500 502 503 504 /50x.html; location = /50x.html { } } } ``` ## test - step1: send request : `$ curl http://192.168.84.38/app/api/classes/` - step2: cat nginx/access.log ```auto 172.16.4.13 - - [28/Aug/2018:07:50:32 +0000] "GET /app/api/classes/ HTTP/1.1" 200 142 "-" "curl/7.35.0" 0.360 0.360 . ``` - step3: cat elasticsearch filebeat data ```auto [centos@liyuanjun-nginx-filebeat ~]$ curl -ssl http://192.168.84.25:9200/filebeat-*/_search?pretty | python -m json.tool { "_shards": { "failed": 0, "skipped": 0, "successful": 3, "total": 3 }, "hits": { "hits": [ { "_id": "WBODf2UBeM8H3-brZX4B", "_index": "filebeat-6.4.0-2018.08.28", "_score": 1.0, "_source": { "@timestamp": "2018-08-28T07:50:41.180Z", "beat": { "hostname": "liyuanjun-nginx-filebeat.novalocal", "name": "liyuanjun-nginx-filebeat.novalocal", "version": "6.4.0" }, "error": { "message": "Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}", "type": "json" }, "host": { "name": "liyuanjun-nginx-filebeat.novalocal" }, "input": { "type": "log" }, "message": "172.16.4.13 - - [28/Aug/2018:07:50:32 +0000] \"GET /app/api/classes/ HTTP/1.1\" 200 142 \"-\" \"curl/7.35.0\" 0.360 0.360 .", "offset": 240, "prospector": { "type": "log" }, "source": "/var/log/nginx/access.log" }, "_type": "doc" }, { "_id": "WRODf2UBeM8H3-brZX4B", "_index": "filebeat-6.4.0-2018.08.28", "_score": 1.0, "_source": { "@timestamp": "2018-08-28T07:50:32.000Z", "beat": { "hostname": "liyuanjun-nginx-filebeat.novalocal", "name": "liyuanjun-nginx-filebeat.novalocal", "version": "6.4.0" }, "fileset": { "module": "nginx", "name": "access" }, "host": { "name": "liyuanjun-nginx-filebeat.novalocal" }, "input": { "type": "log" }, "nginx": { "access": { "body_sent": { "bytes": "142" }, "http_version": "1.1", "method": "GET", "referrer": "-", "remote_ip": "172.16.4.13", "remote_ip_list": [ "172.16.4.13" ], "response_code": "200", "url": "/app/api/classes/", "user_agent": { "device": "Other", "major": "7", "minor": "35", "name": "curl", "os": "Other", "os_name": "Other", "patch": "0" }, "user_name": "-" } }, "offset": 240, "prospector": { "type": "log" }, "read_timestamp": "2018-08-28T07:50:41.193Z", "source": "/var/log/nginx/access.log" }, "_type": "doc" } ], "max_score": 1.0, "total": 2 }, "timed_out": false, "took": 0 } ``` I don't know why filebeat sends two data to ES, but I just want one JSON data (`WRODf2UBeM8H3-brZX4B`). Is this a configuration problem or a legacy problem? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [August 28, 2018, 7:38pm UTC](https://discuss.elastic.co/t/filebeat-monitor-nginx-1-record-send-records-data-to-elasticsearch-one-json-format-good-the-other-bad/146383/2 "2018-08-28T19:38:30Z")
You have essentially configured 2 inputs reading your nginx logs. As you are using the filebeat module, remove the `filebeat.inputs` section (or set enabled to false): ```auto filebeat.inputs: - type: log enabled: false paths: - /var/log/nginx/*.log ``` The nginx module will configure it's own input. ---
**Author:** ![tomoncle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomoncle/32/34890_2.png) [@tomoncle](https://discuss.elastic.co/u/tomoncle)\ **Post date:** [August 29, 2018, 2:38am UTC](https://discuss.elastic.co/t/filebeat-monitor-nginx-1-record-send-records-data-to-elasticsearch-one-json-format-good-the-other-bad/146383/3 "2018-08-29T02:38:53Z")
Well, thank you for reminding me．👍 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 26, 2018, 2:38am UTC](https://discuss.elastic.co/t/filebeat-monitor-nginx-1-record-send-records-data-to-elasticsearch-one-json-format-good-the-other-bad/146383/4 "2018-09-26T02:38:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.