- filebeat Version:
filebeat version 6.4.0 (amd64), libbeat 6.4.0 - Operating System:
centos 7 - filebeat.yaml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/*.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
host: "192.168.84.14:5601"
output.elasticsearch:
hosts: ["192.168.84.25:9200"]
- filebeat-pipline: filebeat deafult.json.
- elasticsearch:
Version: 6.4.0, Build: default/tar/595516e/2018-08-17T23:18:47.308994Z, JVM: 1.8.0_181 - elasticsearch-plugins:
ingest-geoip&ingest-user-agent - elasticsearch.yaml :
network.host: 192.168.84.25
- nginx.conf:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format timed_combined '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'$request_time $upstream_response_time $pipe';
access_log /var/log/nginx/access.log timed_combined;
include /etc/nginx/conf.d/*.conf;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
root /usr/share/nginx/html;
include /etc/nginx/default.d/*.conf;
location / {
proxy_pass http://192.168.1.23:8000;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
}
test
- step1: send request :
$ curl http://192.168.84.38/app/api/classes/ - step2: cat nginx/access.log
172.16.4.13 - - [28/Aug/2018:07:50:32 +0000] "GET /app/api/classes/ HTTP/1.1" 200 142 "-" "curl/7.35.0" 0.360 0.360 .
- step3: cat elasticsearch filebeat data
[centos@liyuanjun-nginx-filebeat ~]$ curl -ssl http://192.168.84.25:9200/filebeat-*/_search?pretty | python -m json.tool
{
"_shards": {
"failed": 0,
"skipped": 0,
"successful": 3,
"total": 3
},
"hits": {
"hits": [
{
"_id": "WBODf2UBeM8H3-brZX4B",
"_index": "filebeat-6.4.0-2018.08.28",
"_score": 1.0,
"_source": {
"@timestamp": "2018-08-28T07:50:41.180Z",
"beat": {
"hostname": "liyuanjun-nginx-filebeat.novalocal",
"name": "liyuanjun-nginx-filebeat.novalocal",
"version": "6.4.0"
},
"error": {
"message": "Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}",
"type": "json"
},
"host": {
"name": "liyuanjun-nginx-filebeat.novalocal"
},
"input": {
"type": "log"
},
"message": "172.16.4.13 - - [28/Aug/2018:07:50:32 +0000] \"GET /app/api/classes/ HTTP/1.1\" 200 142 \"-\" \"curl/7.35.0\" 0.360 0.360 .",
"offset": 240,
"prospector": {
"type": "log"
},
"source": "/var/log/nginx/access.log"
},
"_type": "doc"
},
{
"_id": "WRODf2UBeM8H3-brZX4B",
"_index": "filebeat-6.4.0-2018.08.28",
"_score": 1.0,
"_source": {
"@timestamp": "2018-08-28T07:50:32.000Z",
"beat": {
"hostname": "liyuanjun-nginx-filebeat.novalocal",
"name": "liyuanjun-nginx-filebeat.novalocal",
"version": "6.4.0"
},
"fileset": {
"module": "nginx",
"name": "access"
},
"host": {
"name": "liyuanjun-nginx-filebeat.novalocal"
},
"input": {
"type": "log"
},
"nginx": {
"access": {
"body_sent": {
"bytes": "142"
},
"http_version": "1.1",
"method": "GET",
"referrer": "-",
"remote_ip": "172.16.4.13",
"remote_ip_list": [
"172.16.4.13"
],
"response_code": "200",
"url": "/app/api/classes/",
"user_agent": {
"device": "Other",
"major": "7",
"minor": "35",
"name": "curl",
"os": "Other",
"os_name": "Other",
"patch": "0"
},
"user_name": "-"
}
},
"offset": 240,
"prospector": {
"type": "log"
},
"read_timestamp": "2018-08-28T07:50:41.193Z",
"source": "/var/log/nginx/access.log"
},
"_type": "doc"
}
],
"max_score": 1.0,
"total": 2
},
"timed_out": false,
"took": 0
}
I don't know why filebeat sends two data to ES, but I just want one JSON data (WRODf2UBeM8H3-brZX4B). Is this a configuration problem or a legacy problem?
