So I have 1 more idea
Clean up the indices...
Install a local tar.gz dist for a test
$ cd /tmp
$ curl -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.12.0-linux-x86_64.tar.gz
$ tar -xvf filebeat-7.12.0-linux-x86_64.tar.gz
$ filebeat-7.12.0-linux-x86_64
cp in my minimal filebeat.yml
run these commands make sure you use the ./
$ ./filebeat setup -e
$ ./filebeat -e
see what happens
Thank you for your reply.
I installed Filebeat using the package for linux (.tar.gz) and tried again.
The result is the same as #37.
In addition to the index with the specified name, a separate index of the form filebeat-%{[agent.version]}-%{+yyyy.MM.dd} is created, which seems to be the default name.
The steps performed are as follows
# rpm -e filebeat
# systemctl daemon-reload
# curl -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.12.0-linux-x86_64.tar.gz
# tar -xvf filebeat-7.12.0-linux-x86_64.tar.gz
# cd filebeat-7.12.0-linux-x86_64
# vi filebeat.yml
filebeat.modules:
- module: nginx
access:
enabled: true
input.index: "filebeat-%{[agent.version]}-nginx-access-%{+yyyy.MM.dd}"
#input.index: "else02-nginx-access-%{+yyyy.MM.dd}"
input.tags: ["customer-a"]
var.paths: ["/var/log/nginx/access.log"]
setup.template.settings:
index.number_of_shards: 1
index.number_of_replica: 0
#setup.kibana:
output.elasticsearch:
hosts: ["XXX.XXX.XXX.XXX:9200"]
# ./filebeat setup -e
# ./filebeat -e
# curl -X GET "XXX.XXX.XXX.XXX:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana-event-log-7.12.0 VBJPx8d9SPuItMyyIFg_cA 1 1 1 0 6.9kb 6.9kb
green open .kibana_task_manager_7.12.0_001 d6WvTHvXR963epPgZJLO6g 1 0 9 1708 257.1kb 257.1kb
green open .apm-custom-link z8w4mV7ySPWEj4ODSirapw 1 0 0 0 208b 208b
green open .apm-agent-configuration aeIElP-4TVqIe5oDKnMA2Q 1 0 0 0 208b 208b
green open .async-search EKWvKjbjQqO_z9z5cM3WUg 1 0 8 21 41.4kb 41.4kb
green open .kibana_7.12.0_001 6T4UGiR6RBuF_GdTeO4shg 1 0 44 3 2.1mb 2.1mb
# curl -X GET "XXX.XXX.XXX.XXX:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open filebeat-7.12.0-nginx-access-2022.08.04 xPkY5wdxQfyvWDfhCykiSw 1 1 1 0 23.6kb 23.6kb
yellow open .kibana-event-log-7.12.0 VBJPx8d9SPuItMyyIFg_cA 1 1 1 0 6.9kb 6.9kb
green open .kibana_task_manager_7.12.0_001 d6WvTHvXR963epPgZJLO6g 1 0 9 1717 223.8kb 223.8kb
green open .apm-custom-link z8w4mV7ySPWEj4ODSirapw 1 0 0 0 208b 208b
green open .apm-agent-configuration aeIElP-4TVqIe5oDKnMA2Q 1 0 0 0 208b 208b
green open .async-search EKWvKjbjQqO_z9z5cM3WUg 1 0 8 21 41.4kb 41.4kb
yellow open filebeat-7.12.0-2022.08.04-000001 HXAFD2stRgukHnVBezU98g 1 1 1 0 12.2kb 12.2kb
green open .kibana_7.12.0_001 6T4UGiR6RBuF_GdTeO4shg 1 0 44 3 2.1mb 2.1mb
Of the indexes created, filebeat-7.12.0-nginx-access-2022.08.04 is the one described in filebeat.yml and is the one intended to be created.
However, filebeat-7.12.0-2022.08.04-000001 is created at the same time.
When I check these on Kibana, they both collect the same access logs.
Interesting and it looks like the default named index has the message field and the other does not...
Do the documents look the same or is one parse with all the fields and the other is not.
Did both have the tag with customer-a?
I need to think about this it is very odd / strange... it is behaving like there are 2 harvesters / collectors running when we only have 1 defined.
can you add two more settings and try again... clean up everything?
setup.ilm.enabled: false
setup.template.enable: false
Take out the other template settings we will work them later.
put them at the root level right before
setup.ilm.enabled: false
setup.template.enable: false
output.elasticsearch:
hosts: ["XXX.XXX.XXX.XXX:9200"]
I have a few more ideas but let me know about this first...
Thanks for your comment.
The @timestamp is slightly off, but the ip, request, and user agent fields look the same.
As you can see with and without the message field, the way the fields are output is very different.
The filebeat-7.12.0-nginx-access-2022.08.04 has richer field entries, such as source.geo.location and traefik.access.user_agent.os_name, which parses the user agent, indicating that it uses Filebeat's nginx module.
However, the filebeat-7.12.0-2022.08.04 has simpler field entries.
There appears to be no parsed information, only an additional message field.
Thanks for the additional configuration info. I will try it now!
Ok that is good that means the module is working ... it actually pretty much means everything is working we just need to find out WHY the extra index is getting created / written too!
Actually it is not off ... the timestamp is the time of the actual event in the logs.. that is exactly correct, I can see that from the message...
We are close we just need to stop that other index... I am really really puzzled where it is coming from.. I can not reproduce
OHHHHH I THINK I may have found it!!!! Give me a moments...
Actually can you share the startup logs from filebeat!
First 30 or so lines... .
look for a line that says
2022-08-03T21:32:13.507-0700 INFO [crawler] beater/crawler.go:71 Loading Inputs: 3
I DID FIND it!!!! it is the ^^^^ I exactly reproduced it
TRY this and tell me if it works then I will explain why.. in short the other filesets are enabled by default and the ingress controller is look at the same place.
@its-ogawa DARN apologies this was my fault... try this then we can got back to the rest!!!
I introduced this when I simplified.
filebeat.modules:
- module: nginx
access:
enabled: true
input.index: "filebeat-%{[agent.version]}-nginx-access-%{+yyyy.MM.dd}"
input.tags: ["customer-a"]
var.paths: ["/Users/sbrown/workspace/sample-data/nginx/nginx-test.log"]
# Disable other nginx filesets.
error:
enabled: false
ingress_controller:
enabled: false
setup.template.enabled: false
setup.ilm.enabled: false
output.elasticsearch:
hosts: ["localhost:9200"]
Excellent!!! It works as expected!
After setting up the above.
# curl -X GET "XXX.XXX.XXX.XXX:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana-event-log-7.12.0 VBJPx8d9SPuItMyyIFg_cA 1 1 1 0 6.9kb 6.9kb
green open .kibana_task_manager_7.12.0_001 d6WvTHvXR963epPgZJLO6g 1 0 9 4015 486.9kb 486.9kb
green open .apm-custom-link z8w4mV7ySPWEj4ODSirapw 1 0 0 0 208b 208b
green open .apm-agent-configuration aeIElP-4TVqIe5oDKnMA2Q 1 0 0 0 208b 208b
green open .async-search EKWvKjbjQqO_z9z5cM3WUg 1 0 8 0 6kb 6kb
green open .kibana_7.12.0_001 6T4UGiR6RBuF_GdTeO4shg 1 0 1962 108 4.7mb 4.7mb
# curl -X GET "XXX.XXX.XXX.XXX:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open filebeat-7.12.0-nginx-access-2022.08.04 hjnYOTS-TQqqXpqYeEJPyA 1 1 0 0 208b 208b
yellow open .kibana-event-log-7.12.0 VBJPx8d9SPuItMyyIFg_cA 1 1 1 0 6.9kb 6.9kb
green open .kibana_task_manager_7.12.0_001 d6WvTHvXR963epPgZJLO6g 1 0 9 4018 486.9kb 486.9kb
green open .apm-custom-link z8w4mV7ySPWEj4ODSirapw 1 0 0 0 208b 208b
green open .apm-agent-configuration aeIElP-4TVqIe5oDKnMA2Q 1 0 0 0 208b 208b
green open .async-search EKWvKjbjQqO_z9z5cM3WUg 1 0 8 0 6kb 6kb
green open .kibana_7.12.0_001 6T4UGiR6RBuF_GdTeO4shg 1 0 1962 108 4.7mb 4.7mb
2022-08-04T15:06:35.200+0900 INFO instance/beat.go:660 Home path: [/root/tmp/filebeat-7.12.0-linux-x86_64] Config path: [/root/tmp/filebeat-7.12.0-linux-x86_64] Data path: [/root/tmp/filebeat-7.12.0-linux-x86_64/data] Logs path: [/root/tmp/filebeat-7.12.0-linux-x86_64/logs]
2022-08-04T15:06:35.200+0900 INFO instance/beat.go:668 Beat ID: 71baa16f-15bc-4024-8b6d-67ac9501790e
2022-08-04T15:06:35.201+0900 INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2022-08-04T15:06:35.201+0900 INFO [beat] instance/beat.go:996 Beat info {"system_info": {"beat": {"path": {"config": "/root/tmp/filebeat-7.12.0-linux-x86_64", "data": "/root/tmp/filebeat-7.12.0-linux-x86_64/data", "home": "/root/tmp/filebeat-7.12.0-linux-x86_64", "logs": "/root/tmp/filebeat-7.12.0-linux-x86_64/logs"}, "type": "filebeat", "uuid": "71baa16f-15bc-4024-8b6d-67ac9501790e"}}}
2022-08-04T15:06:35.201+0900 INFO [beat] instance/beat.go:1005 Build info {"system_info": {"build": {"commit": "08e20483a651ea5ad60115f68ff0e53e6360573a", "libbeat": "7.12.0", "time": "2021-03-18T06:16:51.000Z", "version": "7.12.0"}}}
2022-08-04T15:06:35.201+0900 INFO [beat] instance/beat.go:1008 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.15.8"}}}
2022-08-04T15:06:35.202+0900 INFO [beat] instance/beat.go:1012 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-02-15T12:23:53+09:00","containerized":false,"name":"ITS-ELSE-02","ip":["127.0.0.1/8","::1/128","202.221.140.169/26","fe80::250:56ff:fe89:f117/64"],"kernel_version":"3.10.0-1160.15.2.el7.x86_64","mac":["00:50:56:89:f1:17"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"JST","timezone_offset_sec":32400,"id":"f90df61f81c0432e8875d3d5489faf19"}}}
2022-08-04T15:06:35.202+0900 INFO [beat] instance/beat.go:1041 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/root/tmp/filebeat-7.12.0-linux-x86_64", "exe": "/root/tmp/filebeat-7.12.0-linux-x86_64/filebeat", "name": "filebeat", "pid": 23503, "ppid": 23572, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-08-04T15:06:34.210+0900"}}}
2022-08-04T15:06:35.202+0900 INFO instance/beat.go:304 Setup Beat: filebeat; Version: 7.12.0
2022-08-04T15:06:35.203+0900 INFO eslegclient/connection.go:99 elasticsearch url: http://XXX.XXX.XXX.XXX:9200
2022-08-04T15:06:35.203+0900 INFO [publisher] pipeline/module.go:113 Beat name: ELSE-02
2022-08-04T15:06:35.204+0900 INFO beater/filebeat.go:117 Enabled modules/filesets: nginx (access), ()
2022-08-04T15:06:35.205+0900 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2022-08-04T15:06:35.205+0900 INFO instance/beat.go:468 filebeat start running.
2022-08-04T15:06:35.205+0900 INFO memlog/store.go:119 Loading data file of '/root/tmp/filebeat-7.12.0-linux-x86_64/data/registry/filebeat' succeeded. Active transaction id=0
2022-08-04T15:06:35.210+0900 INFO memlog/store.go:124 Finished loading transaction log file for '/root/tmp/filebeat-7.12.0-linux-x86_64/data/registry/filebeat'. Active transaction id=250
2022-08-04T15:06:35.210+0900 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 5
2022-08-04T15:06:35.210+0900 INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2022-08-04T15:06:35.211+0900 INFO log/input.go:157 Configured paths: [/var/log/nginx/access.log]
2022-08-04T15:06:35.211+0900 INFO [crawler] beater/crawler.go:141 Starting input (ID: 68902100857948553)
2022-08-04T15:06:35.211+0900 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
If it has Lens Visualization, you can first use Count in the Y-axis and then use the Top Values option under the Horizontal Axis ..
I really like nginx's module functionality because it parses location, user agent, and OS information.
I have a simple question: if I use Filebeat's module functionality, can I register documents with Elasticsearch via Logstash?
When I put both Logstash and Elasticsearch in the destination, I get the following error
error unpacking config data: more than one namespace configured accessing 'output' (source:'filebeat.yml')
In this case we used the module functionality to get nginx logs, but we have multiple logs on the same server that we want to collect.
It would be nice if there was a module function for those as well, but not for common logs like nginx.
So far we have been parsing them with Logstash filters and registering the documents in Elasticsearch.
1st Yay.
2nd now if you want you can go back to a filebeat.yml and nginx.yml if you want and set or you can keep in a simple version...
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 0
We should really open a new topic on this ...
You can not have more than 1 output in filebeat that is a limitation of filebeat right now.. either elasticsearch or logstash not both.
So you have 2 choices of architecture
Filebeat-> Elasticsearch
Probably what I would recommend.
Use the ngnix for the nginx logs
And us a normal file / log input and create and ingest pipeline in elasticsearch to parse your other logs.
No logstash needed.
If you shared a sample log line and the parsing you are doing in logstash we could probably create a ingest pipeline
Filebeat -> logstash -> Elasticsearch..
More complicated
Use the ngnix moduled for the nginx logs and then point them to logstash which will work as a pass through.
Collect the logs and parse them in logsstash and send them to elastic.
This will require addition logic in logstash in the input and output sections.
Which one would you like then you should open a new thread on that
Thanks for replying.
I will ask a new question about the REPLICA settings in another topic.
I am sure this is true, but I am very disappointed.
Because I have been using Logstash.
Thanks for the two suggestions.
I will consider them carefully.
I found something called geoip, useragent filter in Logstash.
These may help, but I am unclear about geoip.
I hope to clarify this also in another topic.
You can put those back in.
I meant we should open another topic on the logstash / filebeat stuff.
The filebeat to logstash will be fine not too hard.
We can show you...
I will be away on for a while.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.