Hello everyone,
i have my filebeat running and queries the data but on Kibana, i am unable to get kibana to move reading from logstash to filebeat.
any suggestions?
Do you have filebeat data in Elasticsearch?
curl http://localhost:9200/filebeat-*/_count?pretty
Did you load the filebeat index pattern provided with Filebeat? Before you will see the filebeat-* index pattern you should run the ./scripts/import_dashboards tool then refresh the Kibana page. This will write the index pattern into the .kibana index used by Kibana.
For Linux when installed by rpm or deb the command is:
/usr/share/filebeat/scripts/import_dashboards -es http://elasticsearch:9200
If you are using the tar or zip package the command is located in the scripts directory of the package.
It seems like you have 17k events in Elasticsearch. Could it be that this is old log data and you only look at the last x minutes in Kibana? Or that the timestamps are off?
Please paste code / logs as text and not screenshots.
Hello Rufflin,
i am working with old samples data and i am trying to get familiar with filters.
filebeat -c /etc/filebeat/filebeat.yml -e -d "*" -path.config /etc/filebeat
2017/02/01 14:32:37.670237 beat.go:267: INFO Home path: [/home/ctf] Config path: [/etc/filebeat] Data path: [/home/ctf/data] Logs path: [/home/ctf/logs]
2017/02/01 14:32:37.670533 beat.go:177: INFO Setup Beat: filebeat; Version: 5.1.2
2017/02/01 14:32:37.670778 logp.go:219: INFO Metrics logging every 30s
2017/02/01 14:32:37.670749 processor.go:43: DBG Processors:
2017/02/01 14:32:37.670886 beat.go:183: DBG Initializing output plugins
2017/02/01 14:32:37.671743 output.go:167: INFO Loading template enabled. Reading template file: /etc/filebeat/filebeat.template.json
2017/02/01 14:32:37.673415 output.go:178: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /etc/filebeat/filebeat.template-es2x.json
2017/02/01 14:32:37.674798 client.go:120: INFO Elasticsearch url: http://localhost:9200
2017/02/01 14:32:37.675092 outputs.go:106: INFO Activated elasticsearch as output plugin.
2017/02/01 14:32:37.675147 publish.go:234: DBG Create output worker
2017/02/01 14:32:37.675531 publish.go:276: DBG No output is defined to store the topology. The server fields might not be filled.
2017/02/01 14:32:37.676857 publish.go:291: INFO Publisher name: localhost.localdomain
2017/02/01 14:32:37.678315 async.go:63: INFO Flush Interval set to: 1s
2017/02/01 14:32:37.678438 async.go:64: INFO Max Bulk Size set to: 50
2017/02/01 14:32:37.678493 async.go:72: DBG create bulk processing worker (interval=1s, bulk size=50)
2017/02/01 14:32:37.679263 beat.go:207: INFO filebeat start running.
2017/02/01 14:32:37.681436 registrar.go:85: INFO Registry file set to: /home/ctf/data/registry
2017/02/01 14:32:37.681791 registrar.go:106: INFO Loading registrar data from /home/ctf/data/registry
2017/02/01 14:32:37.684321 registrar.go:123: INFO States Loaded from registrar: 6
2017/02/01 14:32:37.684706 crawler.go:34: INFO Loading Prospectors: 1
2017/02/01 14:32:37.685139 registrar.go:236: INFO Starting Registrar
2017/02/01 14:32:37.685315 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017/02/01 14:32:37.685048 sync.go:41: INFO Start sending events to output
2017/02/01 14:32:37.685175 prospector_log.go:41: DBG exclude_files: []
A few more log lines are needed. This is only the first second of log events. Not events are sent yet.
hello do you mind looking at this and let me know what can be the problem
eb 14 12:25:57 localhost.localdomain systemd[1]: Started filebeat.
Feb 14 12:25:57 localhost.localdomain systemd[1]: Starting filebeat...
[ctf@localhost ~]$ sudo filebeat -c /etc/filebeat/filebeat.yml -e -d "*"-path.config /etc/filebeat
[sudo] password for ctf:
2017/02/14 18:11:45.762963 beat.go:267: INFO Home path: [/home/ctf] Config path: [/home/ctf] Data path: [/home/ctf/data] Logs path: [/home/ctf/logs]
2017/02/14 18:11:45.763304 beat.go:177: INFO Setup Beat: filebeat; Version: 5.1.2
2017/02/14 18:11:45.763447 logp.go:219: INFO Metrics logging every 30s
2017/02/14 18:11:45.765014 output.go:167: INFO Loading template enabled. Reading template file: /etc/filebeat/filebeat.template.json
2017/02/14 18:11:45.767400 output.go:178: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /home/ctf/filebeat.template-es2x.json
2017/02/14 18:11:45.767696 outputs.go:100: ERR failed to initialize elasticsearch plugin as output: Error loading template /home/ctf/filebeat.template-es2x.json: open /home/ctf/filebeat.template-es2x.json: no such file or directory
2017/02/14 18:11:45.767951 beat.go:288: CRIT Exiting: error initializing publisher: Error loading template /home/ctf/filebeat.template-es2x.json: open /home/ctf/filebeat.template-es2x.json: no such file or directory
Exiting: error initializing publisher: Error loading template /home/ctf/filebeat.template-es2x.json: open /home/ctf/filebeat.template-es2x.json: no such file or directory
[ctf@localhost ~]$ ./filebeat -c .test/filebeat.yml -configtest
bash: ./filebeat: No such file or directory
[ctf@localhost ~]$ ./filebeat -c test/filebeat.yml -configtest
bash: ./filebeat: No such file or directory
[ctf@localhost ~]$ sudo filebeat -c /etc/filebeat/filebeat.yml -e -d "*"-path.config /etc/filebeat
2017/02/14 18:14:54.957629 beat.go:267: INFO Home path: [/home/ctf] Config path: [/home/ctf] Data path: [/home/ctf/data] Logs path: [/home/ctf/logs]
2017/02/14 18:14:54.957970 beat.go:177: INFO Setup Beat: filebeat; Version: 5.1.2
2017/02/14 18:14:54.958324 logp.go:219: INFO Metrics logging every 30s
2017/02/14 18:14:54.959051 output.go:167: INFO Loading template enabled. Reading template file: /etc/filebeat/filebeat.template.json
2017/02/14 18:14:54.960683 output.go:178: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /home/ctf/filebeat.template-es2x.json
2017/02/14 18:14:54.960901 outputs.go:100: ERR failed to initialize elasticsearch plugin as output: Error loading template /home/ctf/filebeat.template-es2x.json: open /home/ctf/filebeat.template-es2x.json: no such file or directory
2017/02/14 18:14:54.961165 beat.go:288: CRIT Exiting: error initializing publisher: Error loading template /home/ctf/filebeat.template-es2x.json: open /home/ctf/filebeat.template-es2x.json: no such file or directory
Exiting: error initializing publisher: Error loading template /home/ctf/filebeat.template-es2x.json: open /home/ctf/filebeat.template-es2x.json: no such file or directory
[ctf@localhost ~]$
compared to the last config that i had at the beginning of the month
Feb 14 12:25:57 localhost.localdomain systemd[1]: Started filebeat.
Feb 14 12:25:57 localhost.localdomain systemd[1]: Starting filebeat...
[ctf@localhost ~]$ sudo filebeat -c /etc/filebeat/filebeat.yml -e -d ""-path.config /etc/filebeat
[sudo] password for ctf:
2017/02/14 18:11:45.762963 beat.go:267: INFO Home path: [/home/ctf] Config path: [/home/ctf] Data path: [/home/ctf/data] Logs path: [/home/ctf/logs]
2017/02/14 18:11:45.763304 beat.go:177: INFO Setup Beat: filebeat; Version: 5.1.2
2017/02/14 18:11:45.763447 logp.go:219: INFO Metrics logging every 30s
2017/02/14 18:11:45.765014 output.go:167: INFO Loading template enabled. Reading template file: /etc/filebeat/filebeat.template.json
2017/02/14 18:11:45.767400 output.go:178: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /home/ctf/filebeat.template-es2x.json
2017/02/14 18:11:45.767696 outputs.go:100: ERR failed to initialize elasticsearch plugin as output: Error loading template /home/ctf/filebeat.template-es2x.json: open /home/ctf/filebeat.template-es2x.json: no such file or directory
2017/02/14 18:11:45.767951 beat.go:288: CRIT Exiting: error initializing publisher: Error loading template /home/ctf/filebeat.template-es2x.json: open /home/ctf/filebeat.template-es2x.json: no such file or directory
Exiting: error initializing publisher: Error loading template /home/ctf/filebeat.template-es2x.json: open /home/ctf/filebeat.template-es2x.json: no such file or directory
[ctf@localhost ~]$ ./filebeat -c .test/filebeat.yml -configtest
bash: ./filebeat: No such file or directory
[ctf@localhost ~]$ ./filebeat -c test/filebeat.yml -configtest
bash: ./filebeat: No such file or directory
[ctf@localhost ~]$ sudo filebeat -c /etc/filebeat/filebeat.yml -e -d ""-path.config /etc/filebeat
2017/02/14 18:14:54.957629 beat.go:267: INFO Home path: [/home/ctf] Config path: [/home/ctf] Data path: [/home/ctf/data] Logs path: [/home/ctf/logs]
2017/02/14 18:14:54.957970 beat.go:177: INFO Setup Beat: filebeat; Version: 5.1.2
2017/02/14 18:14:54.958324 logp.go:219: INFO Metrics logging every 30s
2017/02/14 18:14:54.959051 output.go:167: INFO Loading template enabled. Reading template file: /etc/filebeat/filebeat.template.json
2017/02/14 18:14:54.960683 output.go:178: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /home/ctf/filebeat.template-es2x.json
2017/02/14 18:14:54.960901 outputs.go:100: ERR failed to initialize elasticsearch plugin as output: Error loading template /home/ctf/filebeat.template-es2x.json: open /home/ctf/filebeat.template-es2x.json: no such file or directory
2017/02/14 18:14:54.961165 beat.go:288: CRIT Exiting: error initializing publisher: Error loading template /home/ctf/filebeat.template-es2x.json: open /home/ctf/filebeat.template-es2x.json: no such file or directory
Exiting: error initializing publisher: Error loading template /home/ctf/filebeat.template-es2x.json: open /home/ctf/filebeat.template-es2x.json: no such file or directory
[ctf@localhost ~]$
I have a hard time to figure out what is logs above and what is text. Can you add 3 ticks ` around the log entries. Most of the cases above I could see, filebeat stopped with an error?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.