this is my lab and i want to collect my router logs with syslog to filebeat installed in the same server of elasticsearch i follow this steps but i don't receive the logs my elastic server is a ubuntu server 22.04 , my elastic version is 8.3.2 and filebeat and kibana also
We can't help without details...
What does the filebeat.yml and cisco.yml look like?
What do the filebeat logs show?
journalctl -u filebeat.service
Have you tested connectivity between all components?
Filebeat.yml:
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
# filestream is an input for collecting log messages from files.
- type: filestream
# Unique ID among all inputs, an ID is required.
id: my-filestream-id
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: true
# Period on which files under path should be checked for changes
#reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 56>
# In case you specify and additional path, the scheme is required: http://loc>
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "https://192.168.37.2:5601"
ssl.verification_mode: none
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By defau>
# the Default Space will be used.
#space.id:
# ---------------------------- Elasticsearch Output -------------------->
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
username: "elastic"
password: "passok"
ssl:
enabled: true
ca_trusted_fingerprint: "58809f135a319b6e02d22882bc4f5df3c5f289f55ca>
# ================================= Processors =========================>
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
# ================================= Migration ==========================>
# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
setup.ilm.overwrite: true
Cisco.yml:
ios:
enabled: true
# Set which input to use between syslog (default) or file.
var.input: syslog
# The interface to listen to syslog traffic. Defaults to
# localhost. Set to 0.0.0.0 to bind to all available interfaces.
var.syslog_host: 0.0.0.0
# The port to listen on for syslog traffic. Defaults to 9002.
var.syslog_port: 9002
# Set which protocol to use between udp (default) or tcp.
#var.syslog_protocol: udp
# Set custom paths for the log files when using file input. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
What do the filebeat logs show?
journalctl -u filebeat.service
juil. 25 12:25:56 elkfiras systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
juil. 25 12:25:56 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:56.661Z","log.origin":{"file.name":"instance/beat.go","file.line":702},"m>
juil. 25 12:25:56 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:56.662Z","log.origin":{"file.name":"instance/beat.go","file.line":710},"m>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"warn","@timestamp":"2022-07-25T12:25:59.666Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.673Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccom>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.680Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go">
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.683Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go">
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.683Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go">
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.686Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go">
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.688Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go">
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.688Z","log.origin":{"file.name":"instance/beat.go","file.line":293},"m>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"warn","@timestamp":"2022-07-25T12:25:59.699Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/conf>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.700Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclien>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.702Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/mod>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.702Z","log.logger":"modules","log.origin":{"file.name":"fileset/module>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.705Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go">
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.706Z","log.origin":{"file.name":"instance/beat.go","file.line":470},"m>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.714Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"me>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.718Z","log.logger":"registrar","log.origin":{"file.name":"registrar/re>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.718Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.721Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.726Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.727Z","log.logger":"input.filestream","log.origin":{"file.name":"compa>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.727Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.734Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":164},">
juil. 25 12:26:02 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:02.671Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add>
juil. 25 12:26:02 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:02.762Z","log.logger":"publisher_pipeline_output","log.origin":{"file.nam>
juil. 25 12:26:02 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:02.846Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_conf>
juil. 25 12:26:02 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:02.846Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_conf>
juil. 25 12:26:02 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:02.882Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclien>
juil. 25 12:26:02 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:02.886Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclien>
juil. 25 12:26:02 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:02.886Z","log.logger":"index-management","log.origin":{"file.name":"idxmg>
juil. 25 12:26:03 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:03.041Z","log.logger":"index-management.ilm","log.origin":{"file.name":"i>
juil. 25 12:26:03 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:03.041Z","log.logger":"index-management","log.origin":{"file.name":"idxmg>
juil. 25 12:26:03 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:03.043Z","log.logger":"template","log.origin":{"file.name":"template/load>
juil. 25 12:26:05 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:05.795Z","log.logger":"template_loader","log.origin":{"file.name":"templa>
juil. 25 12:26:07 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:07.123Z","log.logger":"template_loader","log.origin":{"file.name":"templa>
juil. 25 12:26:07 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:07.161Z","log.logger":"template_loader","log.origin":{"file.name":"templa>
juil. 25 12:26:07 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:07.942Z","log.logger":"template_loader","log.origin":{"file.name":"templa>
juil. 25 12:26:07 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:07.943Z","log.logger":"index-management","log.origin":{"file.name":"idxmg>
juil. 25 12:26:07 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:07.946Z","log.logger":"publisher_pipeline_output","log.origin":{"file.nam>
juil. 25 12:26:09 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:09.749Z","log.logger":"modules","log.origin":{"file.name":"fileset/module>
juil. 25 12:26:09 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:09.755Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclien>
lines 1-43
Have you tested connectivity between all components? yes it's work they are in the same network
Unfortunately You truncated the logs. The important information is on the right hand side of the log lines. You truncated them so I can't tell what's going on.
Plus you need more of them probably 2 or 3 times as many. You can put them here or in a pastebin or something like that, but I can't help with only seeing the front section of the logs...
You should be able to see a line that says opening UDP for example.
Also, why do you have that filestsream input enabled? Are you collecting other logs?
And to be clear, you ran the setup command correct??
Can you see port 9002 open and listening?
Are you sure the cisco is sending to the correct host / IP / port?
@morad_della3 I feel like we have had this conversation before
@stephenb thanks for a response, like always you are helpful and the good and sorry about this conversation is not over yet You didn't answer my last question and I left out Fortinet and now I added a router and I am collecting these logs and I have this problem and I am looking for a solution
Right so if you want help... start answering these questions in detail?
And no one will be able to help without the complete logs...
So the logs will help...
There is something basic / fundamental not correct this all should take about 5 mins to set up.
Also, why do you have that filestsream input enabled? Are you collecting other logs?
- type: filestream
# Unique ID among all inputs, an ID is required.
id: my-filestream-id
# Change to true to enable this input configuration.
enabled: false
And to be clear, you ran the setup command correct??
elkfiras@elkfiras:~$ sudo filebeat setup
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded Ingest pipelines
Can you see port 9002 open and listening?
when i use this cmd i don't see the elk server listen in this port (9002) IS THIS CMD is the right cmd ?and how to let the server listen in this port
elkfiras@elkfiras:~$ sudo ss -ltn
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 511 192.168.37.2:5601 0.0.0.0:*
LISTEN 0 4096 *:9200 *:*
LISTEN 0 4096 [::ffff:192.168.37.2]:9300 *:*
LISTEN 0 128 [::]:22 [::]:*
also when i try this cmd to check if the elk listen in this port :
elkfiras@elkfiras:~$ sudo lsof -i:9002
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
filebeat 13153 root 31u IPv6 884048 0t0 UDP *:9002
Are you sure the cisco is sending to the correct host / IP / port?
logging host 192.168.37.2 transport udp port 9002
when I try to show the logging in the router
Trap logging: level informational, 181 message lines logged
Logging to 192.168.37.2 (udp port 9002, audit disabled,
link up),
180 message lines logged,
0 message lines rate-limited,
0 message lines dropped-by-MD,
xml disabled, sequence number disabled
filtering disabled
Logging Source-Interface: VRF Name:
the logs in the right side i can't copy them because when i select the logs to copy them the logs in the right side are hidden you know how to copy them all ?
Try directing the logs to a file?
journalctl -u filebeat -f > test.log
Did you try netcat to port 9002 on the filebeat machine
This will send message to the UDP port 9002 and see if it shows up in elasticsearch
$ nc -u 0.0.0.0 9002
Hello World
Bye World
Ack
^C
I do find it interesting that you are having the same basic issue with 2 different FWs / Modules... there is something basic going on.
sudo ufw status verbose
That should show you what's open on the server firewall rules "I use red hat mostly so not sure". It does look open from your command output.
If it's not showing 9002 then:
sudo ufw allow 9002/udp
What IOS version? I've had the same issue on some of the really old version and some current. To be honest IOS syslog is really weak SNMP polling provides far better detail but is a serious time sync on the snmp walks and pattern creation.
when i try this cmd , it don't give me an answer
lkfiras@elkfiras:~$ journalctl -u filebeat -f > test.log
do you mean this cmd "journalctl -u filebeat -f" or journalctl -u filebeat -f > test.log
when i try the first it give me this anser
elkfiras@elkfiras:~$ journalctl -u filebeat -f
août 22 13:14:02 elkfiras filebeat[159193]: {"log.level":"info","@timestamp":"2022-08-22T13:14:02.920Z","log.origin":{"file.name":"instance/beat.go","file.line":391},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
août 22 13:14:02 elkfiras filebeat[159193]: {"log.level":"error","@timestamp":"2022-08-22T13:14:02.920Z","log.origin":{"file.name":"instance/beat.go","file.line":1051},"message":"Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).","service.name":"filebeat","ecs.version":"1.6.0"}
août 22 13:14:02 elkfiras filebeat[159193]: Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
août 22 13:14:02 elkfiras systemd[1]: filebeat.service: Main process exited, code=exited, status=1/FAILURE
août 22 13:14:02 elkfiras systemd[1]: filebeat.service: Failed with result 'exit-code'.
août 22 13:14:03 elkfiras systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 1261.
août 22 13:14:03 elkfiras systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
août 22 13:14:03 elkfiras systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
août 22 13:14:03 elkfiras filebeat[159199]: {"log.level":"info","@timestamp":"2022-08-22T13:14:03.356Z","log.origin":{"file.name":"instance/beat.go","file.line":702},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
août 22 13:14:03 elkfiras filebeat[159199]: {"log.level":"info","@timestamp":"2022-08-22T13:14:03.356Z","log.origin":{"file.name":"instance/beat.go","file.line":710},"message":"Beat ID: 18f18c9b-6051-49ef-a4ce-e9cce3299b83","service.name":"filebeat","ecs.version":"1.6.0"}
août 22 13:14:06 elkfiras filebeat[159199]: {"log.level":"warn","@timestamp":"2022-08-22T13:14:06.359Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
août 22 13:14:06 elkfiras filebeat[159199]: {"log.level":"info","@timestamp":"2022-08-22T13:14:06.363Z","log.origin":{"file.name":"instance/beat.go","file.line":391},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
août 22 13:14:06 elkfiras filebeat[159199]: {"log.level":"error","@timestamp":"2022-08-22T13:14:06.367Z","log.origin":{"file.name":"instance/beat.go","file.line":1051},"message":"Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).","service.name":"filebeat","ecs.version":"1.6.0"}
août 22 13:14:06 elkfiras filebeat[159199]: Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
août 22 13:14:06 elkfiras systemd[1]: filebeat.service: Main process exited, code=exited, status=1/FAILURE
août 22 13:14:06 elkfiras systemd[1]: filebeat.service: Failed with result 'exit-code'.
août 22 13:14:06 elkfiras systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 1262.
août 22 13:14:06 elkfiras systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
août 22 13:14:06 elkfiras systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
août 22 13:14:06 elkfiras filebeat[159208]: {"log.level":"info","@timestamp":"2022-08-22T13:14:06.860Z","log.origin":{"file.name":"instance/beat.go","file.line":702},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
do you mean this cmd "journalctl -u filebeat -f" or journalctl -u filebeat -f > test.log
when i try this cmd , it don't give me an answer
elkfiras@elkfiras:~$ nc -u 0.0.0.0 9002
elkfiras@elkfiras:~$ sudo ufw status verbose
Status: inactive
it's inactive i disable it
Yes because the command directed the logs into the file test.log that is basic Unix stuff and then I wanted to see the rule
BUT what you posted helps...
Your logs clearly say (if you look at the logs there is good information there)
août 22 13:14:06 elkfiras filebeat[159199]: Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
So you have another fiebeat running already and you are trying to start another 1... You can only have 1 filebeat running so I am not sure why you have 1 already running and trying to start another but that is 1 problem.
Hello @stephenb sorry for the delay, Does it means Filebeat or module? because i run the Fortinet and cisco modules in the same Filebeat machine
Might be another Filebeat process has bean stuck. Stop Filebeat and check
ps aux | grep filebeat if is another process running kill it.
I don't know is important, when you start FB, on the first pic is port 9001.
Your configuration is using:
syslog_port: 9002
syslog_protocol: udp
Please check are the same settings on the device side.
2 Likes
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.