Can't collect the logs of cisco router


this is my lab and i want to collect my router logs with syslog to filebeat installed in the same server of elasticsearch i follow this steps but i don't receive the logs my elastic server is a ubuntu server 22.04 , my elastic version is 8.3.2 and filebeat and kibana also

Hi @morad_della3

We can't help without details...

What does the filebeat.yml and cisco.yml look like?

What do the filebeat logs show?

journalctl -u filebeat.service

Have you tested connectivity between all components?

Filebeat.yml:

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  #reload.period: 10s
# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 56>
  # In case you specify and additional path, the scheme is required: http://loc>
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "https://192.168.37.2:5601"
  ssl.verification_mode: none
  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By defau>
  # the Default Space will be used.
  #space.id:
# ---------------------------- Elasticsearch Output -------------------->
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "passok"
  ssl:
    enabled: true
    ca_trusted_fingerprint: "58809f135a319b6e02d22882bc4f5df3c5f289f55ca>
# ================================= Processors =========================>
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================= Migration ==========================>

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
setup.ilm.overwrite: true

Cisco.yml:

 ios:
    enabled: true

    # Set which input to use between syslog (default) or file.
    var.input: syslog

    # The interface to listen to syslog traffic. Defaults to
    # localhost. Set to 0.0.0.0 to bind to all available interfaces.
    var.syslog_host: 0.0.0.0

    # The port to listen on for syslog traffic. Defaults to 9002.
    var.syslog_port: 9002

    # Set which protocol to use between udp (default) or tcp.
    #var.syslog_protocol: udp

    # Set custom paths for the log files when using file input. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

What do the filebeat logs show?

journalctl -u filebeat.service

juil. 25 12:25:56 elkfiras systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
juil. 25 12:25:56 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:56.661Z","log.origin":{"file.name":"instance/beat.go","file.line":702},"m>
juil. 25 12:25:56 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:56.662Z","log.origin":{"file.name":"instance/beat.go","file.line":710},"m>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"warn","@timestamp":"2022-07-25T12:25:59.666Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.673Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccom>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.680Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go">
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.683Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go">
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.683Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go">
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.686Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go">
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.688Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go">
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.688Z","log.origin":{"file.name":"instance/beat.go","file.line":293},"m>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"warn","@timestamp":"2022-07-25T12:25:59.699Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/conf>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.700Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclien>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.702Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/mod>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.702Z","log.logger":"modules","log.origin":{"file.name":"fileset/module>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.705Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go">
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.706Z","log.origin":{"file.name":"instance/beat.go","file.line":470},"m>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.714Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"me>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.718Z","log.logger":"registrar","log.origin":{"file.name":"registrar/re>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.718Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.721Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.726Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.727Z","log.logger":"input.filestream","log.origin":{"file.name":"compa>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.727Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler>
juil. 25 12:25:59 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:25:59.734Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":164},">
juil. 25 12:26:02 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:02.671Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add>
juil. 25 12:26:02 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:02.762Z","log.logger":"publisher_pipeline_output","log.origin":{"file.nam>
juil. 25 12:26:02 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:02.846Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_conf>
juil. 25 12:26:02 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:02.846Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_conf>
juil. 25 12:26:02 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:02.882Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclien>
juil. 25 12:26:02 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:02.886Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclien>
juil. 25 12:26:02 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:02.886Z","log.logger":"index-management","log.origin":{"file.name":"idxmg>
juil. 25 12:26:03 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:03.041Z","log.logger":"index-management.ilm","log.origin":{"file.name":"i>
juil. 25 12:26:03 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:03.041Z","log.logger":"index-management","log.origin":{"file.name":"idxmg>
juil. 25 12:26:03 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:03.043Z","log.logger":"template","log.origin":{"file.name":"template/load>
juil. 25 12:26:05 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:05.795Z","log.logger":"template_loader","log.origin":{"file.name":"templa>
juil. 25 12:26:07 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:07.123Z","log.logger":"template_loader","log.origin":{"file.name":"templa>
juil. 25 12:26:07 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:07.161Z","log.logger":"template_loader","log.origin":{"file.name":"templa>
juil. 25 12:26:07 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:07.942Z","log.logger":"template_loader","log.origin":{"file.name":"templa>
juil. 25 12:26:07 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:07.943Z","log.logger":"index-management","log.origin":{"file.name":"idxmg>
juil. 25 12:26:07 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:07.946Z","log.logger":"publisher_pipeline_output","log.origin":{"file.nam>
juil. 25 12:26:09 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:09.749Z","log.logger":"modules","log.origin":{"file.name":"fileset/module>
juil. 25 12:26:09 elkfiras filebeat[9849]: {"log.level":"info","@timestamp":"2022-07-25T12:26:09.755Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclien>
lines 1-43

Have you tested connectivity between all components? yes it's work they are in the same network

Unfortunately You truncated the logs. The important information is on the right hand side of the log lines. You truncated them so I can't tell what's going on.

Plus you need more of them probably 2 or 3 times as many. You can put them here or in a pastebin or something like that, but I can't help with only seeing the front section of the logs...

You should be able to see a line that says opening UDP for example.

Also, why do you have that filestsream input enabled? Are you collecting other logs?

And to be clear, you ran the setup command correct??

Can you see port 9002 open and listening?

Are you sure the cisco is sending to the correct host / IP / port?

@morad_della3 I feel like we have had this conversation before

@stephenb thanks for a response, like always you are helpful and the good and sorry about this conversation is not over yet You didn't answer my last question and I left out Fortinet and now I added a router and I am collecting these logs and I have this problem and I am looking for a solution

Right so if you want help... start answering these questions in detail?

And no one will be able to help without the complete logs...

So the logs will help...

There is something basic / fundamental not correct this all should take about 5 mins to set up.

Also, why do you have that filestsream input enabled? Are you collecting other logs?

- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: false

And to be clear, you ran the setup command correct??

elkfiras@elkfiras:~$ sudo filebeat setup
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded Ingest pipelines

Can you see port 9002 open and listening?
when i use this cmd i don't see the elk server listen in this port (9002) IS THIS CMD is the right cmd ?and how to let the server listen in this port

elkfiras@elkfiras:~$ sudo ss -ltn
State   Recv-Q  Send-Q           Local Address:Port   Peer Address:Port Process
LISTEN  0       4096             127.0.0.53%lo:53          0.0.0.0:*
LISTEN  0       128                    0.0.0.0:22          0.0.0.0:*
LISTEN  0       511               192.168.37.2:5601        0.0.0.0:*
LISTEN  0       4096                         *:9200              *:*
LISTEN  0       4096     [::ffff:192.168.37.2]:9300              *:*
LISTEN  0       128                       [::]:22             [::]:*

also when i try this cmd to check if the elk listen in this port :

elkfiras@elkfiras:~$ sudo lsof -i:9002
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
filebeat 13153 root   31u  IPv6 884048      0t0  UDP *:9002

Are you sure the cisco is sending to the correct host / IP / port?
logging host 192.168.37.2 transport udp port 9002
when I try to show the logging in the router

Trap logging: level informational, 181 message lines logged
        Logging to 192.168.37.2  (udp port 9002, audit disabled,
              link up),
              180 message lines logged,
              0 message lines rate-limited,
              0 message lines dropped-by-MD,
              xml disabled, sequence number disabled
              filtering disabled
        Logging Source-Interface:       VRF Name:

the logs in the right side i can't copy them because when i select the logs to copy them the logs in the right side are hidden you know how to copy them all ?

Try directing the logs to a file?

journalctl -u filebeat -f > test.log

Did you try netcat to port 9002 on the filebeat machine

This will send message to the UDP port 9002 and see if it shows up in elasticsearch

$  nc -u 0.0.0.0 9002
Hello World
Bye World
Ack
^C

I do find it interesting that you are having the same basic issue with 2 different FWs / Modules... there is something basic going on.

sudo ufw status verbose

That should show you what's open on the server firewall rules "I use red hat mostly so not sure". It does look open from your command output.

If it's not showing 9002 then:
sudo ufw allow 9002/udp

What IOS version? I've had the same issue on some of the really old version and some current. To be honest IOS syslog is really weak SNMP polling provides far better detail but is a serious time sync on the snmp walks and pattern creation.

when i try this cmd , it don't give me an answer
lkfiras@elkfiras:~$ journalctl -u filebeat -f > test.log
do you mean this cmd "journalctl -u filebeat -f" or journalctl -u filebeat -f > test.log
when i try the first it give me this anser

elkfiras@elkfiras:~$ journalctl -u filebeat -f
août 22 13:14:02 elkfiras filebeat[159193]: {"log.level":"info","@timestamp":"2022-08-22T13:14:02.920Z","log.origin":{"file.name":"instance/beat.go","file.line":391},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
août 22 13:14:02 elkfiras filebeat[159193]: {"log.level":"error","@timestamp":"2022-08-22T13:14:02.920Z","log.origin":{"file.name":"instance/beat.go","file.line":1051},"message":"Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).","service.name":"filebeat","ecs.version":"1.6.0"}
août 22 13:14:02 elkfiras filebeat[159193]: Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
août 22 13:14:02 elkfiras systemd[1]: filebeat.service: Main process exited, code=exited, status=1/FAILURE
août 22 13:14:02 elkfiras systemd[1]: filebeat.service: Failed with result 'exit-code'.
août 22 13:14:03 elkfiras systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 1261.
août 22 13:14:03 elkfiras systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
août 22 13:14:03 elkfiras systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
août 22 13:14:03 elkfiras filebeat[159199]: {"log.level":"info","@timestamp":"2022-08-22T13:14:03.356Z","log.origin":{"file.name":"instance/beat.go","file.line":702},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
août 22 13:14:03 elkfiras filebeat[159199]: {"log.level":"info","@timestamp":"2022-08-22T13:14:03.356Z","log.origin":{"file.name":"instance/beat.go","file.line":710},"message":"Beat ID: 18f18c9b-6051-49ef-a4ce-e9cce3299b83","service.name":"filebeat","ecs.version":"1.6.0"}
août 22 13:14:06 elkfiras filebeat[159199]: {"log.level":"warn","@timestamp":"2022-08-22T13:14:06.359Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
août 22 13:14:06 elkfiras filebeat[159199]: {"log.level":"info","@timestamp":"2022-08-22T13:14:06.363Z","log.origin":{"file.name":"instance/beat.go","file.line":391},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
août 22 13:14:06 elkfiras filebeat[159199]: {"log.level":"error","@timestamp":"2022-08-22T13:14:06.367Z","log.origin":{"file.name":"instance/beat.go","file.line":1051},"message":"Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).","service.name":"filebeat","ecs.version":"1.6.0"}
août 22 13:14:06 elkfiras filebeat[159199]: Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
août 22 13:14:06 elkfiras systemd[1]: filebeat.service: Main process exited, code=exited, status=1/FAILURE
août 22 13:14:06 elkfiras systemd[1]: filebeat.service: Failed with result 'exit-code'.
août 22 13:14:06 elkfiras systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 1262.
août 22 13:14:06 elkfiras systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
août 22 13:14:06 elkfiras systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
août 22 13:14:06 elkfiras filebeat[159208]: {"log.level":"info","@timestamp":"2022-08-22T13:14:06.860Z","log.origin":{"file.name":"instance/beat.go","file.line":702},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}

do you mean this cmd "journalctl -u filebeat -f" or journalctl -u filebeat -f > test.log

when i try this cmd , it don't give me an answer
elkfiras@elkfiras:~$ nc -u 0.0.0.0 9002

elkfiras@elkfiras:~$ sudo ufw status verbose
Status: inactive
it's inactive i disable it

Yes because the command directed the logs into the file test.log that is basic Unix stuff and then I wanted to see the rule

BUT what you posted helps...

Your logs clearly say (if you look at the logs there is good information there)

août 22 13:14:06 elkfiras filebeat[159199]: Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).

So you have another fiebeat running already and you are trying to start another 1... You can only have 1 filebeat running so I am not sure why you have 1 already running and trying to start another but that is 1 problem.

Hello @stephenb sorry for the delay, Does it means Filebeat or module? because i run the Fortinet and cisco modules in the same Filebeat machine

Might be another Filebeat process has bean stuck. Stop Filebeat and check
ps aux | grep filebeat if is another process running kill it.

I don't know is important, when you start FB, on the first pic is port 9001.
Your configuration is using:

syslog_port: 9002 
syslog_protocol: udp

Please check are the same settings on the device side.

2 Likes