Cisco filebeat module not listening on port as configured

Elastic Stack Elasticsearch
1

We have an existing functional Elastic instance running with Filebeat 8.9, running on Ubuntu 22.04. We're attempting to add Cisco logs using the Cisco filebeat module. However, we're not seeing any logs coming in. We have verified connectivity between the hosts.

Our cisco.yml file is as follows:

  ios:
    enabled: true
    var.syslog_host: 0.0.0.0
    var.syslog_port: 9002

Our filbeat.yml file is as follows:

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: false
  paths:
    - /var/log/*.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: "[redacted]"
  password: "[redacted]"
  ssl:
    enabled: true
    ca_trusted_fingerprint: "[redacted]"

I can see that port 9002 does not appear to be open according to netstat:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      2110/master
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1297/sshd: /usr/sbi
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      686349/node
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      1216/systemd-resolv
tcp6       0      0 :::25                   :::*                    LISTEN      2110/master
tcp6       0      0 :::22                   :::*                    LISTEN      1297/sshd: /usr/sbi
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      717799/java
tcp6       0      0 :::9200                 :::*                    LISTEN      717799/java
tcp6       0      0 ::1:9300                :::*                    LISTEN      717799/java
tcp6       0      0 :::27768                :::*                    LISTEN      2675/zeek
tcp6       0      0 :::27764                :::*                    LISTEN      2670/zeek
tcp6       0      0 :::27765                :::*                    LISTEN      2671/zeek
tcp6       0      0 :::27766                :::*                    LISTEN      2677/zeek
tcp6       0      0 :::27767                :::*                    LISTEN      2678/zeek
tcp6       0      0 :::27761                :::*                    LISTEN      2464/zeek
tcp6       0      0 :::27762                :::*                    LISTEN      2514/zeek
tcp6       0      0 :::27763                :::*                    LISTEN      2570/zeek

Here's my syslog | grep filebeat:

Also, what's the best way to search for logs coming into this module? Should we search for agent.name: "cisco.ios" or something?

Any help would be greatly appreciated!

2

can you try

filebeat test config

Also your logs do not show the startup of filebeat where the UDP listener would get opened. First 100 lines or so.

Also those logs show other modules / paths are you sure you are running what you are showing above

"source_file":"/mnt/Bro/current/pe.log",

Aslo you did not show the full cisco.yml

I do not see

- module: cisco
3

Thank you stephenb.

filebeat test config

Shows:

Config OK

When I restart filebeat using

filebeat -e

I see the following:

My full cisco.yml is:

- module: cisco
  asa:
    enabled: false
  ftd:
    enabled: false
  ios:
    enabled: true
    var.syslog_host: 0.0.0.0
    var.syslog_port: 9002
  nexus:
    enabled: false
  meraki:
    enabled: false
  umbrella:
    enabled: false
  amp:
    enabled: false

As to your question about:

"source_file":"/mnt/Bro/current/pe.log",

We're using this ES instance for Zeek log files as well (if I understand your question).

4

So you have alot of modules enable I would disable the other to temporarily debug do you have more than one module opening UDP


# Here lots of modules makes it hard to debug
{"log.level":"info","@timestamp":"2023-08-30T15:43:10.738Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: cisco (ios)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-30T15:43:10.759Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: zeek (smb_files), zeek (smtp), zeek (rfb), zeek (ssl), zeek (ocsp), zeek (ssh), zeek (radius), zeek (stats), zeek (dns), zeek (sip), zeek (tunnel), zeek (irc), zeek (connection), zeek (snmp), zeek (smb_mapping), zeek (kerberos), zeek (http), zeek (dnp3), zeek (intel), zeek (files), zeek (traceroute), zeek (modbus), zeek (weird), zeek (ftp), zeek (ntlm), zeek (pe), zeek (capture_loss), zeek (smb_cmd), zeek (dce_rpc), zeek (syslog), zeek (mysql), zeek (rdp), zeek (notice), zeek (socks), zeek (dpd), zeek (dhcp), zeek (x509)","service.name":"filebeat","ecs.version":"1.6.0"}



{"log.level":"info","@timestamp":"2023-08-30T15:43:10.805Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":163},"message":"Config reloader started","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-30T15:43:10.807Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: cisco (ios)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-30T15:43:10.811Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://localhost:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-30T15:43:10.840Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":162},"message":"'ca_trusted_fingerprint' set, looking for matching fingerprints","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-30T15:43:10.841Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":173},"message":"CA certificate matching 'ca_trusted_fingerprint' found, adding it to 'certificate_authorities'","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-30T15:43:10.844Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.9.1","service.name":"filebeat","ecs.version":"1.6.0"}

# HERE 
{"log.level":"info","@timestamp":"2023-08-30T15:43:10.847Z","log.logger":"syslog","log.origin":{"file.name":"syslog/input.go","file.line":147},"message":"Starting Syslog input","service.name":"filebeat","protocol":"udp","ecs.version":"1.6.0"}



{"log.level":"info","@timestamp":"2023-08-30T15:43:10.847Z","log.logger":"UDP","log.origin":{"file.name":"dgram/server.go","file.line":99},"message":"Started listening for UDP connection","service.name":"filebeat","ecs.version":"1.6.0"}

I also see this closing...

{"log.level":"info","@timestamp":"2023-08-30T15:46:18.667Z","log.logger":"udp","log.origin":{"file.name":"dgram/handler.go","file.line":68},"message":"Connection has been closed","service.name":"filebeat","address":"0.0.0.0:9002","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-30T15:46:18.667Z","log.origin":{"file.name":"input/input.go","file.line":134},"message":"input ticker stopped","service.name":"filebeat","ecs.version":"1.6.0"}

not sure if you were shutting it down...

I would isolate / turn of everything and you and also start with the following which will show every message in/out

filebeat -e -d "*"

You could also set logging to debug.

You also need to look through the logs there is info there.

5

Ok thanks again.

Yes I noticed the error as you pointed out:

"Connection has been closed"

I think I had issued ctrl-c and not known that action would shut down filebeat. I tried again and don't see that message.

I disabled all but the Cisco module, and issued

filebeat -e -d "*"

I sent a log while this was running. Here's what I'm seeing:

I don't see anything in there that looks suspicious, except this on line 89:

"message":"Pipeline already exists in Elasticsearch."
6

Yup Looks pretty normal

Do you see the UDP port open?

did you try to netcat a couple of messages?

if it is reading them you should see them show up in the logs when you use the -d "*"

when you are running netstat are you asking for udp? can you show the command?

7

Aha, I think I had been using netcat incorrectly. I tried this to see UDP:

 sudo netstat -nulp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
udp        0      0 127.0.0.53:53           0.0.0.0:*                           1216/systemd-resolv
udp6       0      0 :::9002                 :::*                                1108401/filebeat

This seems to suggest I'm listening on IPv6 not IPv4?

1 Like
8

that I can not really help you with....
mine on mac say udp46 I suspect it is running on both unless you have some special network settings

udp46 0 0 *.9002

9

Looks like someone else had the same issue: Filebeat Cisco Module: Listening on IPV6 only?

So in the cisco.yml file, I specified the host IP instead of 0.0.0.0. That seems to have helped:

netstat -nulp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
udp        0      0 127.0.0.53:53           0.0.0.0:*                           -
udp        0      0 [host IP]:9002         0.0.0.0:*                           -

I tried to netcat a message:

$ echo ‘hello’ | nc -v -u -w 0 [host IP] 9002
Connection to [host IP] 9002 port [udp/*] succeeded!

However, I see nothing in the logs.

10

it takes more than hello... there is a buffer size etc...

nc -u 127.0.0.1 9002
hello lsakdjfaslkdfjh lsakdjfhasldkfjh
ladksjfhasldkfjh aslkdfjhasdflkjh

in logs

{"log.level":"debug","@timestamp":"2023-08-30T10:44:21.342-0700","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2023-08-30T17:44:21.342Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.9.0\",\n    \"truncated\": false,\n    \"pipeline\": \"filebeat-8.9.0-cisco-ios-pipeline\"\n  },\n  \"agent\": {\n    \"name\": \"hyperion\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.9.0\",\n    \"ephemeral_id\": \"f2d418c9-8fce-416e-b4d1-36309fcd56f5\",\n    \"id\": \"bc6fa02a-862e-4019-80db-fd6b422889b9\"\n  },\n  \"message\": \"ladksjfhasldkfjh aslkdfjhasdflkjh \\n\",\n  \"log\": {\n    \"source\": {\n      \"address\": \"127.0.0.1:64020\"\n    },\n    \"flags\": [\n      \"dissect_parsing_error\"\n    ]\n  },\n  \"event\": {\n    \"module\": \"cisco\",\n    \"dataset\": \"cisco.ios\",\n    \"timezone\": \"-07:00\",\n    \"original\": \"ladksjfhasldkfjh aslkdfjhasdflkjh \\n\"\n  },\n  \"fileset\": {\n    \"name\": \"ios\"\n  },\n  \"ecs\": {\n    \"version\": \"1.12.0\"\n  },\n  \"error\": {\n    \"message\": \"GoError: could not find delimiter: `` in remaining: `ladksjfhasldkfjh aslkdfjhasdflkjh \\n`, (offset: 0)\"\n  },\n  \"tags\": [\n    \"cisco-ios\",\n    \"forwarded\",\n    \"_js_exception\"\n  ],\n  \"service\": {\n    \"type\": \"cisco\"\n  },\n  \"input\": {\n    \"type\": \"syslog\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
11

Alas, still no.

I was poking around in the GUI, and I see there's an error message related to the Cisco module:

GoError: could not find delimiter: `` in remaining: `
`, (offset: 0)`Preformatted text`

Someone else had this issue, so I'll poke around where he was directed: Error with Cisco module for filebeat

12

Not sure where you got that error message... where you are seeing that.

First is the port open.

2nd when you do -d "*" do you see it in the filebeat logs

Then when you send cisco data is it getting to the filebeat hosts

Then you could see with the -d "*"

Then is the cisco data showing up in the elastic filebeat index

Then is it parsing correctly

Then is it in the dashboards

13

Yes, port is open according to netstat:

netstat -nulp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
udp        0      0 127.0.0.53:53           0.0.0.0:*                           -
udp        0      0 1[host IP]:9002         0.0.0.0:*                           -

No, when I do a -d "*" I don't see anything in the logs: no messages generated by netcat, nor real Cisco logs being sent by a Cisco IOS device.

The log message I mention is in the Elastic Discover dashboard.

14

Untill you see them there ... nothing else matter ... nothing is being read / getting that port ... Firewall?

Try the netcat with multi-line... there is something basic going on ... Firewall?

15

After quite a bit of digging, I discovered that the logs are actually making it to the Elastic dashboard, but not visible in -d "*".

There is one remaining issue: every log entry is generating an error message, for instance: GoError: could not find delimiter: `` in remaining: Interface GigabitEthernet1/0/37, changed state to up, (offset: 0)

All of the suggestions offered so far have been incredibly helpful. So thank you again!

16

I think the -d "*" not working is perhaps related to the error.

And you are sure you are sending it ios logs not asa etc

Or perhaps you have customized the ios logs that could cause an issue because filbeat would be expecting the defaults.

So it turns out that there is a client (filebeat) side processing on the ios data.... along with some other data sets.

It is javascript that runs from filebeat / go my suspicion is your log format and that script are in conflict and that is where the error is coming from.

  - script:
      lang: javascript
      id: cisco_ios
      file: ${path.home}/module/cisco/ios/config/pipeline.js

So what to do .... just for grins you could try just the raw UDP input and see what the messages look like on the elastic side, you might to do some of your own parsing.

17

OK I'll check that out. And yes, I'm sure I'm sending IOS logs. I haven't customized them at all.

18

Ok so apparently I didn't RTFM. The Cisco Filebeat module is apparently only for ACL entries, not generic Syslog events from IOS devices.

1 Like
19

So perhaps try just a generic SYSLOG integration and see what you get...
You could start with that.. .
Then you could add some custom parsing on top of that.

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.