Need help - Cisco syslog events matching in GROK debugger but not showing up in

Elastic Stack Logstash
1

the actual grok pattern(s) used to match Cisco syslog event messages is no longer working despite the match is successful in the grok debugger UI .
other pattern is working fine for Juniper routers though... not sure what has changed to cause this problem

relevant section:

if [fileset][name] == "syslog" {
      grok {
        match => { "message" => [

                 "%{SYSLOGTIMESTAMP:[system][syslog][timestamp]}%{SPACE}%{SYSLOGHOST:[system][syslog][hostname]}%{SPACE}%{CISCOTIMESTAMP:cisco_timestamp}%{SPACE}%{DATA:timezone}:%{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?:%{SPACE}%{GREEDYMULTILINE:[system][syslog][message]}",
                  "%{SYSLOGTIMESTAMP:[system][syslog][timestamp]}%{SPACE}%{SYSLOGHOST:[system][syslog][hostname]}%{SPACE}%{DATA:[system][syslog][program]}%{SPACE}%{GREEDYMULTILINE:[system][syslog][message]}"
                  ]
                 }
        pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
        remove_field => "message"
      }
      date {
        match => [ "[system][syslog][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }

see attached

grok_debugger_working
grok_debugger_working1814×723 43 KB

2

Hi,

Can you add

stdout { codec => rubydebug }

To the output and show us the result.

Cad.

3

see below, thanks
For the Cisco-specific messages, I notice field timezone seems to be added before service --> type --> system..

i.e.

 "timezone" => "CST",
        "service" => {
    "type" => "system"

"timezone" => "JST",
"service" => {
"type" => "system"

output {
  stdout { codec => rubydebug }
#  elasticsearch {
#    hosts => ["localhost:9200"]
#    manage_template => false
#    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
#    document_type => "%{[@metadata][type]}"
#  }
}

root@sjc1upp-logc01 /etc/logstash/conf.d # /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/
Using JAVA_HOME defined java: /usr/lib/jvm/java-8-openjdk-amd64/jre/
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-08-01 09:23:12.303 [main] runner - Starting Logstash {"logstash.version"=>"7.13.4", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 25.292-b10 on 1.8.0_292-8u292-b10-0ubuntu1~18.04-b10 +indy +jit [linux-x86_64]"}
[WARN ] 2021-08-01 09:23:12.977 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-08-01 09:23:14.552 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2021-08-01 09:23:19.960 [Converge PipelineAction::Create<main>] Reflections - Reflections took 69 ms to scan 1 urls, producing 24 keys and 48 values 
[WARN ] 2021-08-01 09:23:22.032 [Converge PipelineAction::Create<main>] beats - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-08-01 09:23:22.333 [Converge PipelineAction::Create<main>] geoip - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-08-01 09:23:30.209 [[main]-pipeline-manager] grok - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2021-08-01 09:23:34.126 [[main]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-7.1.3-java/vendor/GeoLite2-City.mmdb", :healthy_database=>true}
[WARN ] 2021-08-01 09:23:34.393 [[main]-pipeline-manager] grok - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2021-08-01 09:23:34.687 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>24, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>3000, "pipeline.sources"=>["/etc/logstash/conf.d/02-beats-input.conf", "/etc/logstash/conf.d/10-syslog-filter.conf", "/etc/logstash/conf.d/30-elasticsearch-output.conf"], :thread=>"#<Thread:0x6d15fa30 run>"}
[INFO ] 2021-08-01 09:23:39.325 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>4.63}
[INFO ] 2021-08-01 09:23:39.396 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2021-08-01 09:23:39.425 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2021-08-01 09:23:39.564 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2021-08-01 09:23:39.638 [[main]<beats] Server - Starting server on port: 5044
{
       "service" => {
        "type" => "system"
    },
         "geoip" => {
        "location" => {
             "latitude" => "52.205276",
            "longitude" => "0.119167"
        }
    },
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
    "@timestamp" => 2021-08-01T17:36:55.000Z,
         "input" => {
        "type" => "log"
    },
      "@version" => "1",
       "fileset" => {
        "name" => "syslog"
    },
          "host" => {
                  "mac" => [
            [0] "d8:9d:67:2b:b5:74",
            [1] "d8:9d:67:2b:b5:75",
            [2] "f4:03:43:ae:90:e0",
            [3] "d8:9d:67:2b:b5:76",
            [4] "d8:9d:67:2b:b5:77",
            [5] "d8:9d:67:1f:37:74",
            [6] "d8:9d:67:1f:37:75",
            [7] "d8:9d:67:1f:37:76",
            [8] "f4:03:43:ae:90:e4",
            [9] "d8:9d:67:1f:37:77"
        ],
                 "name" => "sjc1upp-logc01",
             "hostname" => "sjc1upp-logc01",
         "architecture" => "x86_64",
                   "id" => "92114a03480d15c315d3352e5db14ce8",
                   "ip" => [
            [0] "10.20.60.200"
        ],
        "containerized" => false,
                   "os" => {
                "name" => "Ubuntu",
              "kernel" => "4.15.0-128-generic",
            "codename" => "bionic",
              "family" => "debian",
            "platform" => "ubuntu",
             "version" => "18.04.5 LTS (Bionic Beaver)",
                "type" => "linux"
        }
    },
         "event" => {
         "dataset" => "system.syslog",
          "module" => "system",
        "timezone" => "-07:00"
    },
         "agent" => {
                "name" => "sjc1upp-logc01",
            "hostname" => "sjc1upp-logc01",
                  "id" => "83f63913-48f3-432a-b907-122605159120",
             "version" => "7.13.4",
                "type" => "filebeat",
        "ephemeral_id" => "0ff63386-145c-4297-b90d-9911840d333a"
    },
           "log" => {
        "offset" => 890818,
          "file" => {
            "path" => "/var/log/network"
        }
    },
           "ecs" => {
        "version" => "1.9.0"
    },
        "system" => {
        "syslog" => {
              "message" => "IKE negotiation failed with error: Timed out. IKE Version: 1, VPN: sjc-vpn Gateway: sjc-wgw2, Local: 185.62.159.40/500, Remote: 192.147.44.247/500, Local IKE-ID: Not-Available, Remote IKE-ID: Not-Available, VR-ID: 0: Role: Initiator",
             "hostname" => "10.250.0.2",
            "timestamp" => "Aug  1 17:36:55"
        }
    }
}


{
           "timezone" => "CST",
            "service" => {
        "type" => "system"
    },
              "geoip" => {
        "location" => {
             "latitude" => "99.999999",
            "longitude" => "99.999999"
        }
    },
               "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
         "@timestamp" => 2021-08-01T09:22:10.000Z,
              "input" => {
        "type" => "log"
    },
           "@version" => "1",
            "fileset" => {
        "name" => "syslog"
    },
               "host" => {
                  "mac" => [
            [0] "d8:9d:67:2b:b5:74",
            [1] "d8:9d:67:2b:b5:75",
            [2] "f4:03:43:ae:90:e0",
            [3] "d8:9d:67:2b:b5:76",
            [4] "d8:9d:67:2b:b5:77",
            [5] "d8:9d:67:1f:37:74",
            [6] "d8:9d:67:1f:37:75",
            [7] "d8:9d:67:1f:37:76",
            [8] "f4:03:43:ae:90:e4",
            [9] "d8:9d:67:1f:37:77"
        ],
                 "name" => "sjc1upp-logc01",
             "hostname" => "sjc1upp-logc01",
         "architecture" => "x86_64",
                   "id" => "92114a03480d15c315d3352e5db14ce8",
                   "ip" => [
            [0] "10.20.60.200"
        ],
        "containerized" => false,
                   "os" => {
                "name" => "Ubuntu",
              "kernel" => "4.15.0-128-generic",
            "codename" => "bionic",
              "family" => "debian",
            "platform" => "ubuntu",
             "version" => "18.04.5 LTS (Bionic Beaver)",
                "type" => "linux"
        }
    },
              "agent" => {
                "name" => "sjc1upp-logc01",
            "hostname" => "sjc1upp-logc01",
                  "id" => "83f63913-48f3-432a-b907-122605159120",
             "version" => "7.13.4",
                "type" => "filebeat",
        "ephemeral_id" => "0ff63386-145c-4297-b90d-9911840d333a"
    },
              "event" => {
         "dataset" => "system.syslog",
          "module" => "system",
        "timezone" => "-07:00"
    },
                "log" => {
        "offset" => 894244,
          "file" => {
            "path" => "/var/log/network"
        }
    },
                "ecs" => {
        "version" => "1.9.0"
    },
             "system" => {
        "syslog" => {
              "message" => "Line protocol on Interface GigabitEthernet1/0/43, changed state to down",
             "hostname" => "10.15.200.11",
              "program" => "%LINEPROTO-5-UPDOWN",
            "timestamp" => "Aug  1 09:22:10"
        }
    },
    "cisco_timestamp" => "Aug  2 2021 00:22:09.787"
}
{
           "timezone" => "JST",
            "service" => {
        "type" => "system"
    },
              "geoip" => {
        "location" => {
             "latitude" => "35.654902",
            "longitude" => "139.774605"
        }
    },
               "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
         "@timestamp" => 2021-08-01T09:21:31.000Z,
            "fileset" => {
        "name" => "syslog"
    },
           "@version" => "1",
              "input" => {
        "type" => "log"
    },
               "host" => {
                  "mac" => [
            [0] "d8:9d:67:2b:b5:74",
            [1] "d8:9d:67:2b:b5:75",
            [2] "f4:03:43:ae:90:e0",
            [3] "d8:9d:67:2b:b5:76",
            [4] "d8:9d:67:2b:b5:77",
            [5] "d8:9d:67:1f:37:74",
            [6] "d8:9d:67:1f:37:75",
            [7] "d8:9d:67:1f:37:76",
            [8] "f4:03:43:ae:90:e4",
            [9] "d8:9d:67:1f:37:77"
        ],
                 "name" => "sjc1upp-logc01",
             "hostname" => "sjc1upp-logc01",
                   "id" => "92114a03480d15c315d3352e5db14ce8",
                   "ip" => [
            [0] "10.20.60.200"
        ],
         "architecture" => "x86_64",
        "containerized" => false,
                   "os" => {
                "name" => "Ubuntu",
              "kernel" => "4.15.0-128-generic",
            "codename" => "bionic",
              "family" => "debian",
            "platform" => "ubuntu",
             "version" => "18.04.5 LTS (Bionic Beaver)",
                "type" => "linux"
        }
    },
              "event" => {
         "dataset" => "system.syslog",
          "module" => "system",
        "timezone" => "-07:00"
    },
              "agent" => {
                "name" => "sjc1upp-logc01",
            "hostname" => "sjc1upp-logc01",
                  "id" => "83f63913-48f3-432a-b907-122605159120",
             "version" => "7.13.4",
                "type" => "filebeat",
        "ephemeral_id" => "0ff63386-145c-4297-b90d-9911840d333a"
    },
                "log" => {
        "offset" => 891953,
          "file" => {
            "path" => "/var/log/network"
        }
    },
                "ecs" => {
        "version" => "1.9.0"
    },
             "system" => {
        "syslog" => {
              "message" => "processing Get of entSensorValue.1012 (30017 msecs)",
             "hostname" => "10.110.4.5",
              "program" => "%SNMP-3-RESPONSE_DELAYED",
            "timestamp" => "Aug  1 09:21:31"
        }
    },
    "cisco_timestamp" => "Aug  2 2021 01:21:30.119"
}
4

That appears to have worked. What do you see wrong with it?

5

issue is that the Cisco-specific events are not making it into Kibana... I don't have any filters applied blocking them so I'm stumped.

and just in case I have this for my output block:

output {
  stdout { codec => rubydebug }
  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
6

Why are you setting document_type? That option is deprecated and will be removed, I would guess in 8.0. If this is not the same on every document then documents will get rejected.

I would expect you to be getting the warning and exceptions posted here.

7

ok thanks , let me remove and re-run.. used to be from the previous 6.8.x setup..

8

even after removing that line, still no improvement..
I've tried reindexing again, and although output to console shows the Cisco-based messages parsing, they don't appear in "Analytics --> Discover" , nor in " Observability --> Logs"
the only message that are showing up are the Juniper-based messages... it's so weird...

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.