CSV not parsing logs

Elastic Stack Logstash
1

Hi there,
I'm new to the elastic stack, so please forgive me if i ask some obvious questions. I currently have logstash collecting logs from our palo alto firewall and im not able to see any new fields pop up from the filters i have in place. here is the filter:

these are the only fields in Kibana:

image
image.png256×680 12.1 KB

2

Please add

output { stdout { codec => rubydebug } }

and show us what one event looks like, with data sanitized according to your needs. We cannot see which tags were added, so we cannot tell which conditions were satisfied. Is the csv filter executing? Is it failing? We cannot tell until we see the whole event.

Alternatively, instead of posting an image of the Table tab, you could copy text from the JSON tab and post that with triple backtick before and after.

```
JSON goes here
```
3

Thank you for the reply Badger! Sorry for my late response
here is an event:

{
  "_index": "network-2018.06.13",
  "_type": "doc",
  "_id": "f9eu-WMBqWuAu2Ip5NEe",
  "_version": 1,
  "_score": null,
  "_source": {
    "port": 43810,
    "type": "syslog",
    "raw_message": "1,2018/06/13 08:06:18,serialnum,SYSTEM,dhcp,0,2018/06/13 08:06:17,,lease-start,,0,0,general,informational,\"DHCP lease started ip 10.xxx.xxx.xxx --> mac  - hostname iPhone, interface ethernet1/5\",6984574,0x8000000000000000,0,0,0,0,,PALOALTO",
    "message": "<14>Jun 13 08:06:18 Panorama 1,2018/06/13 08:06:18,serialnum,SYSTEM,dhcp,0,2018/06/13 08:06:17,,lease-start,,0,0,general,informational,\"DHCP lease started ip 10.xxx.xxx.xxx --> mac - hostname iPhone, interface ethernet1/5\",6984574,0x8000000000000000,0,0,0,0,,PALOALTO",
    "@version": "1",
    "syslog_pri": "14",
    "host": "Panorama",
    "role": "api-hello-app",
    "tags": [
      "netsyslog",
      "SYSTEM"
    ],
    "hostname": "Panorama",
    "@timestamp": "2018-06-13T15:06:18.422Z"
  },
  "fields": {
    "@timestamp": [
      "2018-06-13T15:06:18.422Z"
    ]
  },
  "sort": [
    1528902378422
  ]
}
4

Panorama does not match your hostname regex.

Also where you have 'if [tag] == "SYSTEM"' that should be 'if "SYSTEM" in [tags]'.

The csv filter works just fine.

5

Panorama is the sanitized hostname. its more like "Panorama-100_som_letters_b12-sw.domain"

so maybe im misunderstanding, csv doesnt break out the values in "raw_message" into more available fields? or do i need to write a separate grok for that?

6

If I take that message, change Panorama to Panorama-100_som_letters_b12-sw.domain, and make the changes to the check for [tags] then csv splits out the fields. This is truncated, but shows some of the fields...

           "Description" => "DHCP lease started ip 10.xxx.xxx.xxx --> mac - hostname iPhone, interface ethernet1/5",
       "dg_hier_level_2" => "0",
            "@timestamp" => 2018-06-13T15:39:27.526Z,
             "Log-Level" => "0",
           "ReceiveTime" => "2018/06/13 08:06:18",
       "dg_hier_level_3" => "0",
              "Severity" => "informational",
            "FUTURE_USE" => "1",
             "LogNumber" => "6984574",
                  "Type" => "SYSTEM",
                "Object" => nil,
              "hostname" => "Panorama-100_som_letters_b12-sw.domain",
       "dg_hier_level_1" => "0",
                  "type" => "syslog",
            "syslog_pri" => "14",
7

Ahhh i see it was the 'if "SYSTEM" in tags'. THANK YOU! you're awesome

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.