Need help with Logstash Configuration Error - _grokparsefailure

[taweisse]

Hi all, I have been struggling with my Logstash configuration file for 2 days now and cannot figure out what the issue is. I am trying to parse regularly structured data using the grok filter, and no matter what I try, I keep getting _grokparsefailure showing up in tags, even though all my fields are being parsed.

I am trying to parse the following log lines:

12/20/2016 09:00:03 Tran/s: 15534 CPU/sec: 3547172 CPU %: 5 DB2-cpu %: 2 Db2-mem%: 69 Swap free: 6881

07/31/2017 12:00:02 Tr/s: 12568 CPU/s: 3972144 CPU %: 6 DB-cpu %: 5 Db-mem%: 67 Swp free: 6466 TPOINTR: 25 TSHUDFC: 321 TSMTHST: 393 TSMTUNT: 642

11/06/2017 15:00:02 Tr/s: 0 CPU/s: 29 CPU %: 7 DB-cpu %: 6 Db-mem%: 70 Swp free: 3701 TOGZFLO: 11 TPOINTR: 24 TSHUDFC: 426 TSMTHST: 410 TSMTUNT: 750

Using the the filter:

%{DATESTAMP:ts} Tr(an)?/s: (%{NUMBER:tr_sec})? CPU/s(ec)?: (%{NUMBER:cpu_sec})? CPU %: (%{NUMBER:cpu_pct})? DB(2)?-cpu %: (%{NUMBER:dbcpu_pct})? Db(2)?-mem%: (%{NUMBER:dbmem_pct})? Sw(a)?p free: (%{NUMBER:swp_free})?( (TOGZFLO: (%{NUMBER:togzflo})? )?TPOINTR: (%{NUMBER:tpointr})? TSHUDFC: (%{NUMBER:tshudfc})? TSMTHST: (%{NUMBER:tsmthst})? TSMTUNT: (%{NUMBER:tsmtunt})?)?

As well as:

11/29/2017 20:00:03 d485aa Std Metrics; Tr/s: 34396 CPU/s: CPU %: 11 DB-cpu %: 8 Db-mem%: 74 Swp free: 11252

Using this filter:

%{DATESTAMP:ts} %{USERNAME:db_name} Std Metrics; Tr/s: (%{NUMBER:tr_sec})? CPU/s: (%{NUMBER:cpu_sec})? CPU %: (%{NUMBER:cpu_pct})? DB-cpu %: (%{NUMBER:dbcpu_pct})? Db-mem%: (%{NUMBER:dbmem_pct})? Swp free: (%{NUMBER:swp_free})?

The first filter sometimes fails, and sometimes succeeds, even with lines in the exact same format. (Only the numbers change). The second filter fails no matter what. I have also tried to strip down the grok filter to the timestamp only, which still fails. When the filter does fail, all of the expected fields are present, but the Elasticsearch index does not include the successfully parsed fields. These two filters are in 2 seperate configuration files, but each configuration only acts on logs of the expected format. (The file names are different). I have tried with only one config file installed at a time with the same result. I even tried running a configuration file based on the grok documentation page using the filter:

%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}

and lines:

55.3.244.1 GET /index.html 15824 0.043

which STILL fails. Any help regarding this would be greatly appreciated, I am really stuck trying to figure it out myself.

Thanks.

[lueneburger (Dirk L.)]

i think your logstash config would be a great help

[taweisse]

input {
        file {
                path => "/home/db2inst/scripts/mon/logs/metrics_prod_orig.txt"
                stat_interval => "5"
        }
}

filter {
        grok {
                match => {
                        "message" => "%{DATESTAMP:ts} Tr(an)?/s:  (%{NUMBER:tr_sec})? CPU/s(ec)?:  (%{NUMBER:cpu_sec})? CPU %: (%{NUMBER:cpu_pct})? DB(2)?-cpu %: (%{NUMBER:dbcpu_pct})? Db(2)?-mem%: (%{NUMBER:dbmem_pct})? Sw(a)?p free: (%{NUMBER:swp_free})?( (TOGZFLO: (%{NUMBER:togzflo})? )?TPOINTR: (%{NUMBER:tpointr})? TSHUDFC: (%{NUMBER:tshudfc})? TSMTHST: (%{NUMBER:tsmthst})? TSMTUNT: (%{NUMBER:tsmtunt})?)?"
                }
        }

        date {
                locale => "en"
                match => [ "ts", "MM/dd/yyyy HH:mm:ss" ]
                timezone => "US/Eastern"
                target => "@timestamp"
                remove_field => [ "ts", "message" ]
        }

        mutate {
                convert => { "tr_sec" => "integer" }
                convert => { "cpu_sec" => "integer" }
                convert => { "cpu_pct" => "integer" }
                convert => { "dbcpu_pct" => "integer" }
                convert => { "dbmem_pct" => "integer" }
                convert => { "swp_free" => "integer" }
                convert => { "togzflo" => "integer" }
                convert => { "tpointr" => "integer" }
                convert => { "tshudfc" => "integer" }
                convert => { "tsmthst" => "integer" }
                convert => { "tsmtunt" => "integer" }
        }
}

output {
        elasticsearch
        {
                hosts => ["localhost:9200"]
                index => "metrics-prod-%{+YYYY.MM}"
        }
        stdout { codec => rubydebug }
}

========= And:

input {
        file {
                path => "/home/db2inst/scripts/mon/logs/*_std_metrics.txt"
                stat_interval => "5"
        }
}

filter {
        grok {
                match => {
                        "message" => "%{DATESTAMP:ts} %{USERNAME:db_name} Std Metrics; Tr/s: (%{NUMBER:tr_sec})? CPU/s: (%{NUMBER:cpu_sec})? CPU %: (%{NUMBER:cpu_pct})? DB-cpu %: (%{NUMBER:dbcpu_pct})? Db-mem%: (%{NUMBER:dbmem_pct})? Swp free: (%{NUMBER:swp_free})?"
                }
        }

        date {
                locale => "en"
                match => [ "ts", "MM/dd/yyyy HH:mm:ss" ]
                timezone => "US/Eastern"
                target => "@timestamp"
                remove_field => [ "ts", "message" ]
        }

        mutate {
                convert => { "tr_sec" => "integer" }
                convert => { "cpu_sec" => "integer" }
                convert => { "cpu_pct" => "integer" }
                convert => { "dbcpu_pct" => "integer" }
                convert => { "dbmem_pct" => "integer" }
                convert => { "swp_free" => "integer" }
        }
}

output {
        elasticsearch
        {
                hosts => ["localhost:9200"]
                index => "metrics-prod-%{+YYYY.MM}"
        }
        stdout { codec => rubydebug }
}

[magnusbaeck (Magnus Bäck)]

Your log lines are being filtered by both your grok filters and apparently only one of them matches. All events from all inputs will reach all filters and all outputs. You can avoid that by using the multi-pipeline feature in Logstash 6 or by wrapping filters and outputs in conditionals.

This is a FAQ so you'll find dozens of more explanations in past threads.