Strange Grok with message

Elastic Stack Logstash
1

Hello guys

I have this logstash filter

filter {
  if [log][file][path] == "/dwdwdosw/Top/log/services/uds-default-pippo.it" or [log][file][path] == "/aledwgosw/Top/log/services/uds-default-pluto.it" {
    mutate {
      gsub => [
        "message", '\",\s+\"', '","'  # sostituisce ", " con "," per tutti i campi
      ]
    }

     csv {
      separator => ","
      quote_char => '"'
      autogenerate_column_names => false
      columns => [
        "timestamp",
        "loglevel",
        "server",
        "app",
        "service",
        "message",
        "extra"
      ]
    }

    mutate {
      remove_field => ["app", "service", "extra"]
      strip => ["timestamp", "loglevel", "server", "message"]
    }
  } else {

  grok {
    match => {
      "message" => [
        "%{TIMESTAMP_ISO8601:timestamp}\s+%{NOTSPACE:loglevel}\s+\[%{DATA}\s*\]%{DATA:message}$",
        "%{TIMESTAMP_ISO8601:timestamp}\s+%{USERNAME:loglevel}\s+\[%{USERNAME}]\s+%{DATA:message}$",
        "%{DATE:date}_%{TIME:time}\s+\[%{USERNAME}\]\s+%{WORD:loglevel}\s+%{DATA:message}$",
        "%{INT:date}?\s+%{TIME:time}?\s+%{USERNAME:loglevel}?\s+%{DATA}\s+%{DATA:message}$",
        "\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{NOTSPACE:loglevel}\]\[%{DATA}\s*\]%{DATA:message}$",
        "\[%{TIMESTAMP_ISO8601:timestamp}\]\s+%{DATA:message}$"
      ]
    }
  }
 }

  date {
    match => ["timestamp", "YYYY-MM-dd'T'HH:mm:ss.SSSZ", "YYYY-MM-dd HH:mm:ss,SSS", "YYYY-MM-dd HH:mm:ss"]
    target => "@timestamp"
  }

  # Rimuove il campo originale `timestamp` per mantenere pulito il documento
  mutate {
    remove_field => ["timestamp"]
  }

}

I have many log format.

taking this for an example

1126 13:27:21,692 INFO  [main] java.rmi.server.hostname = 'pippo.pluto.it'

The correct pattern for this on is :

 "%{INT:date}?\s+%{TIME:time}?\s+%{USERNAME:loglevel}?\s+%{DATA}\s+%{DATA:message}$"

Why in kibana i see all the field “date”, “time” “loglevel” in correct format. But for message is see all the log instead : java.rmi.server.hostname = 'pippo.pluto.it' ?

What am I missing?

Thanks for help

2

I am sorry, but you have one open thread on this type of thing already, and the csv filter here seems like a copy from there? Please dont do this.

You also marked this as URGENT - if you need urgent help, then I suggest you consider to pay someone? I am available, for a very reasonably hourly rate :slight_smile:

This is a community forum. We dont have P1, P2, whatever tickets/incidents here.

1126 matches to %{INT:date}
13:27:21,692 marches to %{TIME:time}
INFO matches to %{USERNAME:loglevel}
[main] matches to %{DATA}
java.rmi.server.hostname = 'pippo.pluto.it' matches to %{DATA:message}$

with a few \s+ in between

I dont know.

3

the pipeline is the same but different problem. To be correct i share all my filter but the csv filter is not relevant for this topic.

4

You are parsing the field [message] to create the field [message]. This results in an array. Try adding overwrite => [ "message" ] to your grok filter.

   "message" => [
    [0] "1126 13:27:21,692 INFO  [main] java.rmi.server.hostname = 'pippo.pluto.it'",
    [1] "java.rmi.server.hostname = 'pippo.pluto.it'"
],

I would also recommend anchoring all your patterns to the start of the line using ^ at the beginning.

1 Like
5

Oh. Missed that, good spot. I guess I wont be getting a call from a new customer :slight_smile: