Csv filter plugin not working

Elastic Stack Logstash
1

I would like to use logstash to format my logs.

Specifically, I have a log with the following format

format:
timestamp [thread-name] log-level class-name - log-message

example:
2021-03-30 09:38:20.201 [ConnectionChecker] ERROR c.b.a.websocket.WebSocketOnlineUser - SendPing failed. roomHash:null,roomUserHashnull,error:org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]

I am hoping that I can use the filter plugin to get the 5 items separately, but that doesn't work.
If the problem can be solved with a method other than csv filter, that method is also welcome.

Here is the description and output of the filter I set in config.

... snip ...
filter {
  csv {
    separator => "	"
    skip_header => true
  }
}
... snip ...

{
          "host" => "ITS-ELS-01",
       "column1" => "2021-03-30 09:38:20.201\t[ConnectionChecker]\tERROR\tc.b.a.websocket.WebSocketOnlineUser - SendPing failed. roomHash:null,roomUserHashnull,error:org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]",
    "@timestamp" => 2021-03-30T11:05:40.582Z,
       "message" => "2021-03-30 09:38:20.201\t[ConnectionChecker]\tERROR\tc.b.a.websocket.WebSocketOnlineUser - SendPing failed. roomHash:null,roomUserHashnull,error:org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]",
      "@version" => "1"
}
2

I would suggest using a dissect filter.

3

Thanks for the advice.
I read the dissect page and rewrote the filter plugin as follows.

filter {
  dissect {
    mapping => {
      "message" => "%{timestamp}  [%{thread}] %{loglevel} %{class-name} - %{logmessage}"
    }
  }
}

However, the result is as follows

{
          "tags" => [
        [0] "_dissectfailure"
    ],
      "@version" => "1",
    "@timestamp" => 2021-03-31T01:08:25.984Z,
       "message" => "2021-03-30 09:38:20.201\t[ConnectionChecker]\tERROR\tc.b.a.websocket.WebSocketOnlineUser - SendPing failed. roomHash:null,roomUserHashnull,error:org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]",
          "host" => "ITS-ELS-01"
}

Can you please tell me more about this?

4

If your message has tab separated fields then you need to use tab as the separator in the dissect.

5

Thanks for the advice.

I tried both the direct tab and the \t method, but the result was the same.

Is there anything else that could cause this?

6

Many things. I suggest you read this, which talks about grok, but the same applies to dissect. Build the pattern one field at a time.

7

Build the pattern one field at a time.
Hmmm. I'm not sure.

I tried it with spaces and got the following split.

dissect {
  mapping => {
     "message" => "%{a} %{b} %{c} %{d}"
  }
}

{
     "@timestamp" => 2021-03-31T02:49:55.503Z,
           "host" => "ITS-ELS-01",
              "c" => "-",
              "d" => "SendPing failed. roomHash:null,roomUserHashnull,error:org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]",
       "@version" => "1",
              "a" => "2021-03-30",
        "message" => "2021-03-30 09:38:20.201\t[ConnectionChecker]\tERROR\tc.b.a.websocket.WebSocketOnlineUser - SendPing failed. roomHash:null,roomUserHashnull,error:org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]",
              "b" => "09:38:20.201\t[ConnectionChecker]\tERROR\tc.b.a.websocket.WebSocketOnlineUser"
}

To get the intended split, I want to split item "b" further with a tab. Do I use the actual notation " " for the tabs instead of \t?

8

I'd like to add a few things.
It doesn't seem to work when the delimiter is tab.
I've tried running both, but I can't get it to split properly.

  mapping => {
     "message" => "%{a} %{b} %{c} %{d}"
     "b" => "%{be}\t%{bf}\t%{bg}\t%{bh}"
  }

  mapping => {
    "message" => "%{a} %{b} %{c} %{d}"
    "b" => "%{be}  %{bf}  %{bg}  %{bh}"
  }

Does dissect support tab as delimiter?

9

We have sorted out the problems.

  • want to
    I want to use logstash to format tab-delimited logs.

  • issue
    Tabbed input cannot be split.

  • experiment
    input:

    A	B

    A and B are separated by a tab.

    settings:

    ... snip ...
filter {
  dissect {
    mapping => {
      "message" => "%{a}  %{b}"
    }
  }
}
... snip ...

    expect:

    {
  ... snip ...
          "a" => "A",
          "b" => "B",
    "message" => "A\tB",
  ... snip ...
}

    actual:

    {
  ... snip ...
  "message" => "A\tB",
  ... snip ...
}
10

Changing the config.support_escapes setting in logstash.yml from false to true did not change the result.

11

dissect does support tabs

input { generator { count => 1 lines => [ 'a        b' ] } }
filter {
    dissect { mapping => { "message" => "%{a}       %{b}" } }
}
output  { stdout { codec => rubydebug { metadata => false } } }

Use a literal tab in the dissect filter. That results in

         "b" => "b",
   "message" => "a\tb",
         "a" => "a",
12

Does the item mapped by dissect (e.g. %{a}) get added to the field?

Do I need to use mutate's add_field?
I'm not sure if I need to use the mutate add_field or not, but the following description shows "%{a}" as it is, not the retrieved value.

I want to use dissect to display the value corresponding to %{a} in the field.

mutate {
  add_field => { "a" => "%{a}" }
}
13

Perhaps I should have used different values

input { generator { count => 1 lines => [ 'x        y' ] } }
filter {
    dissect { mapping => { "message" => "%{a}       %{b}" } }
}
output  { stdout { codec => rubydebug { metadata => false } } }

would produce

      "b" => "y",
"message" => "x\ty",
      "a" => "x",

If you use

mutate { add_field => { "a" => "%{a}" } }

then that is a no-op. It replaces the value of field [a] with the value of the field [a]

14

mutate does not work properly.

"%{a}" is still displayed as %{a}.
I want to output the value that applies to %{a}.

15

Perhaps I should have used different values

I did the following.

dissect {
  mapping => {
    "message" => "%{ia} %{ib}"
  }
}
mutate {
  add_field => { "a" => "%{ia}" }
  add_field => { "b" => "%{ib}" }
}

In debug, it displays nicely as follows

output  { stdout { codec => rubydebug { metadata => false } } }

"a" => "a",
"b" => "b"

However, when I send it to elasticsearch, I get the same notation (e.g., "%{ia}").
Why is this?

16

In Kibana, it looks like this

image

17

Please do not post pictures of text. They are unreadable and unsearchable. I have no idea what your Kibana data looks like.

18

I'm sorry.
It is the same as the explanation above.
It means that the variables are displayed, but the values are not.
This is the way it is in Kibana I have seen.
I just wanted to show that.

19

Specifically, I want to say that the value of field "a" will not be displayed as "x", but as %{ia}, when set as follows.

input { generator { count => 1 lines => [ 'x        y' ] } }
filter {
    dissect { mapping => { "message" => "%{a}       %{b}" } }
    add_field => { "a" => "%{ia}" }
}
output {
  elasticsearch {
    hosts => ["localhost"]
    index => "other-%{+YYYY.MM.dd}"
  }
}
20

Does anyone have any good ideas?