Grok help not getting data

Elastic Stack Logstash
1

Hi!

Been scratching my head on this all day. Could use some help. I am using elasticsearch 7.17.6. I have filebeat set to read log file, use a dissect processor, sends to logstash.

My problem is I am trying to grok this dissect field but I am unable to get any value.

The field is dissect.msg.

    "dissect": {
      "level": "D",
      "timestamp": "2024-06-01T11:55:50,070",
      "thread": "FL",
      "instance": "aa1",
      "msg": "[FM] (-156376144) Received 8=FIX.4.4\u00019=84\u000135=0\u000149=aa1-fix-trade1p\u000156=TEST\u000134=299\u000157=TEST\u000152=20240601-16:55:50.070\u0001112=TEST\u000110=173\u0001 ",
      "logger": "com.firm.nx.fix.monitoring.Log4jDebugAppender"
    },

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/share/filebeat/ingest_data/*.log
  multiline:
    pattern: '^\t'
    match: after
  processors:
    - add_tags:
        tags: ['fts']
    - dissect:
        field: message
        tokenizer: "%{level} %{timestamp} [%{thread}] %{logger} - %{msg}"
    - dissect:
        field: log.file.path
        tokenizer: "%{?path}/fts.%{instance}.%{?ext}"

logstash.conf

  if "fts" in [tags] and [dissect][msg] =~ /^\[FM\]/ {
    grok {
      match => { 'msg' => '^%{GREEDYDATA:fix-message}$'}
      add_tag => [ "fix_message" ]
      add_field => {
        "test" => "fix message %{fix-message}"
      }
    }
  }

I am expecting the test field to be

fix message [FM] (-156376144) Received 8=FIX.4.4\u00019=84\u000135=0\u000149=aa1-fix-trade1p\u000156=TEST\u000134=299\u000157=TEST\u000152=20240601-16:55:50.070\u0001112=TEST\u000110=173\u0001

But instead when I look in kibana it shows:

image
image709×60 3.25 KB

2

I'm not sure. May be you need to add another filter after the grok filter and move the add_field there?

The add_field operation might run before the grok is computed so there's no value yet in fix-message...

3

What you want to achieve with this grok filter? It would basically match the entire message, it does not make much sense to have this grok, a mutate copy field would do the same thing.

Also, does the field msg exists in your document? It seems that the correct field name is [dissect][msg] and not [msg].

Assuming that the correct field is dissect.msg, you could replace your current grok that just matches the entire message with these filters:

filter {
    mutate {
        copy => {
            "msg" => "fix-message"
        }
        add_tag => ["fix_message"]
    }
    mutate {
        add_field => {
            "teste" => "fix message %{fix-message}"
        }
    }
}

Also, when the order of the operation is important, you need to use different mutate blocks.

4

Hi!

Looks to be an issue with trying to get the dissect.msg field. I replaced dissect.msg with message and it captured everything in the message field as I would expect from GREEDYDATA. For now at least I know the issue is to do with logstash trying to get the data from the dissect.msg field. I don't need to take this any further. The reason I am using this seemingly useless capture is just as a proof to myself of how this grok works.

Thanks!