FixProtocol delimiter confusion

[Ramesh_Janagam]

Hi All,

We are processing FIX protocol messages in logstash( without Fix protocol plugin). but still I am not able to figure out 'delimiter'. We know its '\001' in ascii but still it not working. In logstash its displaying as '\u' so tried that also but still not parsed in kv plugin.

We are replacing '\u' with pipe.

mutate {
     #gsub => [ "message", "\\u(?:[0-1])", "|"]
     #gsub => [ "message", "\\(?:[u001]{5})", "|"]
     gsub => [ "message", "\\u", "|"]
  }    
  grok {
    match => [ "message", "%{TIME:logtime} \<%{NUMBER:threadid}\> RECV: %{GREEDYDATA:app_data}" ]
  }

  kv {
     source => ["app_data"]
     field_split => "|"
  }

any suggestion please?

[Tribbles (Thorsten)]

Hi Ramesh,

had the same Problem in processing FIX Logfiles. You already are on the correct track, so to say.

The solution I use ist to first of all get rid of the HEX01 Delimiter:

mutate {
gsub => [
"message", "\x01", "^"
]
}

After this, you can split up the fields like this:

# split each of these filed with Key = Value and set new fields by the name of Key
kv {
  field_split => "^"
  source => "message"
}
# drop the heartbeat messages
if [35] == "0" {
  drop { }
}
# And now we rename some fields to be more verbose ...
mutate {
  rename => [ "35", "MsgType" ]
}

Hope this helps,

Thorsten