Multiline work strange

Elastic Stack Logstash
1

Good day!

I created a theme about udp and multiline. After that I switched to tсp.
After watching for a while I found strange things, they end if I disable multiline.
config is:

input {
  tcp {
    port => 5200
    codec => multiline {
       pattern => "^((gwmd|mddump|trade)[a-zA-Z0-9-]+.example.net) (DEBUG|INFO|ERROR|WARNING) (.+)"
      negate => true
      what => "previous"
    }
  }
}

filter {
    dissect {
      mapping => { "message" => "%{host} %{levelname} %{name} %{asctime} %{+asctime} %{module} %{process} %{thread} %{message}"}
    }
    date {
      match => [ "asctime", "YYYY-MM-dd HH:mm:ss,SSS" ]
      timezone => "Etc/UTC"
    }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

strange things are:

word "process:" in "message" looks like

process:
proc ess:
p rocess:
pro cess:
proce ss:
pr ocess:
proces s:

also host, but for host i apply additional regexp.
But even with this additional regexp, i have strange message as this:

"message": "Response:[17236, 7422.9, 1, 2]\ngwmd-btc-usd.\nexample.net DEBUG mdengines_Bitfinex_BTC_USD 2018-02-06 22:40:56,001 bitfinex 5151 140172730033984 asks = [], bids = [[7422.9, 2]], updates = [[7422.9, 1, 2]]",
"tags": [
  "multiline"
],

new string symbol in the middle of host name cause multiline for valid string and it be invalid after this, and glue to the previous line

Is there some recommendations about solve this stranges?

logs send to logstash from hosts by syslog-ng with very simple configuration

destination d_remote_server {
   tcp("kibana.example.net" port(5200) template("$LOGHOST $MSG\n"));
};
2

this is solve problem, but requires a lot of resources

filter {
    dissect {
      mapping => { "message" => "%{host} %{levelname} %{name} %{asctime} %{+asctime} %{module} %{process} %{thread} %{message}"}
    }
  mutate {
    gsub => [
      "host", "\n", "",
      "levelname", "\n", "",
      "name", "\n", "",
      "asctime", "\n", "",
      "module", "\n", "",
      "process", "\n", "",
      "thread", "\n", "",
      "message", "\n", ""
    ]
  }
    date {
      match => [ "asctime", "YYYY-MM-dd HH:mm:ss,SSS" ] #2018-01-29 22:16:59,537
      timezone => "Etc/UTC"
    }
}
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.