Multiline in TCP input

[elmar.vonlanthen]

Hello

I am wondering, if it is possible to use multiline parsing in TCP input. In my tests it is not working. But in UDP input, multiline parsing is working (but not recommended).

Filebeat version: 6.5.3
OS: CentOS 7.6

Configuration file filebeat.yml:

filebeat.inputs:
  - type: tcp
    host: "localhost:9003"
    multiline:
      pattern: '^\s+'
      negate: false
      match: after
  - type: udp
    host: "localhost:9003"
    multiline:
      pattern: '^\s+'
      negate: false
      match: after

output.console:
  pretty: true

Test commands:

TCP:

$ echo -e "hello\n  world" | nc 127.0.0.1 9003

UDP:

$ echo -e "hello\n  world" | nc -u 127.0.0.1 9003

Filebeat console output:

{
  "@timestamp": "2018-12-18T08:07:40.763Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.5.3"
  },
  "message": "hello",
  "source": "127.0.0.1:54206",
  "input": {
    "type": "tcp"
  },
  "prospector": {
    "type": "tcp"
  },
  "beat": {
    "name": "atlantis.vagrant.local",
    "hostname": "atlantis.vagrant.local",
    "version": "6.5.3"
  },
  "host": {
    "name": "atlantis.vagrant.local"
  }
}
{
  "@timestamp": "2018-12-18T08:07:40.764Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.5.3"
  },
  "message": "  world",
  "source": "127.0.0.1:54206",
  "input": {
    "type": "tcp"
  },
  "prospector": {
    "type": "tcp"
  },
  "beat": {
    "name": "atlantis.vagrant.local",
    "hostname": "atlantis.vagrant.local",
    "version": "6.5.3"
  },
  "host": {
    "name": "atlantis.vagrant.local"
  }
}
{
  "@timestamp": "2018-12-18T08:07:45.824Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.5.3",
    "truncated": false
  },
  "beat": {
    "name": "atlantis.vagrant.local",
    "hostname": "atlantis.vagrant.local",
    "version": "6.5.3"
  },
  "host": {
    "name": "atlantis.vagrant.local"
  },
  "message": "hello\n  world\n",
  "source": "127.0.0.1:41991",
  "prospector": {
    "type": "udp"
  },
  "input": {
    "type": "udp"
  }
}

Thanks for clarification.

Kind regards
Elmar

[pierhugues (Pier-Hugues Pellerin)]

Hello @elmar.vonlanthen

Both the UDP and the TCP are not supporting the multiline option that you are defining. The option is just simply ignored. Beats don't throw errors when it doesn't know the options.

From your example, it appears that the UDP support multiline but the behavior is actually that the input is reading the full datagram it received. It's because in the UDP input we do not track any clients we just act on the received datagram, to have multiline working we would need to keep track of where the events are coming from.

In the case of TCP, I see the need to support multiline and the state could be keep at the connected client level.

[elmar.vonlanthen]

Hello @pierhugues

Thanks for the explanations. Is there a chance, that multiline support for TCP will be implemented soon?

Kind regards
Elmar