Anyway to force event to logstash which logs only includes line with no line terminator?

[YuWatanabe]

filebeat version : 5.2.0
logstash version : 5.2.0

Hello .

I have logs conjunction with a logthat has only one line with no line terminators.

I am using multiline to send the whole file to logstash
but only log that has single line and with no line terminator is not sent to logstash.

I am executing the filebeat with command line mode.

/usr/share/filebeat/bin/filebeat -c /opt/elasticsearch/filebeat.yml -e -once

My filebeat configuration is below.

- input_type: log
  paths:
    - /var/tmp/some_file

  multiline.pattern  : .*
  multiline.mathc    : after
  multiline.max_lines : 50000
  close_eof                   : true
  harvester_buffer_size : 32768

I have also tried setting multiline.timeout but event was not sent to logstash. I have also checked this link
but is there any way to force event from reader to multiline and to logstash?

[YuWatanabe]

I have checked the debug log and there seems to be difference in logs when the event is sent and not sent.

Tested configuration is below.

- input_type: log
  paths:
    - /var/tmp/some_file

  multiline.pattern  : .*
  multiline.mathc    : after
  multiline.max_lines : 50000
  multiline.timeout : 0
  close_eof                   : true
  harvester_buffer_size : 32768

Below is the snippet of debug log.

Result1

Read event
Microsft NCSI\n

Debug Log

2017/02/15 05:32:31.601213 prospector_log.go:254: DBG  Resuming harvesting of file: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG, offset: 0
2017/02/15 05:32:31.601477 log.go:256: DBG  Setting offset for file based on seek: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG
2017/02/15 05:32:31.601508 log.go:241: DBG  Setting offset for file: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG. Offset: 0
2017/02/15 05:32:31.601578 prospector_log.go:83: DBG  Prospector states cleaned up. Before: 3, After: 3
2017/02/15 05:32:31.601648 log.go:84: INFO Harvester started for file: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG
2017/02/15 05:32:31.601802 log.go:107: INFO End of file reached: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG. Closing because close_eof is enabled.
2017/02/15 05:32:31.601832 log.go:283: DBG  Stopping harvester for file: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG
2017/02/15 05:32:31.601857 log.go:291: DBG  Closing file: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG
2017/02/15 05:32:31.601877 log.go:262: DBG  Update state: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG, offset: 16
2017/02/15 05:32:31.601959 crawler.go:56: DBG  Prospector 0 stopped
2017/02/15 05:32:31.601986 filebeat.go:131: INFO All data collection completed. Shutting down.
2017/02/15 05:32:31.602035 crawler.go:69: INFO Stopping Crawler
2017/02/15 05:32:31.602059 crawler.go:75: INFO Stopping 1 prospectors
2017/02/15 05:32:31.602083 prospector.go:187: INFO Stopping Prospector
2017/02/15 05:32:31.602127 prospector.go:129: INFO Prospector channel stopped
2017/02/15 05:32:31.602153 crawler.go:82: INFO Crawler stopped
2017/02/15 05:32:36.600353 spooler.go:89: DBG  Flushing spooler because of timeout. Events flushed: 6
2017/02/15 05:32:36.600667 client.go:184: DBG  Publish: {
  "@timestamp": "2017-02-15T05:32:31.601Z",
  "beat": {
    "hostname": "FOR-ELA02",
    "name": "FOR-ELA02",
    "version": "5.2.0"
  },
  "input_type": "log",
  "message": "Microsoft NCSI\n",
  "offset": 16,
  "source": "/var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG",
  "type": "log"
}
2017/02/15 05:32:36.600756 output.go:109: DBG  output worker: publish 1 events

Result 2(This has no line terminator)

Read event

Microsoft NCSI

Debug Log

2017/02/15 05:33:31.620774 prospector_log.go:231: DBG  Start harvester for new file: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG
2017/02/15 05:33:31.621023 log.go:256: DBG  Setting offset for file based on seek: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG
2017/02/15 05:33:31.621057 log.go:241: DBG  Setting offset for file: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG. Offset: 0
2017/02/15 05:33:31.621117 state.go:64: DBG  New state added for /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG
2017/02/15 05:33:31.621157 prospector_log.go:83: DBG  Prospector states cleaned up. Before: 1, After: 1
2017/02/15 05:33:31.621217 log.go:84: INFO Harvester started for file: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG
2017/02/15 05:33:31.621272 log.go:107: INFO End of file reached: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG. Closing because close_eof is enabled.
2017/02/15 05:33:31.621304 log.go:283: DBG  Stopping harvester for file: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG
2017/02/15 05:33:31.621328 log.go:291: DBG  Closing file: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG
2017/02/15 05:33:31.621355 log.go:262: DBG  Update state: /var/tmp/DC170206ND9P_ED_20170209115249/HTTPFILE_74pYyG, offset: 0
2017/02/15 05:33:31.621386 crawler.go:56: DBG  Prospector 0 stopped
2017/02/15 05:33:31.621408 filebeat.go:131: INFO All data collection completed. Shutting down.
2017/02/15 05:33:31.621428 crawler.go:69: INFO Stopping Crawler
2017/02/15 05:33:31.621445 crawler.go:75: INFO Stopping 1 prospectors
2017/02/15 05:33:31.621469 prospector.go:187: INFO Stopping Prospector
2017/02/15 05:33:31.621507 prospector.go:129: INFO Prospector channel stopped
2017/02/15 05:33:31.621534 crawler.go:82: INFO Crawler stopped
2017/02/15 05:33:36.620092 spooler.go:89: DBG  Flushing spooler because of timeout. Events flushed: 2
2017/02/15 05:33:36.620156 client.go:128: DBG  No events to publish
2017/02/15 05:33:36.620208 sync.go:68: DBG  Events sent: 2

Usually, when the line terminator exists program goes to the publishing flow after spooler timeout elapses but otherwise it does not. I assume this is because actually there is no event in the spooler.

I appreciate if someone can educate me how to force events which does not include line terminator.