Help needed | Issue - panic: runtime error: invalid memory address or nil pointer dereference

[jatin03k (Jatin K)]

Filebeat is failing with above mentioned error when trying to read a standard tomcat error log (stderr.log). I tried to debug the issue by testing with subset, and found that the header might be creating the failure, but after removing the header from the actual file, the issue repeats.

Below are the various cases tested.

• stderr1.log: Log subset with 2 events & header - Fail
• stderr5.log: Only single event with no header or carriage return at the end of event - Success
• stderr6.log: 2 events (no header) with preceding and succeeding line feeds - Success
• stderr.log: Header removed from original file - Fail

The log file is being captured when I feed it to Logstash directly. Not able to attach any of the log or configuration files due to type limitations. Please look into this and suggest.

Version 5.1.1 is being used for the complete setup being run on Windows server.
Filebeat --> LogStash --> ElasticSearch --> Kibana

[andrewkroh]

What operating system?

Please provide config, logs, and the stack trace that occurs when it crashes. You can paste them into the message or post them to a gist or pastebin.

[jatin03k (Jatin K)]

Hi Andrew, thanks for reply. Please find the link to gist where required artifacts have been pasted.

@andrewkroh

What operating system?

The Windows server where ELK stack is installed and am also testing filebeat on has Windows Server 2012 R2 Standard operating system. Actual application server where the filebeat is harvesting logs from is also a Windows server.

[andrewkroh]

@jatin03k

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths: 
    - E:\ELK\IO\FilebeatInput\stderr7.log

  document_type: "TomcatErrTest"
  multiline.pattern: ['%{CATALINA_DATESTAMP} ']
  multiline.negate: true
  multiline.match: after
  include_lines: ['%{CATALINA_DATESTAMP} ']

Filebeat doesn't accept grok patterns. You must specify a single regular expression for multiline.pattern (you have a list). And you must also specify a regular expression for include_lines.

Here's a regex example that I think matches your logs. You shouldn't need the include_lines option once you have mutliline work properly.

multiline.pattern: ' ^[J|F|M|A|M|S|O|N|D][a-z]{2} [0-9]{1,2}, [0-9]{2}'

[andrewkroh]

We are also going to investigate the panic that you observed. https://github.com/elastic/beats/issues/3566