Filebeat not sending single line include_lines until multiline is added to the configuration, then removed and service is restarted

[mattsdevop]

I am facing what appears to be a bug, but wanted to bring it here first in order to confirm this before filing a bug report on github.

Overview: I have a filebeat.yml that includes multiline and single line log type sections. About 22 sections in total. 21 of them have multiline.pattern, multiline.negate, and multiline.match similar to the following:

- type: log
  enabled: true
  paths:
    -  ${filebeat.logpath}\SomeService-[0-9]*
  include_lines: ['.*ERR', '.*WARN','.* took.*ms to start']
  exclude_lines: ["some uninteresting message"]  
  fields:
    env: *env
    product: *product
    customer: *customer
    service: "Some"
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

The single section without the multiline is like the following:

- type: log
  enabled: true
  paths:
    - ${filebeat.differentlogpath}\another_service_log*.txt
  include_lines: ["some 1 line message that we are looking for"]
  fields:
    env: *env
    product: *product
    customer: *customer
    service: "another-service"

Description of behavior: Filebeat does not send any logs for matches on the single line section. It apparently doesn't think that it matches at all, since there are no logs generated for it. It appears all is well based on the logging.

Replication details: In order to replicate, save the filebeat.yml file with multiline and single line configuration. Restart the filebeat service on the server. Watch the multiline matches be sent, but no single line matches are sent.

Logs: none are produced related to any warnings or errors. All appears well!

Workaround: In order to get this configuration to work, I have to go in to the filebeat.yml and add the 3 multiline statements to my single line section, save the filebeat.yml, restart the filebeat service on the server. After doing so, since the multiline configuration is obviously not correct, I get incorrect log lines sent to logstash. I must go back in to the configuration file, remove the exact same 3 multiline statements that I just added to the single line section, then restart the filebeat service again. Once these steps are completed, I begin receiving the include_lines for the single line section as expected.

As an aside, this obviously isn't a configuration issue as it ultimately sends the correct include_lines after breaking it (by adding the multiline sections), then fixing it and restarting the service. I just wanted to ensure I'm not missing something else here. It appears that filebeat isn't parsing my single line include_lines section until I break it, then "unbreak" it.

[steffens (Steffen Siering)]

I don't see why breaking and repairing your configuration file should fix something in Filebeat. Filebeat is restarted. There is no particular state safed between restarts/reconfigurations which will affect the include_lines setting.

Any messages/errors in your logs? Can you try running filebeat in debug mode and share logs?

Can you check the registry file (it's a JSON file) between each of these steps? The registry file stores the file offset where to read next from if filebeat is restarted. I wonder if the offset get's stuck if include_lines is set, and get's unstuck when the multiline setting is applied. Note, the offset is only changed if events are to be send. If all events are filtered out due to include_lines, then no event will ever be send.

[mattsdevop]

Thank you for the reply.
Haven't had an opportunity to perform your suggested steps. Currently, everything was already configured as I stated in my initial post on production environments prior to posting here, and is working. I am not risking breaking anything in production for something that's currently working in order to gather log details and debug information. Hopefully, as I roll filebeat out to more environments, I can find an opportunity to gather this information.