How to send two different log file types from the same source directory to logstash

Elastic Stack Beats
1

I have been sending two different log files to logstash from two different source directories. One type is an "Index log" that is always named "IndexService*.log", for example:

IndexService-sv_Toolkit-GBWLLSLO015-31012019-110628.293.log

The other logs are not named as consistently.

I have been using the following filebeat.yml configuration:

filebeat.inputs:

  • type: log
    enabled: true
    paths:
    • C:\Logs*
      tags: ["Reference","Jobs"]
      fields:
      document_type: JobLogs
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
  • type: log
    enabled: true
    paths:
    • C:\LogsIndexService*
      tags: ["Reference","Indexes","Jobs"]
      fields:
      document_type: IndexServiceLogsJob

The specs on the job have just changed where both kind of logs need to be prospected from the same directory. Is there some kind of if/else logic I can apply to filter the files by filename and properly package them for logstash?

2

hey @mgolubov55
you can use regex patterns in paths in a way to specify that IndexServiceLogs starts with index service
other logs do not start with index service.
for more capable parsing and condition you can use Logstash

3

The following in filebeat.yml works for IndexService but not for nonIndexService. What am I doing wrong?

  • type: log
    enabled: true
    paths:
    • C:\Logs\IndexService-sv_*.log
      tags: ["Reference","Indexes","Jobs"]
      fields:
      document_type: IndexServiceLogsJob
  • type: log
    enabled: true
    paths:
    • C:\Logs\'^?![IndexService-sv_*.log]'
      tags: ["Reference","Jobs"]
      fields:
      document_type: JobLogs
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
4

I also tried the following - the IndexService log is read, but the other kind of log is ignored:

  • type: log
    enabled: true
    paths:
    • C:\Logs\IndexService-sv_*.log
      tags: ["Reference","Indexes","Jobs"]
      fields:
      document_type: IndexServiceLogsJob
  • type: log
    enabled: true
    paths:
    • C:\Logs'^(?!IndexService-sv_)*.log'
      tags: ["Reference","Jobs"]
      fields:
      document_type: JobLogs
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
5

Hi,

path doesn’t take a regex but a go glob pattern.
Here’s the link to an explanation of what that is:
https://golang.org/src/path/filepath/match.go?s=1226:1284#L34

6

Thank you,
I found that out yesterday and revised my filebeat.yml accordingly and solved the issue:

  • type: log
    enabled: true
    paths:
    • C:\Logs\IndexService-sv_*.log
      tags: ["Reference","Indexes","Jobs"]
      fields:
      document_type: IndexServiceLogsJob
  • type: log
    enabled: true
    paths:
    • C:\Logs[^IndexService]*.log
      tags: ["Reference","Jobs"]
      fields:
      document_type: JobLogs
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.