How to send two different log file types from the same source directory to logstash

[mgolubov55 (Michael Golubov)]

I have been sending two different log files to logstash from two different source directories. One type is an "Index log" that is always named "IndexService*.log", for example:

IndexService-sv_Toolkit-GBWLLSLO015-31012019-110628.293.log

The other logs are not named as consistently.

I have been using the following filebeat.yml configuration:

filebeat.inputs:

• type: log
  enabled: true
  paths:
  • C:\Logs*
    tags: ["Reference","Jobs"]
    fields:
    document_type: JobLogs
    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    multiline.negate: true
    multiline.match: after
• type: log
  enabled: true
  paths:
  • C:\LogsIndexService*
    tags: ["Reference","Indexes","Jobs"]
    fields:
    document_type: IndexServiceLogsJob

The specs on the job have just changed where both kind of logs need to be prospected from the same directory. Is there some kind of if/else logic I can apply to filter the files by filename and properly package them for logstash?

[Michal_Pristas]

hey @mgolubov55
you can use regex patterns in paths in a way to specify that IndexServiceLogs starts with index service
other logs do not start with index service.
for more capable parsing and condition you can use Logstash

[mgolubov55 (Michael Golubov)]

The following in filebeat.yml works for IndexService but not for nonIndexService. What am I doing wrong?

• type: log
  enabled: true
  paths:
  • C:\Logs\IndexService-sv_*.log
    tags: ["Reference","Indexes","Jobs"]
    fields:
    document_type: IndexServiceLogsJob
• type: log
  enabled: true
  paths:
  • C:\Logs\'^?![IndexService-sv_*.log]'
    tags: ["Reference","Jobs"]
    fields:
    document_type: JobLogs
    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    multiline.negate: true
    multiline.match: after

[mgolubov55 (Michael Golubov)]

I also tried the following - the IndexService log is read, but the other kind of log is ignored:

• type: log
  enabled: true
  paths:
  • C:\Logs\IndexService-sv_*.log
    tags: ["Reference","Indexes","Jobs"]
    fields:
    document_type: IndexServiceLogsJob
• type: log
  enabled: true
  paths:
  • C:\Logs'^(?!IndexService-sv_)*.log'
    tags: ["Reference","Jobs"]
    fields:
    document_type: JobLogs
    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    multiline.negate: true
    multiline.match: after

[martinr_ubi (Martin R.)]

Hi,

path doesn’t take a regex but a go glob pattern.
Here’s the link to an explanation of what that is:
https://golang.org/src/path/filepath/match.go?s=1226:1284#L34

[mgolubov55 (Michael Golubov)]

Thank you,
I found that out yesterday and revised my filebeat.yml accordingly and solved the issue:

• type: log
  enabled: true
  paths:
  • C:\Logs\IndexService-sv_*.log
    tags: ["Reference","Indexes","Jobs"]
    fields:
    document_type: IndexServiceLogsJob
• type: log
  enabled: true
  paths:
  • C:\Logs[^IndexService]*.log
    tags: ["Reference","Jobs"]
    fields:
    document_type: JobLogs
    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    multiline.negate: true
    multiline.match: after