How to use one Filebeat directly with ES to process multiple applications log files

[sezanawa]

Hallo Everybody

i am using filebeat directly with ES. I have installed filebeat on my Web Frontend Server where multiple applications are deployed. Each application wirte its own log. I want to use filebeat to parse the logfiles of each application and send data to ES. ES should create a index for each application depending upon fields, application name or tags. is it possible without logstash in middle ?

Here is my Filebeat.yml configuration.

filebeat.prospectors:
   - input_type: log
  paths:
- d:/logs/tmp/Catalog/*.log
include_lines: ['^Error'] 
tags: ["catalog"]
fields:
   app_id: catalog
   level: error
scan_frequency: 10s   

filebeat.prospectors:
- input_type: log
  paths:
- d:/logs/tmp/Onsurance/*.log
include_lines: ['^Error'] 
tags: ["onsurance"]
fields:
   app_id: onsurance
   level: error
scan_frequency: 5s      


#-------------------------- Multiline options------------------------------
 multiline.pattern: ^Error;.*(?:\r?\n(?!Error;|Verbose;).*)*
multiline.negate: false
multiline.match: after

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.202.170.77:9200"]
  template.enabled: true
  template.path: "filebeat.template.json"
  template.overwrite: false
  indices:
    - index: "Catalog-%{+yyyy.MM.dd}"
      when.contains:
        tags: ["catalog"]
    - index: "Onsurance-%{+yyyy.MM.dd}"
      when.contains:
        tags: ["onsurance"]

currently i have Following issues:

1. Currently it give me error on multiline.pattern. It does not accept my regular expression although regex is working in regex101.com. so i have disabled multiline.
2. It parse every line, although i have specified only to parse lines starting with Error;
3. It create a Index filebeat-*. i want to have a index for each file.

any suggestions ? how do i can achieve it.

best regards

[steffens (Steffen Siering)]

The config file is very much off, the indentation needs to be fixed in multiple places. Is this a copy'n paste error, or indeed the original config? Check out the intro to beat config file format.

Also have a look at the filebeat.full.yml, which is shipped with filebeat as reference.

The multiline setting is per prospector, as different files might have different requirements on settings

Put regular expressions in single quote '<regex>', so the YAML parser is not trying to interpret those. What pattern exactly do you want to capture? The \r and \n might not do as you expect, as filebeat splits the file into lines before presenting them to the regular expression engine.

Filebeat does only ship the raw content. For additional parsing you will have to use ingest node pipelines or logstash