Filebeat processors not working as expected

Elastic Stack Beats
1

Here is my filebeat.yml file.

First Issue : It was working fine with single processor when I was not testing the and condition, as soon as I added the and condition. I am getting error.

$ ./filebeat
Exiting: error loading config file: yaml: line 31: found character that cannot start any token

$ egrep -v "^(#|  #|$)" filebeat.yml
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /Users/viaggarw/Documents/ELK/exercise_1.log
  processors:
    - drop_event:
        when:                <<<<<< Line 31
	  and:
            - contains:
                message: "INFO"
            - contains:
                message: "start"
    #- drop_fields:
    #    fields: ["offset"]
    #    when:
    #- c:\programdata\elasticsearch\logs\*
  exclude_files: ['.gz$']
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.console:
  pretty: true

my log file is :slight_smile:

$ cat exercise_1.log
[DEBUG] 2018-12-07 03:57:27.064 [https-jsse-nio-8443-exec-10] RequestProcessor - Status >200
[INFO ] 2018-12-07 04:00:41.015 [main] Application - Starting Application v0.0.1-SNAPSHOT on c62e0ddde6f5 with PID 141 (/etc/app1/bin/app1.jar started by app1 in /etc/app1/bin)
[DEBUG] 2018-12-07 04:00:41.022 [main] Application - Running with Spring Boot v1.5.2.RELEASE, Spring v4.3.7.RELEASE
[INFO ] 2018-12-07 04:00:41.023 [main] Application - No active profile set, falling back to default profiles: default
[INFO ] 2018-12-07 04:00:44.265 [main] Application - Started Application in 3.824 seconds (JVM running for 4.755)
[DEBUG] 2018-12-07 18:31:09.868 [https-jsse-nio-8443-exec-4] RestProcessor - https://127.0.0.1:8200/v1/auth/ldap/login/user1
[DEBUG] 2018-12-07 18:31:10.241 [https-jsse-nio-8443-exec-4] RequestProcessor - Status >200

I know we may follow the other ways to get the end results but I am just trying to understand the working of processors.

Second Issue : When using the following filebeat.yml file.

$ egrep -v "^(#|  #|$)" filebeat.yml  | grep -v "#"
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /Users/viaggarw/Documents/ELK/exercise_1.log
  processors:
    - drop_event:
        when:
          contains:
            message: "INFO"
    - drop_fields:
        fields: ["offset"]
        when:                      <<<<< Line no 36
	  contains:
	    message: "DEBUG"
  exclude_files: ['.gz$']
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.console:
  pretty: true

Found this issue:

$ ./filebeat
Exiting: error loading config file: yaml: line 36: found character that cannot start any token

I am not sure what I am doing wrong. Can anyone please help to identify the issue?

2

Both issues are caused by bad indentation. You are mixing spaces and tabs. It might look well indented in your editor but as you can see when pasted here it looks wrong.

It's recommended to use only use spaces with yaml.

3

Thanks. It helps to make progress.

$ egrep -v "^(#|  #|$)" filebeat.yml  | grep -v "#"
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /Users/viaggarw/Documents/ELK/exercise_1.log
  processors:
    - drop_event:
      when:
        contains:
          message: "INFO"
    - drop_fields:
      fields: ["offset"]
      when:
        contains:
          message: "DEBUG"
  exclude_files: ['.gz$']
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.console:
  pretty: true

Verified that yaml is having proper indentation.

$ python -c 'import yaml,sys;yaml.safe_load(sys.stdin)' < filebeat.yml
$

It's giving me this error now.

$ ./filebeat
Exiting: Error in initing prospector: each processor needs to have exactly one action, but found 2 actions

As per this link it should work.

4

Hi,

Your config was still not OK according to the link you provided, the difference is subtle but important.

You need to add an extra level of indent to the contents of - drop_event: and - drop_fields, like this:

processors:
    - drop_event:
        when:
          contains:
            message: "INFO"
    - drop_fields:
        fields: ["offset"]
        when:
          contains:
            message: "DEBUG"
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.