Filebeat gets error "harvester:: error while connecting to output with pipeline: wrong type, expect map accessing 'processors.0.drop_event.when.contains' (source:'/etc/filebeat/prospector-clickhouse.yml')"

[silentfilm (Bruce Calvert)]

I am setting up a new prospector for my clickhouse logs, but am seeing this error below. What does it mean?:

{"log.level":"debug","@timestamp":"2024-02-20T22:46:51.999Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":177},"message":"A new file /clickhouse/logs/clickhouse-server-2024-02-20-21:53:28.err.log has been found","service.name":"filebeat","id":"clickhouse","prospector":"file_prospector","operation":"create","source_name":"native::101038678-64768","os_id":"101038678-64768","new_path":"/clickhouse/logs/clickhouse-server-2024-02-20-21:53:28.err.log","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-02-20T22:46:51.999Z","log.logger":"input.filestream","log.origin":{"file.name":"input-logfile/harvester.go","file.line":138},"message":"Starting harvester for file","service.name":"filebeat","id":"clickhouse","source_file":"filestream::clickhouse::native::101038678-64768","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-02-20T22:46:51.999Z","log.logger":"input.filestream","log.origin":{"file.name":"task/group.go","file.line":69},"message":"harvester:: error while connecting to output with pipeline: wrong type, expect map accessing 'processors.0.drop_event.when.contains' (source:'/etc/filebeat/prospector-clickhouse.yml')","service.name":"filebeat","id":"clickhouse","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-02-20T22:46:51.999Z","log.logger":"input.filestream","log.origin":{"file.name":"input-logfile/harvester.go","file.line":224},"message":"Stopped harvester for file","service.name":"filebeat","id":"clickhouse","source_file":"filestream::clickhouse::native::101038678-64768","ecs.version":"1.6.0"}

My prospector file:

[root@aln-nbadev4 filebeat]# cat prospector-clickhouse.yml
---
# Clickhouse
- type: filestream
  id: clickhouse
  enabled: true
  paths:
    - "/clickhouse/logs/clickhouse-server*.log"
  encoding: utf-8
  exclude_files:
    - '.gz$'
    - "/clickhouse/logs/clickhouse-server*.err.log"
  fields:
    application: clickhouse
    hostname: aln-nbadev4.labs.server.com
  fields_under_root: true
  scan_frequency: 5s
  multiline.pattern: '^\d{4}.\d{2}.\d{2}'
  multiline.negate: true
  multiline.match: after
  recursive_glob.enabled: true
  pipeline: clickhouse_pipeline
  processors:
    - drop_event:
        when:
          contains:
            - message: "<Debug>"
            - message: "<Trace>"
  tags: ['clickhouse']

I've got ten other prospectors and they are working just fine. I'm using Elasticsearch and Filebeat 8.8.2.

[stephenb (Stephen Brown)]

Hi @silentfilm

syntax / yml parsing error

Think you are missing the or

or perhaps this which is simpler and seems to be read / run.

The field can be a string or an array of strings. The condition accepts only a string value.

  processors:
    - drop_event:
        when:
          contains:
            message: ["<Debug>", "<Trace>"]