Unable to use drop_event to filter json

[cabecada (Vijaykumar Jain)]

i am unable to filter lines (json) based on json key value.

wrt https://www.elastic.co/guide/en/beats/filebeat/current/configuration-processors.html

i have the below file content that grows.

    root@ubuntu1404-4:/etc/filebeat# cat /tmp/baz
    {"host": "test", "timestamp": "", "module": "libbeat", "message": "hello json world"}
    {"host": "test", "timestamp": "", "module": "topbeat", "message": "hello topbeat"}
    {"host": "test", "timestamp": "", "module": "filebeat", "message": "hello filebeat"}
    {"host": "test", "timestamp": "", "module": "libbeat", "message": "hello json world"}
    {"host": "test", "timestamp": "", "module": "topbeat", "message": "hello topbeat"}
    {"host": "test", "timestamp": "", "module": "filebeat", "message": "hello filebeat"}
    {"host": "test", "timestamp": "", "module": "libbeat", "message": "hello json world"}
    {"host": "test", "timestamp": "", "module": "topbeat", "message": "hello topbeat"}
    {"host": "test", "timestamp": "", "module": "filebeat", "message": "hello filebeat"}
    {"host": "test", "timestamp": "", "module": "libbeat", "message": "hello json world"}
    {"host": "test", "timestamp": "", "module": "topbeat", "message": "hello topbeat"}
    {"host": "test", "timestamp": "", "module": "filebeat", "message": "hello filebeat"}
    {"host": "test", "timestamp": "", "module": "libbeat", "message": "hello json world"}
    {"host": "test", "timestamp": "", "module": "topbeat", "message": "hello topbeat"}
    {"host": "test", "timestamp": "", "module": "filebeat", "message": "hello filebeat"}
    {"host": "test", "timestamp": "", "module": "libbeat", "message": "hello json world"}
    {"host": "test", "timestamp": "", "module": "topbeat", "message": "hello topbeat"}
    {"host": "test", "timestamp": "", "module": "filebeat", "message": "hello filebeat"}
    {"host": "test", "timestamp": "", "module": "libbeat", "message": "hello json world"}
    {"host": "test", "timestamp": "", "module": "topbeat", "message": "hello topbeat"}
    {"host": "test", "timestamp": "", "module": "filebeat", "message": "hello filebeat"}
    {"host": "test", "timestamp": "", "module": "libbeat", "message": "hello json world"}
    {"host": "test", "timestamp": "", "module": "topbeat", "message": "hello topbeat"}
    {"host": "test", "timestamp": "", "module": "filebeat", "message": "hello filebeat"}
    {"host": "test", "timestamp": "", "module": "libbeat", "message": "hello json world"}
    {"host": "test", "timestamp": "", "module": "topbeat", "message": "hello topbeat"}
    {"host": "test", "timestamp": "", "module": "filebeat", "message": "hello filebeat"}
    {"host": "test", "timestamp": "", "module": "libbeat", "message": "hello json world"}
    {"host": "test", "timestamp": "", "module": "topbeat", "message": "hello topbeat"}

i want to drop all the lines that has module: filebeat using drop_event

    root@ubuntu1404-4:/etc/filebeat# cat conf.d/growth-sensu-events.yml
    filebeat:
      prospectors:
        - paths: ['/tmp/baz']
          input_type: log
          document_type: json
          json.message_key: module
          json.keys_under_root: true
          json.add_error_key: true
     processors:
       - drop_event:
           when:
              equals:
                 module: "libbeat"

    root@ubuntu1404-4:/etc/filebeat# cat filebeat.yml
    ### Filebeat configuration managed by Puppet ###
    ---
      filebeat:
        spool_size: 1024
        idle_timeout: "5s"
        registry_file: ".filebeat"
        config_dir: /etc/filebeat/conf.d
      output.file:
        enabled: true
        path: "/tmp/filebeat"
        rotate_every_kb: 10000
        number_of_files: 7
      shipper: {}
      logging:
        to_files: true
        level: debug
        files:
          rotateeverybytes: 10485760
          keepfiles: 5
          path: /var/log/beats
          name: filebeat.log
      runoptions: {}

    root@ubuntu1404-4:/etc/filebeat# dpkg -l | grep fileb
    ii  filebeat                            5.1.2

i still see json with modue: libeat

    root@ubuntu1404-4:/etc/filebeat# tail -f /tmp/filebeat/filebeat
    {"@timestamp":"2017-01-18T19:16:49.407Z","beat":{"hostname":"ubuntu1404-4","name":"ubuntu1404-4","version":"5.1.2"},"host":"test","input_type":"log","message":"hello filebeat","module":"filebeat","offset":2032,"source":"/tmp/baz","timestamp":"","type":"json"}
    {"@timestamp":"2017-01-18T19:16:50.417Z","beat":{"hostname":"ubuntu1404-4","name":"ubuntu1404-4","version":"5.1.2"},"host":"test","input_type":"log","message":"hello json world","module":"libbeat","offset":2118,"source":"/tmp/baz","timestamp":"","type":"json"}
    {"@timestamp":"2017-01-18T19:16:50.417Z","beat":{"hostname":"ubuntu1404-4","name":"ubuntu1404-4","version":"5.1.2"},"host":"test","input_type":"log","message":"hello topbeat","module":"topbeat","offset":2201,"source":"/tmp/baz","timestamp":"","type":"json"}
    {"@timestamp":"2017-01-18T19:16:50.417Z","beat":{"hostname":"ubuntu1404-4","name":"ubuntu1404-4","version":"5.1.2"},"host":"test","input_type":"log","message":"hello filebeat","module":"filebeat","offset":2286,"source":"/tmp/baz","timestamp":"","type":"json"}
    {"@timestamp":"2017-01-18T19:16:51.428Z","beat":{"hostname":"ubuntu1404-4","name":"ubuntu1404-4","version":"5.1.2"},"host":"test","input_type":"log","message":"hello json world","module":"libbeat","offset":2372,"source":"/tmp/baz","timestamp":"","type":"json"}
    {"@timestamp":"2017-01-18T19:16:51.428Z","beat":{"hostname":"ubuntu1404-4","name":"ubuntu1404-4","version":"5.1.2"},"host":"test","input_type":"log","message":"hello topbeat","module":"topbeat","offset":2455,"source":"/tmp/baz","timestamp":"","type":"json"}

not sure but i am not able to get a single example of how a complete conf file of prospectors, processors etc would look like.

Any help or guidance would be appreciated.

[andrewkroh]

Can you share your configuration file? What version are you using?

[cabecada (Vijaykumar Jain)]

I have the config file in the content pasted.
Filebeat version is 5.1.2

Thanks
Vijay

[andrewkroh]

Here's an example. input.json contains what you pasted from /tmp/baz.

$ cat filebeat.dev.yml 
filebeat.config_dir: ./conf.d

logging.level: debug

output.file:
  path: .
  filename: fb.out.json

processors:
- drop_event:
    when:
      equals:
        module: "libbeat"

$ cat conf.d/someapp.yml 
filebeat.prospectors:
- paths:
    - input.json
  document_type: json
  json.message_key: message
  json.keys_under_root: true
  json.add_error_key: true

$ ./filebeat -version
filebeat version 5.1.2 (amd64), libbeat 5.1.2
$ ./filebeat -c filebeat.dev.yml -e -d "*"
^C
$ grep libbeat fb.out.json
$ wc -l input.json 
      29 input.json
$ wc -l fb.out.json
      19 fb.out.json

[cabecada (Vijaykumar Jain)]

made my day. Thank You.
i will create a test repo with examples on my understanding (so that it is helpful to others like me) and probably try to improve on formatting the question