Packetbeat Processors - Dropping fields from nested objects in an array

Elastic Stack Beats
1

I recently started working with Packetbeat.

For my use-case, I only need some specific fields (to the point where if I could, I would completely rewrite the mapping, but am leaving that as a last resort).

I tried removing some of the fields from the "dns.answers" array of objects, but what I did doesn't seem to have any effect:

    - include_fields:
        fields:
          - dns.question.name
          - dns.question.type
          - dns.answers
          - dns.answers_count
          - dns.resolved_ip
    - drop_fields:
        fields:
          - dns.answers.name

In addition, I also tried including only the fields I want but that didn't seem to work either, e.g:

- include_fields:
        fields:
          - dns.question.name
          - dns.question.type
          - dns.answers.data
          - dns.answers_count
          - dns.resolved_ip

Any ideas?
If rewriting the template/mapping of the index is the best choice, or perhaps using the Ingest Node Pipelines is a better approach, I'd love to hear it.

Thanks

2

Could you please share you whole configuration? Your configuration seems correct, but you might have some indentation error.

3

Sure. here is my entire processors section of the YAML file.

processors:
    - drop_event:
        when:
          not:
            equals:
              dns.response_code: "NOERROR"
    - include_fields:
        fields:
          - dns.question.name
          - dns.question.type
          - dns.answers
          - dns.answers_count
          - dns.resolved_ip
          - dns.txt_answers
    - drop_fields:
        fields:
          - dns.answers.name
    - copy_fields:
        fields:
          - from: dns.answers.data
            to: dns.txt_answers
        ignore_missing: true
        fail_on_error: false

In addition, if you see I tried copying the "dns.answers.data" field to a separate field (like what is done in the dns.resolved_ip field), and this doesn't seem to work either.
The field configuration in the mapping is:

setup.template.overwrite: true
setup.template.append_fields:
- name: dns.txt_answers
  type: text
  level: extended
4

Also, here is the error I get:
Failed to copy fields in copy_fields processor: could not fetch value for key: dns.answers.data, Error: expected map but type is common.MapStr

I think it might have to do with the following issue:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.