Keep attribute value in threatintel.misp.attribute

[parosio]

Hello,
I’m trying to modify the threatintel/misp/ingest/pipeline.yml of filebeat [7.17] to avoid the removal of threatintel.misp.attribute.value field, but to no effect so far…

The pipeline is full of rename processors that assign the value of threatintel.misp.attribute.value to other specific field (like threatintel.indicator.file.name).

I commented out the remove part of the field in the pipeline:

 # - remove:
 #     field:
 #       - threatintel.misp.attribute.value
 #     ignore_missing: true
 #     if: ctx?.threatintel?.indicator?.type != 'unknown'

but the field keeps missing from final documents.

Any suggestion on how to keep all the attributes values (regardless of their type) in a single field?

Thank you,
Paolo Arosio

[leandrojmp (Leandro Pereira)]

@parosio

I commented out the remove part of the field in the pipeline

Did you edit the ingest pipeline in Elasticsearch? I do not use Filebeat anymore, just Elastic Agent, but if I'm not wrong modules work in the same way as integration, so you would need to edit the ingest pipeline in Elasticsearch.

Changing the yaml will make no difference as the it was already loaded when running setup.

@Tortoise

{
  "set": {
    "field": "threatintel.misp.attribute.value.s",
    "value": "{{threatintel.misp.attribute.value}}"
  }
}

This would cause a mapping conflict that would drop all events, the field is a scalar and this set processor would try to transform it into an object, causing the conflict.

[parosio]

Thank you for your suggestions, @leandrojmp,

it was my first attempt with filebeat, so I kept changing the pipeline on the filebeat host with no effect (obviously).
Finally, I deleted the pipeline on elasticseach and loaded a new one that stores the initial value in a new field, and set it back at the end:

### PA 2025-09-09 save ioc value in another field (to be renamed at the end)
  - set:
      field: threatintel.misp.attribute.original_value
      value: "{{threatintel.misp.attribute.value}}"

So, that was a workaround for my problem

However, the problem I'm facing now is that MISP events with objects are not treated correctly in the input phase.

For example, a MISP event with 26 attributes (24 standard attributes and 1 object with 2 attributes) is ingested to 48 elastic documents: the single values mapped to threatintel.misp.context.attribute.value, and duplicated for each one of the attributes of the object.

Flat MISP events are handled as expected (1 elastic document for each MISP attribute belonging to the event)