Error Ingesting AWS Security Hub Data

Elastic Stack Elasticsearch
1

Hello Team,

I am trying to ingest data from AWS Security Hub into elastic using elastic agent and the integration available, however i am getting the below error. Could someone guide me how i can resolve this issue.

Error - {"type":"document_parsing_exception","reason":"[1:3076] failed to parse field [event.kind] of type [constant_keyword] in document with id 'kNeAIUiexKx0fJ+VL2Z28AVXwIo='. Preview of field's value: 'pipeline_error'","caused_by":{"type":"illegal_argument_exception","reason":"[constant_keyword] field [event.kind] only accepts values that are equal to the value defined in the mappings [state], but got [pipeline_error]"}}, dropping event!","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.logger":"elasticsearch","log.origin":{"file.line":517,"file.name":"elasticsearch/client.go","function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).applyItemStatus"},"service.name":"filebeat","log.type":"event","ecs.version":"1.6.0","ecs.version":"1.6.0"}

2

Are you getting some documents but not all documents?

Or are you getting no documents?

Can you share:

  1. Elastic stack version
  2. Elastic Agent version
  3. Version of the AWS security hub integration
  4. Configuration of the AWS security hub integration
  5. Are you pulling security hub findings or security hub insights

FYI: There was a recent update to the integration that may that fixed your issue https://github.com/elastic/integrations/pull/11158 which will only be visible if you're running kibana 8.16.0 or newer

3

Are you getting some documents but not all documents?
Ans) Getting around 80% documents but for the rest getting failure messages.

  1. Elastic stack version -> 8.16.1
  2. Elastic Agent version -> 8.16.4
  3. Version of the AWS security hub integration -> 2.37.0
  4. Configuration of the AWS security hub integration -> Default configuration with the below settings
    Interval - 1h , Initial Interval - 24h, region - us-east-1
  5. Are you pulling security hub findings or security hub insights -> security hub findings
4

I believe I have identified the issue and have posted a bug here: [AWS Security Hub]: Event.kind should not be a constant keyword · Issue #12970 · elastic/integrations · GitHub it requires a little discussion with the team about the best approach to solving it but you can track progress at that issue

5

Thank you for help with this.