Elasticsearch not parsing cloudtrail index

Elastic Stack Beats
1

I'm running into an issue parsing cloudtrail logs from filebeat. I am getting this error from the the ecs logs.

WARN [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x12db7cb9, ext:63766894617, loc:(*time.Location)(nil)}, Meta:{"_id":"2d2e682040-000000106410","pipeline":"filebeat-7.11.2-aws-cloudtrail-pipeline"}

I've recently enabled organization level logging. It appears as though this has not been working for sometime now. Any help with this issue would be greatly appreciated. It says

 {"type":"illegal_argument_exception","reason":"Limit of total fields [1000] has been exceeded"}

At the bottom of the error, as well, but I've updated the limit to 2000.

Is there a way to modify the filebeat.yml so that the indicies can be configured like so?

indices:
  - index: "cloudtrail-%{[agent.version]}-%{+yyyy.MM.dd}"
    mapping.total_fields.limit: 2000
    when.contains:
      fileset.name: "cloudtrail"
  - index: "elb-%{[agent.version]}-%{+yyyy.MM.dd}"
2

You'd need to configure a custom template to handle that - Configure Elasticsearch index template loading | Filebeat Reference [8.11] | Elastic

3

The template is configured like so

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana.host: "myinstance:9243"
setup.kibana.protocol: "https"
setup.kibana.username: "user"
setup.kibana.password: ${password}
setup.dashboards.enabled: true
setup.dashboards.directory: ${path.config}/kibana
setup.ilm.enabled: false
logging.level: info
output.elasticsearch:
  hosts: ["myinstance:9243"]
  protocol: "https"
  username: "user"
  password: ${pass}
  bulk_max_size: 3200
  worker: 16
  indices:
  - index: "cloudtrail-%{[agent.version]}-%{+yyyy.MM.dd}"
    mapping.total_fields.limit: 2000
    when.contains:
      fileset.name: "cloudtrail"
  - index: "elb-%{[agent.version]}-%{+yyyy.MM.dd}"
    when.contains:
      fileset.name: "elb"
  - index: "vpc-%{[agent.version]}-%{+yyyy.MM.dd}"
    when.contains:
      fileset.name: "vpcflow"
processors:
  - add_fields:
      target: mytarget
      fields:
        environment: ${env}
4

I figured it out. I deleted an index mapping template that was causing issues.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.