# Elasticsearch not parsing cloudtrail index

**URL:** <https://discuss.elastic.co/t/elasticsearch-not-parsing-cloudtrail-index/283913>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 10, 2021, 7:17pm UTC](https://discuss.elastic.co/t/elasticsearch-not-parsing-cloudtrail-index/283913 "2021-09-10T19:17:23Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![EvanGertis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evangertis/32/85191_2.png) [@EvanGertis](https://discuss.elastic.co/u/EvanGertis)\
**Post date:** [September 10, 2021, 7:17pm UTC](https://discuss.elastic.co/t/elasticsearch-not-parsing-cloudtrail-index/283913/1 "2021-09-10T19:17:23Z")

</div>

I'm running into an issue parsing cloudtrail logs from filebeat. I am getting this error from the the ecs logs.

```auto
WARN [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x12db7cb9, ext:63766894617, loc:(*time.Location)(nil)}, Meta:{"_id":"2d2e682040-000000106410","pipeline":"filebeat-7.11.2-aws-cloudtrail-pipeline"}

```

I've recently enabled organization level logging. It appears as though this has not been working for sometime now. Any help with this issue would be greatly appreciated. It says

```auto
 {"type":"illegal_argument_exception","reason":"Limit of total fields [1000] has been exceeded"}

```

At the bottom of the error, as well, but I've updated the limit to 2000.

Is there a way to modify the filebeat.yml so that the indicies can be configured like so?

```auto
indices:
  - index: "cloudtrail-%{[agent.version]}-%{+yyyy.MM.dd}"
    mapping.total_fields.limit: 2000
    when.contains:
      fileset.name: "cloudtrail"
  - index: "elb-%{[agent.version]}-%{+yyyy.MM.dd}"

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 13, 2021, 6:06am UTC](https://discuss.elastic.co/t/elasticsearch-not-parsing-cloudtrail-index/283913/2 "2021-09-13T06:06:27Z")

</div>

> [@EvanGertis](#):
>
> Is there a way to modify the filebeat.yml so that the indicies can be configured like so?

You'd need to configure a custom template to handle that - [Configure Elasticsearch index template loading | Filebeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-template.html)

---

<div class="post-metadata">

**Author:** ![EvanGertis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evangertis/32/85191_2.png) [@EvanGertis](https://discuss.elastic.co/u/EvanGertis)\
**Post date:** [September 13, 2021, 1:47pm UTC](https://discuss.elastic.co/t/elasticsearch-not-parsing-cloudtrail-index/283913/3 "2021-09-13T13:47:24Z")

</div>

The template is configured like so

```auto
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana.host: "myinstance:9243"
setup.kibana.protocol: "https"
setup.kibana.username: "user"
setup.kibana.password: ${password}
setup.dashboards.enabled: true
setup.dashboards.directory: ${path.config}/kibana
setup.ilm.enabled: false
logging.level: info
output.elasticsearch:
  hosts: ["myinstance:9243"]
  protocol: "https"
  username: "user"
  password: ${pass}
  bulk_max_size: 3200
  worker: 16
  indices:
  - index: "cloudtrail-%{[agent.version]}-%{+yyyy.MM.dd}"
    mapping.total_fields.limit: 2000
    when.contains:
      fileset.name: "cloudtrail"
  - index: "elb-%{[agent.version]}-%{+yyyy.MM.dd}"
    when.contains:
      fileset.name: "elb"
  - index: "vpc-%{[agent.version]}-%{+yyyy.MM.dd}"
    when.contains:
      fileset.name: "vpcflow"
processors:
  - add_fields:
      target: mytarget
      fields:
        environment: ${env}

```

---

<div class="post-metadata">

**Author:** ![EvanGertis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evangertis/32/85191_2.png) [@EvanGertis](https://discuss.elastic.co/u/EvanGertis)\
**Post date:** [September 13, 2021, 3:07pm UTC](https://discuss.elastic.co/t/elasticsearch-not-parsing-cloudtrail-index/283913/4 "2021-09-13T15:07:36Z")

</div>

I figured it out. I deleted an index mapping template that was causing issues.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 11, 2021, 5:08pm UTC](https://discuss.elastic.co/t/elasticsearch-not-parsing-cloudtrail-index/283913/5 "2021-10-11T17:08:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.