# Index Speed Capped but Missing ASA logs

**URL:** <https://discuss.elastic.co/t/index-speed-capped-but-missing-asa-logs/283307>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 3, 2021, 4:23pm UTC](https://discuss.elastic.co/t/index-speed-capped-but-missing-asa-logs/283307 "2021-09-03T16:23:02Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Micah\_Barsness](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/micah_barsness/32/83318_2.png) [@Micah\_Barsness](https://discuss.elastic.co/u/Micah_Barsness)\
**Post date:** [September 3, 2021, 4:23pm UTC](https://discuss.elastic.co/t/index-speed-capped-but-missing-asa-logs/283307/1 "2021-09-03T16:23:02Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/8/b/8be9dc37268a22afc95e9fafe1f21cd56a649aeb.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/4/0484f3418ed265f3c2ab40efd8593523c920078f.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/8/78feeb577633281984d01f06fbae82e3d853c0f2.png)

Our current set up is Elasticsearch, Filebeat, and Kibana on the same server for dev purposes (hoping to move to 3 nodes & gold licensing next year) -  
We are using the cisco.yml module in filebeat to receive ASA logs and then directing filebeat to elasticsearch on the localhost.

The ASA is sending many many logs - and we've recently found that it doesnt seem filebeat/elasticsearch can keep up since there are logs missing in Kibana when searching.

We are indexing at a rate of 250/s and that should probably be a lot higher. I've tried changing the bulk\_max\_size in filebeat but to no avail.

We have a 100mb connection between the ASA & the server (not direct, but nothing less than 100m all the way there).

ILM - indexes are set to roll over at 25gb

We are still receiving about 18-20 million documents every day but there should be more.

Does anything stick out as a bottleneck or is there anything I could try to identify where these logs are being dropped?

Here is a condensed version of our Filebeat.yml file.

```auto

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  
  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: elastic
  password: ########
  bulk_max_size: 500
  workers: 6

# ============================= X-Pack Monitoring ==============================
scan_frequency: 1s

```

---

<div class="post-metadata">

**Author:** ![d-ring](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d-ring/32/78772_2.png) [@d-ring](https://discuss.elastic.co/u/d-ring)\
**Post date:** [September 3, 2021, 5:06pm UTC](https://discuss.elastic.co/t/index-speed-capped-but-missing-asa-logs/283307/2 "2021-09-03T17:06:03Z")

</div>

You are running into an issue with the timestamp parsing. It takes the majority of the time when ingesting events. Elastic has an open issue on it the last time I checked.

I ended up ingesting the ASA logs through logstash instead of filebeat to keep up with the load generated by the firewalls.

---

<div class="post-metadata">

**Author:** ![Micah\_Barsness](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/micah_barsness/32/83318_2.png) [@Micah\_Barsness](https://discuss.elastic.co/u/Micah_Barsness)\
**Post date:** [September 7, 2021, 12:12pm UTC](https://discuss.elastic.co/t/index-speed-capped-but-missing-asa-logs/283307/3 "2021-09-07T12:12:56Z")

</div>

Would you mind sharing your .conf file for logstash?

---

<div class="post-metadata">

**Author:** ![Micah\_Barsness](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/micah_barsness/32/83318_2.png) [@Micah\_Barsness](https://discuss.elastic.co/u/Micah_Barsness)\
**Post date:** [September 7, 2021, 12:15pm UTC](https://discuss.elastic.co/t/index-speed-capped-but-missing-asa-logs/283307/4 "2021-09-07T12:15:24Z")

</div>

@d-ring - also would you mind sharing the link to that issue?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 5, 2021, 2:16pm UTC](https://discuss.elastic.co/t/index-speed-capped-but-missing-asa-logs/283307/5 "2021-10-05T14:16:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.