Index Speed Capped but Missing ASA logs

[Micah_Barsness]

Our current set up is Elasticsearch, Filebeat, and Kibana on the same server for dev purposes (hoping to move to 3 nodes & gold licensing next year) -
We are using the cisco.yml module in filebeat to receive ASA logs and then directing filebeat to elasticsearch on the localhost.

The ASA is sending many many logs - and we've recently found that it doesnt seem filebeat/elasticsearch can keep up since there are logs missing in Kibana when searching.

We are indexing at a rate of 250/s and that should probably be a lot higher. I've tried changing the bulk_max_size in filebeat but to no avail.

We have a 100mb connection between the ASA & the server (not direct, but nothing less than 100m all the way there).

ILM - indexes are set to roll over at 25gb

We are still receiving about 18-20 million documents every day but there should be more.

Does anything stick out as a bottleneck or is there anything I could try to identify where these logs are being dropped?

Here is a condensed version of our Filebeat.yml file.

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  
  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: elastic
  password: ########
  bulk_max_size: 500
  workers: 6


# ============================= X-Pack Monitoring ==============================
scan_frequency: 1s

[d-ring (Todd Deering)]

You are running into an issue with the timestamp parsing. It takes the majority of the time when ingesting events. Elastic has an open issue on it the last time I checked.

I ended up ingesting the ASA logs through logstash instead of filebeat to keep up with the load generated by the firewalls.

[Micah_Barsness]

Would you mind sharing your .conf file for logstash?

[Micah_Barsness]

@d-ring - also would you mind sharing the link to that issue?