# Esitmate Logstash Requirement on Centerlized Managed

**URL:** <https://discuss.elastic.co/t/esitmate-logstash-requirement-on-centerlized-managed/377127>\
**Category:** Logstash\
**Created:** [April 15, 2025, 5:33am UTC](https://discuss.elastic.co/t/esitmate-logstash-requirement-on-centerlized-managed/377127 "2025-04-15T05:33:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![kishorkumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kishorkumar/32/132930_2.png) [@kishorkumar](https://discuss.elastic.co/u/kishorkumar)\
**Post date:** [April 15, 2025, 5:33am UTC](https://discuss.elastic.co/t/esitmate-logstash-requirement-on-centerlized-managed/377127/1 "2025-04-15T05:33:55Z")

</div>

I’m currently using centralized Logstash management from Kibana, with a single pipeline in place.  
**Logstash Version:** 8.11  
**Elasticsearch Version:** 8.11

Here’s my pipeline configuration:

```auto
input {
  elasticsearch {
    hosts => [""]
    user => ""
    password => ""
    ssl_enabled => true
    ssl_verification_mode => "none"
    index => ".ds-traces-apm-*"
    query => '{
      "_source": ["span.id", "span.name", "transaction.id"],
      "query": {
        "bool": {
          "must": [
            { "exists": { "field": "transaction.id" } },
            { "term": { "processor.event": "span" } },
            { "range": { "@timestamp": { "gte": "now-1m", "lte": "now" } } }
          ],
          "must_not": [
            { "term": { "span.action": "query" } }
          ]
        }
      }
    }'
    schedule => "* * * * *"
    docinfo => true
  }
}

filter {
  if [transaction][id] {
    elasticsearch {
      hosts => [""]
      user => ""
      password => ""
      ssl_enabled => true
      ssl_verification_mode => "none"
      index => ".ds-traces-apm-*"
      query => 'processor.event:transaction AND transaction.id:"%{[transaction][id]}"'
      fields => { "[transaction][name]" => "[transaction][name]" }
      tag_on_failure => ["_transaction_id_lookup_failure"]
    }
  }

  fingerprint {
    source => ["[span][id]", "[transaction][id]"]
    target => "uuid"
    method => "MD5"
    concatenate_sources => true
  }
}

output {
  elasticsearch {
    document_id => "%{uuid}"
    hosts => [""]
    user => ""
    password => ""
    doc_as_upsert => true
    action => update
    index => "transaction_span_enrichment"
    ssl_enabled => true
    ssl_verification_mode => "none"
  }
}

```

### Current Situation:

- Ingest rate in elasticsearch apm-traces of spans: **~300,000 per minute**
- Logstash Node Monitoring shows: **125 workers active**
- Logstash pipeline settings:
  - `pipeline.workers`: 16
  - `pipeline.batch.size`: 300000
  - `pipeline.batch.delay`: 50
  - `queue.type`: persisted
  - `queue.max_bytes`: 4 (unit not clear—MB/GB?)
  - `queue.page_capacity`: 2048 (again, unit?)

I want to match that number in logstash to elasticsearch same. means indexing into elasticsearch from kibana should be atleat 300000 per minute.

Currently there is only one pipeline but will add more for other logs as well.

Despite these settings, I'm still seeing high load and more workers than expected on the node (125).

And logs ingested by logstash in index are montly have gap of 5 minutes or more after that it ingest about maximum 75000 or 125000 for 1 minute and stops and then gap that way.

Pipeline monitoring.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/9/f90053882c5f995dfe31bb0e37f925dbeae9cc3e.png)

### What I Need:

- Guidance on how to **tune** pipeline parameters (workers, batch size, delay, queue settings).
- **Resource recommendations** (CPU cores, memory) to handle this ingestion rate efficiently.
- Any **best practices** or patterns for handling high-throughput Elasticsearch inputs + lookups in the filter phase.

#Elastic Stack > Logstash #logstash #kibana

---

<div class="post-metadata">

**Author:** ![mashhurs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mashhurs/32/102090_2.png) [@mashhurs](https://discuss.elastic.co/u/mashhurs)\
**Post date:** [April 15, 2025, 7:59pm UTC](https://discuss.elastic.co/t/esitmate-logstash-requirement-on-centerlized-managed/377127/2 "2025-04-15T19:59:32Z")

</div>

Is your issue worker number mismatch? If so, can you let me know which page are you talking about and what monitoring type (metricbeat, self or agent) you are using?  
Can you also provide `/_node/pipelines` API result?  
(Reference about API: [Get settings for pipelines | Logstash API documentation](https://www.elastic.co/docs/api/doc/logstash/operation/operation-nodeinfopipelines))

For your `queue.*` questions, please see this page for the explanation: [Persistent queues (PQ) | Elastic Documentation](https://www.elastic.co/docs/reference/logstash/persistent-queues)