[ERROR] Total Ingest Pipeline Count/Time/Failed Metrics are all returning 0

Elastic Stack Elasticsearch
1

Hi, it seems that whenever we get the final score for ingest pipeline count/time/failed metrics all return 0 when we run esrally race.

The command we are running is:
esrally race --track=http_logs --target-hosts=http://xxx.xxx.xxx.xxx:xxxx/ --pipeline=benchmark-only --report-file=es_report_E4_32_actualgb.md --report-format=csv --kill-running-processes

The x's represent the IP address. The program finishes and returns results! But we observe a:

Total Ingest Pipeline count,,0,

Total Ingest Pipeline time,,0,s

Total Ingest Pipeline failed,,0,

In the results file. We have done this on percolator, http-logs and a custom track of our own but everytime the result returns 0 for all 3 even though we know ingestion must be occuring.

How do we fix tihs?

2

Hi @raphaelli1, thanks for using Rally :slight_smile:

These metrics are specifically related to the usage of the 'Ingest Pipeline' feature, see also the put-pipeline operation, and bulk operation's pipeline parameter. If no Ingest Pipelines are used by the indexing/bulk operations, then these stats are expected to be 0.

To answer your track specific questions;

  • percolator does not use any ingest pipelines,
  • http_logs can use Ingest Pipelines, but you must specify the track param ingest_pipeline (see README here)
    • example invocation:
    esrally race --track=http_logs --target-hosts=http://xxx.xxx.xxx.xxx:xxxx/ --pipeline=benchmark-only --challenge="append-index-only-with-ingest-pipeline"  --track-params="ingest_pipeline:baseline" --report-file=es_report_E4_32_actualgb.md --report-format=csv --kill-running-processes
  • I can't comment on your own custom track, but if you're not using any ingest pipelines then seeing these values as 0 is expected.

Thanks!

1 Like
3

Hi, I tried the command you said; however:

Total Ingest Pipeline count,,0,

Total Ingest Pipeline time,,0,s

Total Ingest Pipeline failed,,0,

I also got this error:

[WARNING] Error rate is 100.0 for operation 'index-append-with-ingest-baseline-pipeline'. Please check the logs.

[WARNING] No throughput metrics available for [index-append-with-ingest-baseline-pipeline]. Likely cause: Error rate is 100.0%. Please check the logs.

But when I saw the logs the only thing that was being printed out was:

2022-06-14 05:59:40,164 -not-actor-/PID:9105 elasticsearch WARNING POST http://xxx.xxx.xxx.xxx:xxxx/_bulk?pipeline=http-log-baseline-pipeline [status:500 request:0.553s]

This seems related to throughput metrics but ingestion metrics should still be there if only throughput metrics are not available.

4

status:500 typically indicates that the Elasticsearch server is erroring out. Given that you're already running an Elasticsearch server (--pipeline=benchmark-only), can you please check that ingest pipelines etc. are supported on that version?

If you want to extract the full error details per bulk request, you'll need do one of the following:

  • Edit the track to set the bulk operation parameter detailed: true, e.g. and check the logs
           "operation": {
         "operation-type": "bulk",
         "detailed-results": true
       }
  • Set the Elasticsearch client logger to DEBUG via editing the following section in ~/.rally/logging.json and check the logs for failed requests (this will be noisy):
    [...]
  "loggers": {
    "elasticsearch": {
      "handlers": ["rally_log_handler"],
      "level": "DEBUG",
      "propagate": false
  },
[...]
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.