Understanding time_in_millis for ingest pipeline running embeddings

Elastic Stack Elasticsearch
1

Hi,

I try to figured out the meaning of time_in_millis field about my ingest pipelines running embeddings.

time_in_millis
(integer) Total time, in milliseconds, spent preprocessing documents in the ingest pipeline.

In my case, I have a deployment for the bge-large-en-v1.5 model with 1 pipeline named embedding_pipeline for embeddings.

"trained_model_stats": [
{
"model_id": "bge-large-en-v1.5",
"model_size_stats": {
"model_size_bytes": 1336505252,
"required_native_memory_bytes": 6012339732
},
"pipeline_count": 1,
"ingest": {
"total": {
"count": 24823,
"time_in_millis": 70866438,
"current": 0,
"failed": 0
},
"pipelines": {
"embedding_pipeline": {
"count": 24823,
"time_in_millis": 70866438,
"current": 0,
"failed": 0,
"ingested_as_first_pipeline_in_bytes": 100573518,
"produced_as_first_pipeline_in_bytes": 638648261,
"processors": [ ...

I run a watcher every minute to collect statistics.

For example, this is results for the pipeline time_in_millis
22:58:00 65,352,075
22:59:00 65,671,555
The difference is 319,480 ms so this is equal to 5 minutes approx.

I don't understand this time regarding the definition. Can someone explain me this result in a interval of 1 minute ???

Regards,

Eric

2

time_in_millis is the cumulative wall clock time spent running the pipeline across all nodes. Is it possible you’re running ingest in multiple concurrent requests over that minute? It could be that one node is running 5 or more requests at once, or you have multiple ingest nodes?

3

Hello,

Yes we run this configuration with 2 ML nodes :

  "deployment_stats": {
    "deployment_id": "bge-large-en-v1.5",
    "model_id": "bge-large-en-v1.5",
    "threads_per_allocation": 4,
    "number_of_allocations": 2,
    "queue_capacity": 1024,
    "state": "started",
    "allocation_status": {
      "allocation_count": 2,
      "target_allocation_count": 2,
      "state": "fully_allocated"
    },...

The Ingest is used for embeddings performed by a python script using helpers.bulk() calls to push the text chunks in Elastic.

In my case, I am looking the best practices for monitoring and troubleshooting the ingest behavior.

It’s why I try to understand the value of time_in_millis between 2 api _stats collect.

Regards, Eric