Missing Logs on Elasticsearch Using OTEL Collector

Logs are being shipped using otelcollector to Kibana from our kubernetes cluster but there appears to be some logs missing -- Find attached error
any help will be appreciated.

error elasticsearchexporter@v0.115.0/bulkindexer.go:346 failed to index document {"kind": "exporter", "data_type": "logs", "name": "elasticsearch/acc", "index": ".ds-apc-otel-2025.01.17-000001", "error.type": "document_parsing_exception", "error.reason": ""}

Helloo!!

Hi @Fuad_Arohunfara Welcome to the community!

Couple things...

  1. that message is a little vague but probably means and invalid/ malformed document is being sent / and or perhaps a mapping conflict which is went a sent document does not match the mapping/ schema of the data stream it is being written to.

These can be a bit tough to debug.

1st what OTEL Collector are you using...

Are you up to date

Also I would try to.enable debug on the collector to see if you can more detail

Hi @stephenb thanks for your response ;
Using version: 0.113.0 on K8s
I have enabled debug and still yet not getting the reason for the exception error.

The only thing I can add is I only get this error when I tried using an ingest pipeline that I created in Kibana and added to otel config file . Without the Index I dont get the error ; The reason why I need the index is that I need to parse the body of the logs from Json ; Here is the index I created ; [ { "json": { "field": "Body", "add_to_root": true, "ignore_failure": true } }

Hi @Fuad_Arohunfara you need to share that ingest pipeline and a sample document that is failing

What is most likely happening is the ingest pipeline is modifying the document which is resulting in a mapping conflict, which then the document cannot be ingested.

We can only help you if you provide the ingest pipeline and sample documents.

Then perhaps we can help debug. Otherwise we're just guessing

[
  {
    "json": {
      "field": "Body",
      "add_to_root": true,
      "strict_json_parsing": true,
      "ignore_failure": true
    }
  }
]

An example of one of the logs from the pod:

{"@t":"2025-01-30T09:43:06.8631170Z","@m":"Request finished \"HTTP/1.1\" \"GET\" \"http\"://\"0.0.0:0000\"\"\"\"/metrics\"\"\" - 200 null \"application/openmetrics-text; version=1.0.0; charset=utf-8\" 4.452ms","@i":"74253ad3","@tr":"1750a74bc7aa38eeb4","@sp":"9e3bdb08ec1f971f","ElapsedMilliseconds":4.452,"StatusCode":200,"ContentType":"application/openmetrics-text; version=1.0.0; charset=utf-8","ContentLength":null,"Protocol":"HTTP/1.1","Method":"GET","Scheme":"http","Host":"0.0.0.0:8000","PathBase":"","Path":"/metrics","QueryString":"","EventId":{"Id":2},"SpanId":"9e08ec1f971f","TraceId":"7aa387bdcb362a8c41deeb4","ParentId":null}

Hi @Fuad_Arohunfara

Apologies not enough / correct information

Please share the entire pipeline not just a snippet.

Also, I don't know what that log is supposed to represent... There is no field namedBody please share the entire JSON of the document in from Kibana Discover from when you don't run the pipeline...

I need a complete example or I can't help.

Thanks!

test

Ok Better

but that is not the complete ingest pipeline you can run

In Kibana -> Dev Tools and share the results

GET _ingest/pipeline/pipelinename

Also interesting

  "_ignored": [
    "Body.keyword"
  ],

How / why this?

Also please share the mapping

GET .ds-we-0001/

So what you can do... to get a better response idea what is going on and post the source with the pipeline... when I did this I did not see and error with just the 1 processor... but I also do not have the mapping

POST .ds-we-0001/_doc/pipeline=pipelinename
{
    "RequestPath": "/metrics",
    "Scheme": "http",
    "ContentType": "application/openmetrics-text; version=1.0.0; charset=utf-8",
    "Attributes": {
      "log": {
        "iostream": "stdout",
        "file": {
          "path": "/var/log/pods/web-debit_debit-969468d55-8vvnl_7e477ddb-5c6c-4cb4-96bf-98daa2b9e367/debit/0.log"
        }
      },
      "time": "2025-01-30T14:27:25.89503147Z",
      "logtag": "F"
    },
    "ElapsedMilliseconds": 4.3396,
    "StatusCode": 200,
    "SourceContext": "Microsoft.AspNetCore.Hosting.Diagnostics",
    "SeverityNumber": 0,
    "ConnectionId": "0HN9HEQ7A9EML",
    "PathBase": "",
    "EventId": {
      "Id": 2
    },
    "Protocol": "HTTP/1.1",
    "Body": "{\"@t\":\"2025-01-30T14:27:25.8949237Z\",\"@m\":\"Request finished \\\"HTTP/1.1\\\" \\\"GET\\\" \\\"http\\\"://\\\"0.0.0.0:8000\\\"\\\"\\\"\\\"/metrics\\\"\\\"\\\" - 200 null \\\"application/openmetrics-text; version=1.0.0; charset=utf-8\\\" 4.3396ms\",\"@i\":\"74253ad3\",\"@tr\":\"59a4911c4762a5f0c2f96b1f8cf1bac8\",\"@sp\":\"bd1b3f34256b9e15\",\"ElapsedMilliseconds\":4.3396,\"StatusCode\":200,\"ContentType\":\"application/openmetrics-text; version=1.0.0; charset=utf-8\",\"ContentLength\":null,\"Protocol\":\"HTTP/1.1\",\"Method\":\"GET\",\"Scheme\":\"http\",\"Host\":\"0.0.0.0:8000\",\"PathBase\":\"\",\"Path\":\"/metrics\",\"QueryString\":\"\",\"EventId\":{\"Id\":2},\"SourceContext\":\"Microsoft.AspNetCore.Hosting.Diagnostics\",\"RequestId\":\"0HN9HEQ7A9EML:0001C354\",\"RequestPath\":\"/metrics\",\"ConnectionId\":\"0HN9HEQ7A9EML\",\"SpanId\":\"bd1b3f34256b9e15\",\"TraceId\":\"59a4911c4762a5f0c2f96b1f8cf1bac8\",\"ParentId\":null}",
    "@sp": "bd1b3f34256b9e15",
    "SpanId": "bd1b3f34256b9e15",
    "Path": "/metrics",
    "ParentId": null,
    "RequestId": "0HN9HEQ7A9EML:0001C354",
    "QueryString": "",
    "Resource": {
      "service": {
        "name": "test-debit"
      },
      "k8s": {
        "container": {
          "restart_count": "0",
          "name": "debit"
        },
        "cluster": {
          "name": "web-stg-euw2-1"
        },
        "node": {
          "name": "ip-10-47-142-3.eu-west-2.compute.internal"
        },
        "pod": {
          "uid": "7e477ddb-5c6c-4cb4-96bf-98daa2b9e367",
          "name": "debit-969468d55-8vvnl"
        },
        "namespace": {
          "name": "web-debit"
        }
      }
    },
    "@i": "74253ad3",
    "Host": "0.0.0.0:8000",
    "Method": "GET",
    "@m": "Request finished \"HTTP/1.1\" \"GET\" \"http\"://\"0.0.0.0:8000\"\"\"\"/metrics\"\"\" - 200 null \"application/openmetrics-text; version=1.0.0; charset=utf-8\" 4.3396ms",
    "TraceFlags": 0,
    "@timestamp": "2025-01-30T14:27:25.895031470Z",
    "Scope": {
      "name": "",
      "version": ""
    },
    "@t": "2025-01-30T14:27:25.8949237Z",
    "TraceId": "59a4911c4762a5f0c2f96b1f8cf1bac8",
    "ContentLength": null,
    "@tr": "59a4911c4762a5f0c2f96b1f8cf1bac8"
  }

Here is the INDEX mapping :

{
  ".ds-web-otel-2025.01.21-000001": {
    "mappings": {
      "_data_stream_timestamp": {
        "enabled": true
      },
      "properties": {
        "@i": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "@l": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "@m": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "@sp": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "@t": {
          "type": "date"
        },
        "@timestamp": {
          "type": "date"
        },
        "@tr": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "@x": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "Attributes": {
          "properties": {
            "log": {
              "properties": {
                "file": {
                  "properties": {
                    "path": {
                      "type": "text",
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      }
                    }
                  }
                },
                "iostream": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "logtag": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "time": {
              "type": "date"
            }
          }
        },
        "AuthenticationScheme": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "Body": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "CertificateFile": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "ConnectionId": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "ContentLength": {
          "type": "long"
        },
        "ContentRoot": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "ContentType": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "Elapsed": {
          "type": "float"
        },
        "ElapsedMilliseconds": {
          "type": "float"
        },
        "EndpointName": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "EnvName": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "EventId": {
          "properties": {
            "Id": {
              "type": "long"
            },
            "Name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "HealthCheckDescription": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "HealthCheckName": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "HealthStatus": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "Host": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "IpAddresses": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "Method": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "ParentId": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "Path": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "PathBase": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "Protocol": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "QueryString": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "Reason": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "RemoteIpAddress": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "RequestId": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "RequestMethod": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "RequestPath": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "Resource": {
          "properties": {
            "k8s": {
              "properties": {
                "cluster": {
                  "properties": {
                    "name": {
                      "type": "text",
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      }
                    }
                  }
                },
                "container": {
                  "properties": {
                    "name": {
                      "type": "text",
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      }
                    },
                    "restart_count": {
                      "type": "text",
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      }
                    }
                  }
                },
                "namespace": {
                  "properties": {
                    "name": {
                      "type": "text",
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      }
                    }
                  }
                },
                "node": {
                  "properties": {
                    "name": {
                      "type": "text",
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      }
                    }
                  }
                },
                "pod": {
                  "properties": {
                    "name": {
                      "type": "text",
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      }
                    },
                    "uid": {
                      "type": "text",
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      }
                    }
                  }
                }
              }
            },
            "service": {
              "properties": {
                "name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }
          }
        },
        "Scheme": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "Scope": {
          "properties": {
            "name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "version": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "SeverityNumber": {
          "type": "long"
        },
        "SourceContext": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "SpanId": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "StatusCode": {
          "type": "long"
        },
        "TraceFlags": {
          "type": "long"
        },
        "TraceId": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "UserAgent": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "Username": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "address": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "commandText": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "commandTimeout": {
          "type": "long"
        },
        "commandType": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "configDirectory": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "edition": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "elapsed": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "filePath": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "grantCount": {
          "type": "long"
        },
        "http": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "https": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "json_target": {
          "properties": {
            "@i": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "@l": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "@m": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "@sp": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "@t": {
              "type": "date"
            },
            "@tr": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "@x": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "AuthenticationScheme": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "ConnectionId": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "ContentLength": {
              "type": "long"
            },
            "ContentType": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "Elapsed": {
              "type": "float"
            },
            "ElapsedMilliseconds": {
              "type": "float"
            },
            "EndpointName": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "EventId": {
              "properties": {
                "Id": {
                  "type": "long"
                },
                "Name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "HealthCheckName": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "HealthStatus": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "Host": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "IpAddresses": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "LdapServerIpAddress": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "Method": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "Path": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "PathBase": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "Protocol": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "QueryString": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "Reason": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "RemoteIpAddress": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "RequestId": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "RequestMethod": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "RequestPath": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "Scheme": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "Scope": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "SourceContext": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "SpanId": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "StatusCode": {
              "type": "long"
            },
            "TraceId": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "UserAgent": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "Username": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "commandText": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "commandTimeout": {
              "type": "long"
            },
            "commandType": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "elapsed": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "grantCount": {
              "type": "long"
            },
            "newLine": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "parameters": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "licenseCompany": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "licenseExpiration": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "netversion": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "newLine": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "parameters": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "scheme": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "secretsDirectory": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "urls": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "version": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

INGEST PIPELINE MAPPING

{
  "OTEL-JSON-PROCESSOR": {
    "description": "For OTEL Web Cluster",
    "processors": [
      {
        "json": {
          "field": "Body",
          "add_to_root": true,
          "strict_json_parsing": true,
          "ignore_failure": true
        }
      }
    ]
  }
}```

Did you run my test... Try to POST the document with the PIpeline that should give the error .. if it does not then something else is going on

Update : I just ran the POST of doc with the pipeline and it seems to work..

So that may not be the issue...

I am a little confused ... these examples are on

".ds-web-otel-2025.01.21-000001 which BTW has seemed to change throughout this topic

While the original Error was on apc...

"elasticsearch/acc", "index": ".ds-apc-otel-2025.01.17-000001", "error.type": "document_parsing_exception", "error.reason": ""} 

So I am not sure if we are testing the correct items....

You said you did not get the error when not using the pipeline...

I suspect there are some documents with malformed JSON

You have

"strict_json_parsing": true,

Perhaps set that to false

Also sending the fields to root can be error prone

"add_to_root": true,

Try setting to a field like details

good luck I will check back later

Here is the result to the test ;

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "index request with op_type=index and no if_primary_term and if_seq_no set targeting backing indices is disallowed, target corresponding data stream [web-otel] instead"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "index request with op_type=index and no if_primary_term and if_seq_no set targeting backing indices is disallowed, target corresponding data stream [web-otel] instead"
  },
  "status": 400
}

Yes we are testing the same thing in the course of me trying to test some fixes the initial pod logs in the first message has changed and here is what I have on the logs

2025-01-30T16:30:44.881Z	error	elasticsearchexporter@v0.115.0/bulkindexer.go:346	failed to index document	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch/accounts", "index": ".ds-web-otel-2025.01.21-000001", "error.type": "document_parsing_exception", "error.reason": ""}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer

Lastly, you did mention changing "strict_json_parsing": true,`

to false; I actually did that yesterday and sadly that didnt work ; I dont quite get what you want me to set to details afaik "add_to_root": can only take a boolean value ?

must be

op_type=create

for a data stream... that is something completely different than your original post, but it definitely will fail to ingest the document with op_type=index into a data stream

Yes you need to set the target field ... look at the processor for that option

ALSO another BIG topic but I would hight recommend naming your datastream

logs-web-defaults

--

As you will benefits from many OOTB best Practices if you do even if not USe Elastic Agent

You should read these ... if you use the correct naming convention there are many benefits

Thanks for the explanation ; however I am struggling to understand how to set to op_type=create ; would this also be done in an ingest pipeline ? Because I dont remember setting op_type index

No typically set at the "shipping" source.

You never did mention what versions you are on.

Exactly what OTEL components are you using.?

Exactly what is your ingestion architecture?

Example

App OTEL SDK -> OTEL Collector > APM Server > Elasticsearch

Please share the OTEL Collector configure

Are you using Upstream or Elastic Distribution

OTEL CONFIG

 config:
    receivers:
      filelog:
        storage: file_storage/file_offset_storage
        multiline:
          line_start_pattern: "^[^ ^Z]+Z"
        include:
          - /var/log/pods/*/*/*.log
        exclude:
          # Exclude logs from otel collector containers. loop avoidance.
          - /var/log/pods/*/otel-collector/*.log
          - /var/log/pods/*/otc-container/*.log
        start_at: beginning
        include_file_path: true
        include_file_name: false
        # https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/filelogreceiver/README.md
        retry_on_failure:
          enabled: true
          # max_elapsed_time: 0
        operators:
          - type: regex_parser
            id: parser-containerd
            regex: "^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$"
            output: extract_metadata_from_filepath
            timestamp:
              parse_from: attributes.time
              layout_type: strptime
              layout: "%Y-%m-%dT%H:%M:%S.%fZ"
          # Extract metadata from file path
          - type: regex_parser
            id: extract_metadata_from_filepath
            regex: '^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]{36})\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$'
            parse_from: attributes["log.file.path"]
            cache:
              size: 128 # default maximum amount of Pods per Node is 110
          # Rename attributes
          - type: move
            from: attributes.log
            to: body
          - type: move
            from: attributes.stream
            to: attributes["log.iostream"]
          - type: move
            from: attributes.container_name
            to: resource["k8s.container.name"]
          - type: move
            from: attributes.namespace
            to: resource["k8s.namespace.name"]
          - type: move
            from: attributes.pod_name
            to: resource["k8s.pod.name"]
          - type: move
            from: attributes.restart_count
            to: resource["k8s.container.restart_count"]
          - type: move
            from: attributes.uid
            to: resource["k8s.pod.uid"]
      processors:
      filter/elasticsearch/accounts:
        error_mode: ignore
        logs:
          log_record:
            - resource.attributes["service.name"] != "test-accounts"
      transform/elasticsearch:
        error_mode: ignore
        log_statements:
          - context: resource
            statements:
              - set(attributes["k8s.cluster.name"], "{{ .Values.current.cluster_name }}")

    exporters:
      elasticsearch/accounts:
        endpoint: "https://stg.test.com"
        logs_index: web-otel
        pipeline: OTEL-JSON-PROCESSOR
        api_key: ${ELASTICSEARCH_API_KEY}

I do not see an apparent reason why optype is wrong... so it is somewhere in the details.... and I don't have enough details

At this point I am a little confused what configuration / state we are in and what the actual issue is.... is the optype error when you are running the collector or when you just tried to POST a document... you did not show the command that resulted in the error.

Apologies but Unfortunately I am not sure I will be able to help... unless we have very clear what was run (show that) and the result (show that) I am only getting parts of information. I think you are familiar with your configurations and test ... but I am not so I am having trouble following...

What I would do is go back to basics and defaults and get it running then perhaps I can help... but I am confused and not sure I am helping...

Perhaps With More answers I can help... if you only give me some info ... I can't help

Which collector are you using...
Upstream or EDOT?
What version of the Collector? Are you running Latest?
What version of Elasticsearch?

Assume you are looking at? Yes / No

Can you take out the following and just let it write to the default data stream and see if you get the same result?

logs_index: web-otel

Collector type : opentelemetry version 0.113.0
version of elastic : 8.15.0
not using upstream or elastic otel distribution i think

Getting the same error in default stream

error	elasticsearchexporter@v0.115.0/bulkindexer.go:346	failed to index document	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch/accounts", "index": ".ds-logs-generic-default-2025.01.31-000001", "error.type": "document_parsing_exception", "error.reason": ""}

If you take out the pipeline does it work? Yes / No

If you are using

That means Upstream, just to be clear. Which is fine...