Documents Dropped in Ingestion Pipeline for Okta Integration

Hello Elastic Community,

I hope you’re all doing well.

I’m reaching out to request your guidance on an issue I’m facing with an ingestion pipeline in Elastic for an Okta 2.11 integration. Some documents are being dropped during ingestion, and I’m struggling to identify the root cause despite multiple troubleshooting steps.

The issue seems related to the field okta.request.ip_chain.geographical_context.geolocation.lat. Initially, debug logs from the Elastic Agent showed the following error:

Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Meta:null, Fields:null, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}, EncodedEvent:(*elasticsearch.encodedEvent)(0xc003aa0e00)} (status=400): {"type":"illegal_argument_exception","reason":"mapper [okta.request.ip_chain.geographical_context.geolocation.lat] cannot be changed from type [float] to [long]"}, dropping event!

To address this, I modified the pipeline to either remove the problematic field or apply a convert operation. After implementing these changes, the error stopped appearing in the logs, which seemed promising at first. However, I then noticed a new issue: while a few documents are successfully indexed, there’s now a significant drop in log ingestion rates (currently at only a couple of logs per minute).

It seems like there might be another underlying issue causing incomplete ingestion or additional fields contributing to the problem.

Could you please provide recommendations on how to further debug or resolve this situation? Any advice or insights would be greatly appreciated!

Thank you so much for your time and support.

Best regards,
Felipe Matamoros