Error in logstash-plain-logs in /applog/logstash

Elastic Stack Logstash
1

Hello All,

We are getting below error in the logstash-plain-logs in /applog/logstash

Below are the complete logs.

[logstash.outputs.opensearch] [main] [2cd0b680af9b49acc4d5ae414a3665aabb9aa76a58d0430b62230cc5c2e971bc] Could not index event to
OpenSearch. (:status=>400, :action=>["index", (:_id=>nil,:_index=>"mulesoft-logs-2024.10.04", :routing=>nil),
("contextMap"=>{"correlationId"=>"50e4b0092b4d", "processorPath"=>"proxy-api/processors/1"), "loggerFqcn"=>"org.apache.logging.slf4j.Log4jLogger", "@version"=>"1", "threadId"=>26387, "loggerData">("priority"=>"DEBUG", "correlationId"=>"48bc2bc1-7c9f-4407-a9e7-50e4b0092b4d", "category"=>"logging", "content"=> ("payload"=>"{\"tranid\": \"**********\",\"aggregatorVPA\": \"**********\",\"customerId\": \"**********\")", "apiName"=>"/demo","httpStatus"=>0),"threadName"=>" (MuleRuntime].uber.1387: [proxy-api].proxy-api.Bl BLOCKING @6592d508",
"timestamp"=>"2024-10-04T14:00:14.6552", "environment"=>"uat", "tracePoint"=>"START", "elapsed"=>1, applicationVersion"=>"1.0.0-SNAPSHOT", "applicationName"=>"proxy-api",
" "message"=>"Start: proxy-api-payload", "locationInfo"=>{"fileName"=>"proxy-api.xml", "rootContainer"=>"proxy-api", "lineInFile"=>"34", "component"=>"json-logger: logger"}},
"appName"=>"proxy-api", "loggerName"=>"logging", "event"=>(), "apiName"=>"proxy-api",
threadPriority"=>5 " , "endOfBatch"=>true, "thread"=>"[MuleRuntime].uber.1387: [dreamll-olive-proxy-exp-api].dreamll-olive-proxy.BLOCKING @6592d508", "level"=>"DEBUG", "timeMillis"=>1728050414655, "@timestamp"=>2024-10-04T14:00:14.6552)), response=>("index"=>("_index"=>"mulesoft-logs-2024.10.04","type"=>" doc",
"_id"=>"C2ray472", "status"=>400, "error"=>("type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [loggerData.content] of type [text] in document with id 'C2ray47Z'. Preview of field's value: '(apiName=/demo,
payload=(\"tranid\": \"**********\",\"aggregatorVPA\": \" **********\",\"customerId\": \"**********\"),
httpStatus=0)'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START OBJECT at 1:336")))))

Can someone please help why this error is occurring. Also what can we do to resolve it ?

Thanks!

2

That's an opensearch exception, but it looks very similar to a mapping exception that elasticsearch will produce. I suggest you read this thread and this post. If those don't help then try asking in the OpenSearch forums.

3

Hi @Badger ,

Thanks for your inputs.

I have created a topic in opensearch.

Pinning the same question here as well.

Is there a way where we can retrigger the failed logs from logstash to opensearch ?

We know we can configure the dead letter queue but is there any other way apart from dead letter queue configuration.

Also, can we setup a predefined mapping schema in .conf file to resolve this issue.