Issue indexing a field on elasticsearch 6.8

Hi Everybody!
I Apologize to post this issue, as I think it should be easy to solve, but I am not finding the way to solve it.
My situation is as follows:
I have a Kong server which pushes his logs to elasticsearch, trough logstash http input plugin.
This configuration works quite well, but I'm seeing the following error constantly on my elasticsearch log.

[2019-12-09T11:31:49,442][DEBUG][o.e.a.b.TransportShardBulkAction] [blc-node-02] [kongprd-20191209][2] failed to execute bulk item (index) index {[kongprd-2019 1209][doc][xd4S624BEAdcJK_O0eG4], source[n/a, actual length: [2.7kb], max length: 2kb]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [api.methods] of type [text] in document with id 'xd4S624BEAdcJK_O0eG4'
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:303) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:488) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:395) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:395) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:96) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:281) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:799) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:775) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:744) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:458) ~[elasticsearch-6.8. 1.jar:6.8.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:481) ~[elasticsearc h-6.8.1.jar:6.8.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:456) ~[elasticsearch-6.8.1.jar:6.8 .1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:220) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:164) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:156) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:143) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:82) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1059) ~[elasti csearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1037) ~[elasti csearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.j ava:433) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:374) ~[ela sticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$14(IndexShard.java:2586) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:996) [elastics earch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:370) [elasticsearch -6.8.1.jar:6.8.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction. java:325) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction. java:312) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportIntercepto r.java:250) [x-pack-security-6.8.1.jar:6.8.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportIn terceptor.java:308) [x-pack-security-6.8.1.jar:6.8.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.8.1.jar:6.8. 1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.1.jar:6.8.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: java.lang.IllegalStateException: Can't get text on a START_OBJECT at 1:2101
at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:85) ~[elasticsearch-x-content-6.8.1.jar:6.8.1]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:279) ~[elasticsearch-x-content-6.8.1.jar:6.8 .1]
at org.elasticsearch.index.mapper.TextFieldMapper.parseCreateField(TextFieldMapper.java:828) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:297) ~[elasticsearch-6.8.1.jar:6.8.1]
... 48 more

Any suggestion to solve this issue?
Hope someone has a similar issue and may help with the solution!

Best Regards!!!!

please take your time to properly format your snippets. This forum supports markdown, so code snippets can easily be marked. This makes exceptions a million times more readable.

That said I assume that you have differently formed documents where in one document the field "api" : { "methods" : "foo" } is simply a string, and in another one it is an inner JSON object like "api" : { "methods" : { "foo" : "bar" }}. The latter one will return an exception, if the mapping is expecting a string.

--Alex

Thanks! I'm going to review this.
If you're right, we need to change the mapping for that field, that's correct?

Best Regards!

Hi!
I have an additional question.
It's ok to run logstash on debug / trace mode in order to capture the failing message?
Like this:
./logstash -f ../config/http_input_pipeline.conf --log.level=trace

Best regards!

I've find a better way than enabling debug or trace.
I've configured the dead_letter_queue.

https://www.elastic.co/guide/en/logstash/6.8/dead-letter-queues.html

Alexander, I've worked with the issue and found the following:
When the event has this values:
"methods" => ["GET"] (or "POST" or "GET","POST","OPTIONS")
it works as expected, but when the event has this value:
"api" => { "upstream_send_timeout" => 30000, "upstream_url" => "http://xxxxx.example.com/xx/yy", "methods" => {}, "https_only" => false, "created_at" => 1519851385877, "preserve_host" => false, "http_if_terminated" => false, "upstream_connect_timeout" => 30000, "retries" => 5, "uris" => [
it fails.
My index is configured to ignore malformed events:
"mapping": {"ignore_malformed": "true"}
and, the particular field, has null value treatment:
"methods": {"type": "text","fields":{"keyword":{"type":"keyword","null_value":"NULL", "ignore_above": 256}
I'm sure there's something I'm missing, but I'm not finding what is...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.