Issue indexing a field on elasticsearch 6.8

mgaspereyra · December 9, 2019, 2:34pm

Hi Everybody!
I Apologize to post this issue, as I think it should be easy to solve, but I am not finding the way to solve it.
My situation is as follows:
I have a Kong server which pushes his logs to elasticsearch, trough logstash http input plugin.
This configuration works quite well, but I'm seeing the following error constantly on my elasticsearch log.

[2019-12-09T11:31:49,442][DEBUG][o.e.a.b.TransportShardBulkAction] [blc-node-02] [kongprd-20191209][2] failed to execute bulk item (index) index {[kongprd-2019 1209][doc][xd4S624BEAdcJK_O0eG4], source[n/a, actual length: [2.7kb], max length: 2kb]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [api.methods] of type [text] in document with id 'xd4S624BEAdcJK_O0eG4'
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:303) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:488) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:395) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:395) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:96) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:281) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:799) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:775) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:744) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:458) ~[elasticsearch-6.8. 1.jar:6.8.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:481) ~[elasticsearc h-6.8.1.jar:6.8.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:456) ~[elasticsearch-6.8.1.jar:6.8 .1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:220) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:164) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:156) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:143) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:82) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1059) ~[elasti csearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1037) ~[elasti csearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.j ava:433) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:374) ~[ela sticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$14(IndexShard.java:2586) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:996) [elastics earch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:370) [elasticsearch -6.8.1.jar:6.8.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction. java:325) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction. java:312) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportIntercepto r.java:250) [x-pack-security-6.8.1.jar:6.8.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportIn terceptor.java:308) [x-pack-security-6.8.1.jar:6.8.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.8.1.jar:6.8. 1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.1.jar:6.8.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: java.lang.IllegalStateException: Can't get text on a START_OBJECT at 1:2101
at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:85) ~[elasticsearch-x-content-6.8.1.jar:6.8.1]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:279) ~[elasticsearch-x-content-6.8.1.jar:6.8 .1]
at org.elasticsearch.index.mapper.TextFieldMapper.parseCreateField(TextFieldMapper.java:828) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:297) ~[elasticsearch-6.8.1.jar:6.8.1]
... 48 more

Any suggestion to solve this issue?
Hope someone has a similar issue and may help with the solution!

Best Regards!!!!

spinscale · December 10, 2019, 12:49pm

please take your time to properly format your snippets. This forum supports markdown, so code snippets can easily be marked. This makes exceptions a million times more readable.

That said I assume that you have differently formed documents where in one document the field "api" : { "methods" : "foo" } is simply a string, and in another one it is an inner JSON object like "api" : { "methods" : { "foo" : "bar" }}. The latter one will return an exception, if the mapping is expecting a string.

--Alex

mgaspereyra · December 10, 2019, 3:28pm

Thanks! I'm going to review this.
If you're right, we need to change the mapping for that field, that's correct?

Best Regards!

mgaspereyra · December 13, 2019, 1:56pm

Hi!
I have an additional question.
It's ok to run logstash on debug / trace mode in order to capture the failing message?
Like this:
./logstash -f ../config/http_input_pipeline.conf --log.level=trace

Best regards!

mgaspereyra · December 13, 2019, 4:22pm

I've find a better way than enabling debug or trace.
I've configured the dead_letter_queue.

https://www.elastic.co/guide/en/logstash/6.8/dead-letter-queues.html

mgaspereyra · December 13, 2019, 4:40pm

Alexander, I've worked with the issue and found the following:
When the event has this values:
"methods" => ["GET"] (or "POST" or "GET","POST","OPTIONS")
it works as expected, but when the event has this value:
"api" => { "upstream_send_timeout" => 30000, "upstream_url" => "http://xxxxx.example.com/xx/yy", "methods" => {}, "https_only" => false, "created_at" => 1519851385877, "preserve_host" => false, "http_if_terminated" => false, "upstream_connect_timeout" => 30000, "retries" => 5, "uris" => [
it fails.
My index is configured to ignore malformed events:
"mapping": {"ignore_malformed": "true"}
and, the particular field, has null value treatment:
"methods": {"type": "text","fields":{"keyword":{"type":"keyword","null_value":"NULL", "ignore_above": 256}
I'm sure there's something I'm missing, but I'm not finding what is...

system · January 10, 2020, 4:40pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.