Could not index event to Elasticsearch eror

rcbart · December 3, 2018, 4:43am

I am getting the following error when parsing an apache log:

[2018-12-02T21:11:37,512][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"demo-index", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x37e63d61], :response=>{"index"=>{"_index"=>"demo-index", "_type"=>"doc", "_id"=>"9r9DcmcB9QxfD38crnmY", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:620"}}}}}

Can someone please help?

Thank you

warkolm · December 3, 2018, 5:42am

What do the Elasticsearch logs show at that time.

rcbart · December 3, 2018, 4:28pm

Here is a snippet of the elasticsearch log file (I have to break it up into two parts):

[2018-12-03T09:25:09,766][DEBUG][o.e.a.b.TransportShardBulkAction] [sX_o-Hp] [demo-index][3] failed to execute bulk item (index) index {[demo-index][doc][xbvjdGcB6fM7ibVUPCrf], source[{"input":{"type":"log"},"message":"Dec 3 09:25:04 Rons-ubuntu kernel: [16717.403995] CPU2: Package temperature above threshold, cpu clock throttled (total events = 1)","@version":"1","host":{"id":"cb497fc7a8634953904c9858d04772ec","architecture":"x86_64","os":{"family":"debian","version":"18.04.1 LTS (Bionic Beaver)","codename":"bionic","platform":"ubuntu"},"containerized":false,"name":"Rons-ubuntu"},"offset":360112,"prospector":{"type":"log"},"@timestamp":"2018-12-03T16:25:04.777Z","beat":{"version":"6.5.1","name":"Rons-ubuntu","hostname":"Rons-ubuntu"},"source":"/var/log/kern.log","tags":["beats_input_codec_plain_applied","_grokparsefailure","_geoip_lookup_failure"]}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [host] of type [text]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:301) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:482) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:499) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:391) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:381) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:96) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:280) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:748) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:725) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:705) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:461) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:483) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:459) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1022) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1000) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:102) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:356) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:296) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:963) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:960) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:271) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2327) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:972) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:97) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:317) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:292) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:300) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.5.1.jar:6.5.1]

rcbart · December 3, 2018, 4:28pm

Part 2:

at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$2(ServerTransportFilter.java:154) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:173) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:167) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:155) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$3(ServerTransportFilter.java:156) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:176) ~[x-pack-security-6.5.1.jar:6.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:209) ~[x-pack-security-6.5.1.jar:6.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:220) [x-pack-security-6.5.1.jar:6.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:174) [x-pack-security-6.5.1.jar:6.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:134) [x-pack-security-6.5.1.jar:6.5.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:104) [x-pack-security-6.5.1.jar:6.5.1]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:130) [x-pack-security-6.5.1.jar:6.5.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307) [x-pack-security-6.5.1.jar:6.5.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:717) [elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.1.jar:6.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

Caused by: java.lang.IllegalStateException: Can't get text on a START_OBJECT at 1:192
at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:86) ~[elasticsearch-x-content-6.5.1.jar:6.5.1]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:269) ~[elasticsearch-x-content-6.5.1.jar:6.5.1]
at org.elasticsearch.index.mapper.TextFieldMapper.parseCreateField(TextFieldMapper.java:719) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:295) ~[elasticsearch-6.5.1.jar:6.5.1]
... 60 more

system · December 31, 2018, 4:28pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.