Failed to parse[host]?


(Paul) #1

I am developing a custom beats client. After a recent upgrade on my elastic server, the beat is now failing to publish with an error of;

    2018-09-30T22:25:17.152+0100    WARN    elasticsearch/client.go:520     
Cannot index event publisher.Event{..., "host":common.MapStr{"name":"DESKTOP-6KQL7DR"}, ...}, Private:interface {}(nil)}, Flags:0x0} (status=400): 
{"type":"mapper_parsing_exception","reason":"failed to parse [host]","caused_by":{"type":"illegal_state_exception",
"reason":"Can't get text on a START_OBJECT at 1:49"}}

There is a corresponding exception in the elastic logs.

I know there is a breaking change in the release notes for this, but I cannot see how to apply it to a custom beat - any suggestions ?

Thanks,


(Shaunak Kashyap) #2

Could you post the complete exception from the elasticsearch log here?


(Paul) #3
[2018-10-01T19:10:01,769][DEBUG][o.e.a.b.TransportShardBulkAction] [chaindata][3] failed to execute bulk item (index) BulkShardRequest [[chaindata][3]] containing [index {[chaindata][_doc][OB8JMWYB2O_hlHL_jpAp], source[_na_]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
    at org.elasticsearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:171) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:72) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:263) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:725) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:702) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:682) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$2(TransportShardBulkAction.java:560) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:579) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:558) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequest(TransportShardBulkAction.java:141) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:247) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:124) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:111) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:73) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1017) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:995) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:101) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:356) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:296) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:958) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:955) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:271) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2249) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:967) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:97) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:317) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:292) ~[elasticsearch-6.4.0.jar:6.4.0]

......        at 
org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:101) [x-pack-security-6.4.0.jar:6.4.0]
    at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:130) [x-pack-security-6.4.0.jar:6.4.0]
    at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307) [x-pack-security-6.4.0.jar:6.4.0]
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:665) [elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.0.jar:6.4.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.elasticsearch.common.compress.NotXContentException: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes
    at org.elasticsearch.common.compress.CompressorFactory.compressor(CompressorFactory.java:56) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.common.xcontent.XContentHelper.createParser(XContentHelper.java:69) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:65) ~[elasticsearch-6.4.0.jar:6.4.0]
    ... 54 more

(Paul) #4

I found the answer. The docs here;

describe how to configure host out of the beat;

processors:
   - drop_fields:
      fields: ["host"]

I find it strange though that libbeat 6.4 does not work out of the box with elastic 6.4.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.