Failed to parse[host]?

I am developing a custom beats client. After a recent upgrade on my elastic server, the beat is now failing to publish with an error of;

    2018-09-30T22:25:17.152+0100    WARN    elasticsearch/client.go:520     
Cannot index event publisher.Event{..., "host":common.MapStr{"name":"DESKTOP-6KQL7DR"}, ...}, Private:interface {}(nil)}, Flags:0x0} (status=400): 
{"type":"mapper_parsing_exception","reason":"failed to parse [host]","caused_by":{"type":"illegal_state_exception",
"reason":"Can't get text on a START_OBJECT at 1:49"}}

There is a corresponding exception in the elastic logs.

I know there is a breaking change in the release notes for this, but I cannot see how to apply it to a custom beat - any suggestions ?

Thanks,

Could you post the complete exception from the elasticsearch log here?

[2018-10-01T19:10:01,769][DEBUG][o.e.a.b.TransportShardBulkAction] [chaindata][3] failed to execute bulk item (index) BulkShardRequest [[chaindata][3]] containing [index {[chaindata][_doc][OB8JMWYB2O_hlHL_jpAp], source[_na_]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
    at org.elasticsearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:171) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:72) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:263) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:725) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:702) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:682) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$2(TransportShardBulkAction.java:560) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:579) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:558) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequest(TransportShardBulkAction.java:141) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:247) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:124) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:111) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:73) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1017) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:995) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:101) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:356) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:296) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:958) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:955) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:271) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2249) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:967) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:97) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:317) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:292) ~[elasticsearch-6.4.0.jar:6.4.0]

......        at 
org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:101) [x-pack-security-6.4.0.jar:6.4.0]
    at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:130) [x-pack-security-6.4.0.jar:6.4.0]
    at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307) [x-pack-security-6.4.0.jar:6.4.0]
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:665) [elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.0.jar:6.4.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.elasticsearch.common.compress.NotXContentException: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes
    at org.elasticsearch.common.compress.CompressorFactory.compressor(CompressorFactory.java:56) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.common.xcontent.XContentHelper.createParser(XContentHelper.java:69) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:65) ~[elasticsearch-6.4.0.jar:6.4.0]
    ... 54 more

I found the answer. The docs here;

describe how to configure host out of the beat;

processors:
   - drop_fields:
      fields: ["host"]

I find it strange though that libbeat 6.4 does not work out of the box with elastic 6.4.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.