Failed to parse[host]?

paulkeogh · September 30, 2018, 9:33pm

I am developing a custom beats client. After a recent upgrade on my elastic server, the beat is now failing to publish with an error of;

    2018-09-30T22:25:17.152+0100    WARN    elasticsearch/client.go:520     
Cannot index event publisher.Event{..., "host":common.MapStr{"name":"DESKTOP-6KQL7DR"}, ...}, Private:interface {}(nil)}, Flags:0x0} (status=400): 
{"type":"mapper_parsing_exception","reason":"failed to parse [host]","caused_by":{"type":"illegal_state_exception",
"reason":"Can't get text on a START_OBJECT at 1:49"}}

There is a corresponding exception in the elastic logs.

I know there is a breaking change in the release notes for this, but I cannot see how to apply it to a custom beat - any suggestions ?

Thanks,

shaunak · October 1, 2018, 2:55am

Could you post the complete exception from the elasticsearch log here?

paulkeogh · October 1, 2018, 7:24pm

[2018-10-01T19:10:01,769][DEBUG][o.e.a.b.TransportShardBulkAction] [chaindata][3] failed to execute bulk item (index) BulkShardRequest [[chaindata][3]] containing [index {[chaindata][_doc][OB8JMWYB2O_hlHL_jpAp], source[_na_]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
    at org.elasticsearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:171) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:72) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:263) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:725) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:702) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:682) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$2(TransportShardBulkAction.java:560) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:579) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:558) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequest(TransportShardBulkAction.java:141) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:247) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:124) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:111) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:73) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1017) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:995) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:101) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:356) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:296) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:958) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:955) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:271) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2249) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:967) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:97) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:317) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:292) ~[elasticsearch-6.4.0.jar:6.4.0]

......        at 
org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:101) [x-pack-security-6.4.0.jar:6.4.0]
    at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:130) [x-pack-security-6.4.0.jar:6.4.0]
    at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307) [x-pack-security-6.4.0.jar:6.4.0]
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:665) [elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.0.jar:6.4.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.elasticsearch.common.compress.NotXContentException: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes
    at org.elasticsearch.common.compress.CompressorFactory.compressor(CompressorFactory.java:56) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.common.xcontent.XContentHelper.createParser(XContentHelper.java:69) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:65) ~[elasticsearch-6.4.0.jar:6.4.0]
    ... 54 more

paulkeogh · October 1, 2018, 8:17pm

I found the answer. The docs here;

github.com

elastic/beats/blob/master/libbeat/docs/breaking.asciidoc#breaking-changes-6.3

:see-relnotes: See the <<release-notes,release notes>> for a complete list of breaking changes, including changes to beta or experimental functionality.

[[breaking-changes]]
== Breaking changes

As a general rule, we strive to keep backwards compatibility between minor
versions (e.g.  6.x to 6.y) so you can upgrade without any configuration file
changes, but there are breaking changes between major versions (e.g. 5.x to
6.y). Migrating directly between non consecutive major versions (e.g. 1.x to
6.x) is not recommended.

See the following topics for a description of breaking changes:

* <<breaking-changes-6.3>>
* <<breaking-changes-6.2>>
* <<breaking-changes-6.1>>
* <<breaking-changes-6.0>>
* {auditbeat}/auditbeat-breaking-changes.html[Breaking changes in Auditbeat 6.2]

[[breaking-changes-6.3]]

This file has been truncated. show original

describe how to configure host out of the beat;

processors:
   - drop_fields:
      fields: ["host"]

I find it strange though that libbeat 6.4 does not work out of the box with elastic 6.4.

system · October 29, 2018, 8:23pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Failed to parse [host] error Logstash	3	801	January 24, 2019
Can't get text on a START_OBJECT Beats filebeat	3	4677	October 26, 2020
Failed to parse the field Elasticsearch	2	698	October 26, 2020
Failed to parse field [host] of type [text] .. can't get text on a START_OBJECT at 1:974 Logstash beats-module	3	12889	June 30, 2020
Filebeat Parsing error Beats filebeat	2	524	October 31, 2019

Failed to parse[host]?

Related topics