また、td-agent に以下のログが出力される前に、
2019-03-07 16:24:25 +0900 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2019-03-07 16:24:26 +0900 chunk="5837bfdbba07c0e583347a95680b13b4" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"localhost", :port=>9200, :scheme=>"http"}): Connection reset by peer (Errno::ECONNRESET)"
2019-03-07 16:24:25 +0900 [warn]: #0 suppressed same stacktrace
2019-03-07 16:24:27 +0900 [warn]: #0 failed to flush the buffer. retry_time=1 next_retry_seconds=2019-03-07 16:24:28 +0900 chunk="5837bfdbba07c0e583347a95680b13b4" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"localhost", :port=>9200, :scheme=>"http"}): Broken pipe (Errno::EPIPE)"
次のようなログが大量に出力されていました。
2019-03-07 16:19:14 +0900 [warn]: #0 dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="syslog.local0.info" time=2019-03-07 16:19:03.000000000 +0900 record={"host"=>"10.x.x.x", "message"=>"EvntSLog: RealSource:"host1.sample.co.jp" [INF] [Source:Service Control Manager] [Category:0] [ID:7036] [User:N\A] 2019-03-07 16:19:03 The Google Update \xA5\xB5\xA1\xBC\xA5\xD3\xA5\xB9 (gupdate) service entered the running state.", "priority"=>"info", "facility"=>"local0"}
2019-03-07 16:19:36 +0900 [warn]: #0 dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="fluent.warn" time=2019-03-07 16:19:14.055454146 +0900 record={"error"=>"#<Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError: 400 - Rejected by Elasticsearch>", "location"=>nil, "tag"=>"syslog.local0.info", "time"=>2019-03-07 16:19:03.000000000 +0900, "record"=>{"host"=>"10.x.x.x", "message"=>"EvntSLog: RealSource:"host1.sample.co.jp" [INF] [Source:Service Control Manager] [Category:0] [ID:7036] [User:N\A] 2019-03-07 16:19:03 The Google Update \xA5\xB5\xA1\xBC\xA5\xD3\xA5\xB9 (gupdate) service entered the running state.", "priority"=>"info", "facility"=>"local0"}, "message"=>"dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="syslog.local0.info" time=2019-03-07 16:19:03.000000000 +0900 record={"host"=>"10.x.x.x", "message"=>"EvntSLog: RealSource:\"host1.sample.co.jp\" [INF] [Source:Service Control Manager] [Category:0] [ID:7036] [User:N\\A] 2019-03-07 16:19:03 The Google Update \xA5\xB5\xA1\xBC\xA5\xD3\xA5\xB9 (gupdate) service entered the running state.", "priority"=>"info", "facility"=>"local0"}"}
elasticsearchのログにも以下のような出力があります。
"Caused by: com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0xa5"とあり、
どうも文字コードが問題でElasticsearchでエラーになっているように思われます。
回避方法をご存知の方がおりましたら、ご教示頂けますでしょうか？
[2019-03-07T16:19:14,028][DEBUG][o.e.a.b.TransportShardBulkAction] [s0055u-logsearch] [logstash-2019.03.07][2] failed to execute bulk item (index) index {[logstash-2019.03.07][fluentd][FSEFV2kB9Q1oFsp6OQxi], source[{"host":"10.x.x.x","message":"EvntSLog: RealSource:"host1.sample.co.jp" [INF] [Source:Service Control Manager] [Category:0] [ID:7036] [User:N\A] 2019-03-07 16:19:03 The Google Update (gupdate) service entered the running state.","priority":"info","facility":"local0","@timestamp":"2019-03-07T16:19:03.000000000+09:00"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [message] of type [text]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:303) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:488) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:616) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:410) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:96) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:281) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:763) ~[elasticsearch-6.6.0.jar:6.6.0]
(～省略～)
Caused by: com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0xa5
at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@5840cbf3; line: 1, column: 193]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.11.jar:2.8.11]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.11.jar:2.8.11]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidInitial(UTF8StreamJsonParser.java:3544) ~[jackson-core-2.8.11.jar:2.8.11]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidChar(UTF8StreamJsonParser.java:3538) ~[jackson-core-2.8.11.jar:2.8.11]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2543) ~[jackson-core-2.8.11.jar:2.8.11]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2469) ~[jackson-core-2.8.11.jar:2.8.11]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:315) ~[jackson-core-2.8.11.jar:2.8.11]
at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83) ~[elasticsearch-x-content-6.6.0.jar:6.6.0]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:269) ~[elasticsearch-x-content-6.6.0.jar:6.6.0]
at org.elasticsearch.index.mapper.TextFieldMapper.parseCreateField(TextFieldMapper.java:719) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:297) ~[elasticsearch-6.6.0.jar:6.6.0]
... 42 more