Fluent parse error, original data now in message field: undefined method merge for nil:NilClass

[patsevanton (Anton Patsev)]

Привет всем!

При обновлении td-agent с 2.3.4 до 2.3.6 перестали поступать логи в elasticsearch.

Logstash ругается на

[2018-02-24T19:28:04,478][ERROR][logstash.codecs.fluent ] Fluent parse error, original data now in message field {:error=>#<NoMethodError: undefined method merge' for 0:Fixnum>, :data=>115} [2018-02-24T19:28:14,374][ERROR][logstash.codecs.fluent ] Fluent parse error, original data now in message field {:error=>#<NoMethodError: undefined method merge' for nil:NilClass>, :data=>[37, 114]}

Лог td-agent говорит что отправляет данные

T XX.XX.XX.XX:54534 -> YY.YY.YY.YY:24224 [A]
.%ZZZZZZZ.nginx-kkkkkk.pub-access.......Z........remote..DD.DD.DD.DD..host..-..user..-..method..GET..path../static/sign-in.html..code..200..size..1060..referer..-..agent.iMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36..http_forwarder_for..-..request_time..0.003..response_time..0.003..upstream_addr..10.233.46.6:8086..cookie..-.HOSTNAME..SSS-frontend-balancer.original_time...Z........remote..DD.DD.DD.DD..host..-..user..-..method..GET..path../static/css/data_face.css..code..200..size..1443..referer.Chttps://kkkkkk-hhh-sss.XXXXXXX.ru/static/sign-in.html..agent.iMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36..http_forwarder_for..-..request_time..0.018..response_time..0.018..upstream_addr..10.233.46.6:8086..cookie..-.HOSTNAME..SSS-frontend-balancer.original_time...Z........remote..DD.DD.DD.DD..host..-..user..-..method..GET..path../static/images/logo-rp.png..code..200..size..6750..referer.Chttps://kkkkkk-hhh-sss.XXXXXXX.ru/static/sign-in.html..agent.iMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36..http_forwarder_for..-..request_time..0.017..response_time..0.017..upstream_addr..10.233.46.6:8086..cookie..-.HOSTNAME..SSS-frontend-balancer.original_time...Z........remote..DD.DD.DD.DD..host..-..user..-..method..GET..pa

T XX.XX.XX.XX:54534 -> YY.YY.YY.YY:24224 [AFP]
th../static/js/login.js..code..200..size..383..referer.Chttps://kkkkkk-hhh-sss.XXXXXXX.ru/static/sign-in.html..agent.iMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36..http_forwarder_for..-..request_time..0.003..response_time..0.003..upstream_addr..10.233.46.6:8086..cookie..-.HOSTNAME..SSS-frontend-balancer.original_time.

При использовани td-agent 2.3.4 logstash не выдает ошибок

yum list logstash

logstash.noarch 1:5.6.4-1

При любой версии td-agent - td-agent выдает ошибку:

plugin/filter_record_transformer.rb:242:log_if_unknown_placeholder: unknown placeholder ${@timestamp} found

Основной конфиг td-agent

@include conf.d/*.conf

<filter jjjjjjjj.**>
@type record_transformer

original_time ${@timestamp}

<match jjjjjjjj.**>
@type forward

host hh.hh.hh.hh port iiiii

heartbeat_type tcp
buffer_type file
buffer_queue_limit 128
buffer_path /var/cache/td-agent/buffer/
flush_interval 10s
flush_at_shutdown true

Конфиг для td-agent для парсинга конкретного лога nginx

<match nginx-lllllll-log.*>
type hostname
key_name HOSTNAME
add_prefix eeeeeeeeee

type better_timestamp tag eeeeeeeeee.nginx-lllllll.pub-access type tail path /var/log/nginx/lllllll-access.log pos_file /var/cache/td-agent/lllllll-nginx-access.pos tag nginx-lllllll-log.access format /^(?[^ ]*) (?[^ ]*) (?[^ ]*) \[(?[^\]]*)\] \"(?:(?[^ ]*)\ +)?(?[^\ ]*)[^\"]*(?: HTTP/[^\"]*)?\" (?[^ ]*) ([-]|(?[^ ]*)) \"(?[^\"]*)\" \"(?[^\"]*)\" \"(?[^\"]*)\"(?: (?[^ ]*) ([-]|(?[^\, ]+))(?: ?[\,\:] ?([-]|(?[^\, ]+)))?(?: ?[\,\:] ?([-]|(?[^\, ]+)))? [\.p] ([-]|(?[^\, ]+))(?: ?[\,\:] ?([-]|(?[^\, ]+)))?(?: ?[\,\:] ?([-]|(?[^\, ]+)))? \"(?[^\"]*)\")?$/ time_format %d/%b/%Y:%H:%M:%S %z

filter.conf

filter {

if [original_time] != "" {
date{
match => ["original_time" , "ISO8601"]
}
}

mutate {
rename => {
"tags" => "tag"
"[beat][hostname]" => "HOSTNAME"
}
replace => { "[@metadata][indexname]" => "%{tag}" }
gsub => [ "[@metadata][indexname]", "(^[1].[^.]).(.*$)", '\1']
add_field => {
"[post_environment]" => "preprod"
}
remove_field => ['original_time', '[beat][name]', '[beat][version]', 'input_type', 'type', 'offset', 'source']
}

if [@metadata][indexname] == "_fluentparsefailure" {
drop { }
}

}

input_fluentd.conf

input {
tcp {
codec => fluent
port => 24224
}
}

output_elasticsearch.conf

output {
elasticsearch {
manage_template => false
hosts => ["XX.XX.XX.XX2:9200","XX.XX.XX.XX3:9200","XX.XX.XX.XX4:9200","XX.XX.XX.XX5:9200","XX.XX.XX.XX6:9200"]
index => "%{[@metadata][indexname]}-%{+YYYY.MM.dd}"
document_type => "jjjjjjjj_mapping"
}
}

Gemspec доп плагинов для td-agent 2.3.6

elasticsearch-5.0.4/elasticsearch.gemspec
elasticsearch-api-5.0.4/elasticsearch-api.gemspec
elasticsearch-transport-5.0.4/elasticsearch-transport.gemspec
excon-0.59.0/excon.gemspec
fluent-mixin-config-placeholders-0.4.0/fluent-mixin-config-placeholders.gemspec
fluent-plugin-better-timestamp-0.1.0/fluent-plugin-better-timestamp.gemspec
fluent-plugin-elasticsearch-1.10.2/fluent-plugin-elasticsearch.gemspec
fluent-plugin-forest-0.3.3/fluent-plugin-forest.gemspec
fluent-plugin-hostname-0.0.2/fluent-plugin-hostname.gemspec
fluent-plugin-tail-multiline-ex-0.0.3/fluent-plugin-tail-multiline-ex.gemspec
fluent-plugin-zabbix-agent-0.2.0/fluent-plugin-zabbix-agent.gemspec
fluentd-0.10.62/fluentd.gemspec
msgpack-0.5.12/msgpack.gemspec
multipart-post-2.0.0/multipart-post.gemspec
zabbix_protocol-0.1.5/zabbix_protocol.gemspec

---

1. ^. ↩︎