# Fluent parse error, original data now in message field: undefined method merge for nil:NilClass

**URL:** <https://discuss.elastic.co/t/fluent-parse-error-original-data-now-in-message-field-undefined-method-merge-for-nil-nilclass/121343>\
**Category:** Вопросы на русском языке\
**Created:** [February 24, 2018, 5:11pm UTC](https://discuss.elastic.co/t/fluent-parse-error-original-data-now-in-message-field-undefined-method-merge-for-nil-nilclass/121343 "2018-02-24T17:11:20Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![patsevanton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patsevanton/32/42692_2.png) [@patsevanton](https://discuss.elastic.co/u/patsevanton)\
**Post date:** [February 24, 2018, 5:11pm UTC](https://discuss.elastic.co/t/fluent-parse-error-original-data-now-in-message-field-undefined-method-merge-for-nil-nilclass/121343/1 "2018-02-24T17:11:21Z")

</div>

Привет всем!

При обновлении td-agent с 2.3.4 до 2.3.6 перестали поступать логи в elasticsearch.

Logstash ругается на

> [2018-02-24T19:28:04,478][ERROR][logstash.codecs.fluent] Fluent parse error, original data now in message field {:error=\>#\<NoMethodError: undefined method `merge' for 0:Fixnum>, :data=>115} [2018-02-24T19:28:14,374][ERROR][logstash.codecs.fluent] Fluent parse error, original data now in message field {:error=>#<NoMethodError: undefined method `merge' for nil:NilClass\>, :data=\>[37, 114]}

Лог td-agent говорит что отправляет данные

> T XX.XX.XX.XX:54534 -\> YY.YY.YY.YY:24224 [A]  
> .%ZZZZZZZ.nginx-kkkkkk.pub-access.......Z........remote..DD.DD.DD.DD..host..-..user..-..method..GET..path../static/sign-in.html..code..200..size..1060..referer..-..agent.iMozilla/5.0 (X11; Linux x86\_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36..http\_forwarder\_for..-..request\_time..0.003..response\_time..0.003..upstream\_addr..10.233.46.6:8086..cookie..-.HOSTNAME..SSS-frontend-balancer.original\_time...Z........remote..DD.DD.DD.DD..host..-..user..-..method..GET..path../static/css/data\_face.css..code..200..size..1443..referer.Chttps://kkkkkk-hhh-sss.XXXXXXX.ru/static/sign-in.html..agent.iMozilla/5.0 (X11; Linux x86\_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36..http\_forwarder\_for..-..request\_time..0.018..response\_time..0.018..upstream\_addr..10.233.46.6:8086..cookie..-.HOSTNAME..SSS-frontend-balancer.original\_time...Z........remote..DD.DD.DD.DD..host..-..user..-..method..GET..path../static/images/logo-rp.png..code..200..size..6750..referer.Chttps://kkkkkk-hhh-sss.XXXXXXX.ru/static/sign-in.html..agent.iMozilla/5.0 (X11; Linux x86\_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36..http\_forwarder\_for..-..request\_time..0.017..response\_time..0.017..upstream\_addr..10.233.46.6:8086..cookie..-.HOSTNAME..SSS-frontend-balancer.original\_time...Z........remote..DD.DD.DD.DD..host..-..user..-..method..GET..pa
> 
> T XX.XX.XX.XX:54534 -\> YY.YY.YY.YY:24224 [AFP]  
> th../static/js/login.js..code..200..size..383..referer.Chttps://kkkkkk-hhh-sss.XXXXXXX.ru/static/sign-in.html..agent.iMozilla/5.0 (X11; Linux x86\_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36..http\_forwarder\_for..-..request\_time..0.003..response\_time..0.003..upstream\_addr..10.233.46.6:8086..cookie..-.HOSTNAME..SSS-frontend-balancer.original\_time.

При использовани td-agent 2.3.4 logstash не выдает ошибок

**yum list logstash**

> logstash.noarch 1:5.6.4-1

При любой версии td-agent - td-agent выдает ошибку:

> plugin/filter\_record\_transformer.rb:242:log\_if\_unknown\_placeholder: unknown placeholder `${@timestamp}` found

Основной конфиг td-agent

> @include conf.d/\*.conf
> 
> \<filter jjjjjjjj.\*\*\>  
> @type record\_transformer  
>   
> original\_time ${@timestamp}
> 
> \<match jjjjjjjj.\*\*\>  
> @type forward
> 
> host hh.hh.hh.hh port iiiii 
> 
> ```
> heartbeat_type tcp
> buffer_type file
> buffer_queue_limit 128
> buffer_path /var/cache/td-agent/buffer/
> flush_interval 10s
> flush_at_shutdown true
> 
> ```

Конфиг для td-agent для парсинга конкретного лога nginx

> \<match nginx-lllllll-log.\*\>  
> type hostname  
> key\_name HOSTNAME  
> add\_prefix eeeeeeeeee
> 
> type better\_timestamp tag eeeeeeeeee.nginx-lllllll.pub-access type tail path /var/log/nginx/lllllll-access.log pos\_file /var/cache/td-agent/lllllll-nginx-access.pos tag nginx-lllllll-log.access format /^(?[^]\*) (?[^]\*) (?[^]\*) \[(?[^\]]\*)\] \"(?:(?[^]\*)\ +)?(?[^\]\*)[^\"]\*(?: HTTP/[^\"]\*)?\" (?`[^]*) ([-]|(?[^]*)) \"(?[^\"]*)\" \"(?[^\"]*)\" \"(?[^\"]*)\"(?: (?[^]*) ([-]|(?[^\,]+))(?: ?[\,\:] ?([-]|(?[^\,]+)))?(?: ?[\,\:] ?([-]|(?[^\,]+)))? [\.p] ([-]|(?[^\,]+))(?: ?[\,\:] ?([-]|(?[^\,]+)))?(?: ?[\,\:] ?([-]|(?[^\,]+)))? \"(?[^\"]*)\")?$/
> time_format %d/%b/%Y:%H:%M:%S %z
> 
> `

`
filter.conf

filter {
if [original_time] != "" {
date{
match => ["original_time" , "ISO8601"]
}
}
mutate {
rename => {
"tags" => "tag"
"[beat][hostname]" => "HOSTNAME"
}
replace => { "[@metadata][indexname]" => "%{tag}" }
gsub => ["[@metadata][indexname]", "(\[1\].[^.]).(.*$)", '\1']
add_field => {
"[post_environment]" => "preprod"
}
remove_field => ['original_time', '[beat][name]', '[beat][version]', 'input_type', 'type', 'offset', 'source']
}
if [@metadata][indexname] == "_fluentparsefailure" {
drop { }
}
}

input_fluentd.conf

input {
tcp {
codec => fluent
port => 24224
}
}

output_elasticsearch.conf

output {
elasticsearch {
manage_template => false
hosts => ["XX.XX.XX.XX2:9200","XX.XX.XX.XX3:9200","XX.XX.XX.XX4:9200","XX.XX.XX.XX5:9200","XX.XX.XX.XX6:9200"]
index => "%{[@metadata][indexname]}-%{+YYYY.MM.dd}"
document_type => "jjjjjjjj_mapping"
}
}

Gemspec доп плагинов для td-agent 2.3.6

elasticsearch-5.0.4/elasticsearch.gemspec
elasticsearch-api-5.0.4/elasticsearch-api.gemspec
elasticsearch-transport-5.0.4/elasticsearch-transport.gemspec
excon-0.59.0/excon.gemspec
fluent-mixin-config-placeholders-0.4.0/fluent-mixin-config-placeholders.gemspec
fluent-plugin-better-timestamp-0.1.0/fluent-plugin-better-timestamp.gemspec
fluent-plugin-elasticsearch-1.10.2/fluent-plugin-elasticsearch.gemspec
fluent-plugin-forest-0.3.3/fluent-plugin-forest.gemspec
fluent-plugin-hostname-0.0.2/fluent-plugin-hostname.gemspec
fluent-plugin-tail-multiline-ex-0.0.3/fluent-plugin-tail-multiline-ex.gemspec
fluent-plugin-zabbix-agent-0.2.0/fluent-plugin-zabbix-agent.gemspec
fluentd-0.10.62/fluentd.gemspec
msgpack-0.5.12/msgpack.gemspec
multipart-post-2.0.0/multipart-post.gemspec
zabbix_protocol-0.1.5/zabbix_protocol.gemspec

^. 

`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 24, 2018, 5:11pm UTC](https://discuss.elastic.co/t/fluent-parse-error-original-data-now-in-message-field-undefined-method-merge-for-nil-nilclass/121343/2 "2018-03-24T17:11:47Z")

</div>

Closed on March 24, 2018, 5:11pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.