Field references sometimes resolved and sometimes not

ElasticSearch + LogStash 6.5 / FileBeat 6.6 / Windows Server

In FileBeat we define the fields "source", "server" and "system" and add them together in Logstash to a field logfile according to the template %{[fields][source]}-%{[fields][server]}-%{[fields][system]}.

mutate {
    add_field => {
        "server" => "%{[fields][server]}"
        "logfile" => "%{[fields][source]}-%{[fields][server]}-%{[fields][system]}"
    }
}

This used to work fine but lately we keep getting (sporadic) error messages and the field often/sometimes being set to a combination of the processed and unprocessed template.

[2019-03-01T14:24:18,518][ERROR][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"access-app02-prod_2019-03-01T13:24:07.000Z_337202708,access-app02-prod,%{[fields][source]}-%{[fields][server]}-%{[fields][system]}2019-03-01T13:24:07.000Z%{offset}", :_index=>"access-prod,%{[fields][system]}", :_type=>"_doc", :_routing=>nil}, #LogStash::Event:0x4d265dfc], :response=>{"index"=>{"_index"=>"access-prod,%{[fields][system]}", "_type"=>"_doc", "_id"=>"access-app02-prod_2019-03-01T13:24:07.000Z_337202708,access-app02-prod,%{[fields][source]}-%{[fields][server]}-%{[fields][system]}2019-03-01T13:24:07.000Z%{offset}", "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [access-prod,%{[fields][system]}], must not contain the following characters [ , ", *, \, <, |, ,, >, /, ?]", "index_uuid"=>"na", "index"=>"access-prod,%{[fields][system]}"}}}}

I feel like there has been some change of late in how field templates %{}are handled. In particular we seem to have no issues with %{field} but more with %[field][subfield].

This issue only goes away if we stop setting document_id in the elasticsearch output. Something is seriously amiss with either ES or LS.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.