Filebeat Management - Output Configuration Block not saving


I'm using filebeat centralized management, and things are setting properly.
But when I try to create a new beat tag with Output configuration block in Kibana UI it hanged in saving page and not saving it for me.
I tried putting Filebeat input config block instead of output config block and it saved, so it should be the output block specifically that is not working.
I didn't even link the tag to any filebeat yet, so it should not be a filebeat issue.

Please see below kibana logs FYI (nothing seems wrong):

{"type":"response","@timestamp":"2018-12-18T17:06:37Z","tags":,"pid":2937,"method":"put","statusCode":400,"req":{"url":"/api/beats/tag/Output","method":"put","headers":{"host":"","connection":"keep-alive","content-length":"96","origin":"","kbn-xsrf":"6.5.3","kbn-version":"6.5.3","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36","credentials":"same-origin","content-type":"application/json","accept":"application/json","referer":"","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"","userAgent":"","referer":""},"res":{"statusCode":400,"responseTime":43,"contentLength":9},"message":"PUT /api/beats/tag/Output 400 43ms - 9.0B"}
{"type":"response","@timestamp":"2018-12-18T17:07:01Z","tags":,"pid":2937,"method":"get","statusCode":200,"req":{"url":"/api/beats/agent/dc6da37a-9652-47d4-9fae-083aa1bfd969/configuration?validSetting=true","method":"get","headers":{"host":"","user-agent":"Go-http-client/1.1","content-length":"4","accept":"application/json","content-type":"application/json","kbn-beats-access-token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjcmVhdGVkIjoiMjAxOC0xMi0xOFQxNjoxMjozMy40MDlaIiwicmFuZG9tSGFzaCI6IjIxMTMwMDFlZWZhYzRiNjhhYzA1OTQzNmE5MGYyZmIxIiwiaWF0IjoxNTQ1MTQ5NTUzfQ.qTS74FzrNdlkXqjdbFNPHfCxyYChTjcZ5wMHfRQGQ5c","accept-encoding":"gzip"},"remoteAddress":"","userAgent":""},"res":{"statusCode":200,"responseTime":205,"contentLength":9},"message":"GET /api/beats/agent/dc6da37a-9652-47d4-9fae-083aa1bfd969/configuration?validSetting=true 200 205ms - 9.0B"}

Please help advise the mechanism behind, and the possible reason for this. Thanks.


Hi @bhavyarm

Would you please advise some updates?

Regards, Perry

When you go to save the tag, are there any errors in your browsers console?

Hi @Matthew_Apperson

No no error prompted.

Hi @Matthew_Apperson,

Sorry, apparently below prompted.

Please check and advise. Please let me know if you need those in text form.

Regards, Perry

Ok, using your browsers dev tools, can you inspect the network request for PUT /api/beats/tag/output? We need to take a look at the response from the server. There seems to be a bug in why it's not presenting the error in the UI. The raw response should at least tell us what is wrong as far as why it won't save.

Here's the response from /api/beats/tag/output

{"statusCode":400,"error":"Bad Request","message":"[mapper_parsing_exception] object mapping for [tag.configuration_blocks.configs.output] tried to parse field [output] as object, but found a concrete value"}

OK, and the request itself? (sorry, should have asked for that all at once)

Request Header:

PUT /api/beats/tag/output HTTP/1.1
Connection: keep-alive
Content-Length: 101
kbn-xsrf: 6.5.3
kbn-version: 6.5.3
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36
credentials: same-origin
Content-Type: application/json
Accept: application/json
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cookie: sid=Fe26.2**1b8a39fca2dd08605dffbcb6a98a397a3915693a7e4b123e1356f8eaf91d8697*uN_nNkUj0lHBOgcHpN0F9Q*Dq9lyipw2g94uhh1mbBhaR6BlB8vwlYd0Qf_N4LueMwF4caHkV_K8zBcE-bD6eRLHM-JISAJFCOo4nMn0HzSofmXxZOtoPbO6khogDSqnof1AX5Q2ZeOOhCXgUX2q8dW_IYbKQ0K0W3JUcg3y96obg**bf31316af8d2ce559a659620bb513caaea1552c67bd8cb6b61fabcd59834e423*JLbvvEwOlkxqDXiKWDZhnDdxgiYQqmkWBve8r7n7mRY

Request Payload:


OK. So the issue here seems to be that at some point an output with a bad config format was pushed up. Has anyone attempted to add or edit a tag via the API in your setup? Or added an output.elasticsearch: {...} inside another config?

Are you referring to es cluster or the kibana or filebeat config? And how should I check this? It is my own sandbox no one will be touching this except myself.

Sorry, I mean in a config block of this or another tag.

The only other tag I has was a filebeat input tag.
Maybe there are some zombie tags on the cluster? Is there an API to show all the tags in the cluster?

There would be no zombie tags. But if a tag was created in the past containing an output in the “other” field. It’s affects on this issue would continue until it has been reindexed.

@Matthew_Apperson Do you mean reindexing .kibana? Or all the live indices?

sorry, reindexing .management-beats. This is the index beats management uses

Thanks. Do I just DELETE the index? Do I need to restart kibana or ES?

The easiest way for you to reindex is probably

Please note we do consider the state your cluster is in right now to be a bug. You can track the status of the fix here -

I do not have any data at all in the cluster.
If I just delete .management-beats will it recreate a brand new in the cluster when I used the beat configuration function?

Ah, then I believe that to be correct, yes