Filebeat Management - Output Configuration Block not saving

Elastic Stack Beats
1

Hi,

I'm using filebeat centralized management, and things are setting properly.
But when I try to create a new beat tag with Output configuration block in Kibana UI it hanged in saving page and not saving it for me.
I tried putting Filebeat input config block instead of output config block and it saved, so it should be the output block specifically that is not working.
I didn't even link the tag to any filebeat yet, so it should not be a filebeat issue.

Please see below kibana logs FYI (nothing seems wrong):

{"type":"response","@timestamp":"2018-12-18T17:06:37Z","tags":,"pid":2937,"method":"put","statusCode":400,"req":{"url":"/api/beats/tag/Output","method":"put","headers":{"host":"ocdt70573375.office.adroot.bmogc.net:5601","connection":"keep-alive","content-length":"96","origin":"http://ocdt70573375.office.adroot.bmogc.net:5601","kbn-xsrf":"6.5.3","kbn-version":"6.5.3","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36","credentials":"same-origin","content-type":"application/json","accept":"application/json","referer":"http://ocdt70573375.office.adroot.bmogc.net:5601/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.0.2.2","userAgent":"10.0.2.2","referer":"http://ocdt70573375.office.adroot.bmogc.net:5601/app/kibana"},"res":{"statusCode":400,"responseTime":43,"contentLength":9},"message":"PUT /api/beats/tag/Output 400 43ms - 9.0B"}
{"type":"response","@timestamp":"2018-12-18T17:07:01Z","tags":,"pid":2937,"method":"get","statusCode":200,"req":{"url":"/api/beats/agent/dc6da37a-9652-47d4-9fae-083aa1bfd969/configuration?validSetting=true","method":"get","headers":{"host":"ocdt70573375.office.adroot.bmogc.net:5601","user-agent":"Go-http-client/1.1","content-length":"4","accept":"application/json","content-type":"application/json","kbn-beats-access-token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjcmVhdGVkIjoiMjAxOC0xMi0xOFQxNjoxMjozMy40MDlaIiwicmFuZG9tSGFzaCI6IjIxMTMwMDFlZWZhYzRiNjhhYzA1OTQzNmE5MGYyZmIxIiwiaWF0IjoxNTQ1MTQ5NTUzfQ.qTS74FzrNdlkXqjdbFNPHfCxyYChTjcZ5wMHfRQGQ5c","accept-encoding":"gzip"},"remoteAddress":"10.0.2.2","userAgent":"10.0.2.2"},"res":{"statusCode":200,"responseTime":205,"contentLength":9},"message":"GET /api/beats/agent/dc6da37a-9652-47d4-9fae-083aa1bfd969/configuration?validSetting=true 200 205ms - 9.0B"}

Please help advise the mechanism behind, and the possible reason for this. Thanks.

Regards,
Perry

2

Hi @bhavyarm

Would you please advise some updates?

Regards, Perry

3

When you go to save the tag, are there any errors in your browsers console?

4

Hi @Matthew_Apperson

No no error prompted.

5

Hi @Matthew_Apperson,

Sorry, apparently below prompted.

beat_config_kibana_console
beat_config_kibana_console.png552×635 32.5 KB

Please check and advise. Please let me know if you need those in text form.

Regards, Perry

6

Ok, using your browsers dev tools, can you inspect the network request for PUT /api/beats/tag/output? We need to take a look at the response from the server. There seems to be a bug in why it's not presenting the error in the UI. The raw response should at least tell us what is wrong as far as why it won't save.

7

Here's the response from /api/beats/tag/output

{"statusCode":400,"error":"Bad Request","message":"[mapper_parsing_exception] object mapping for [tag.configuration_blocks.configs.output] tried to parse field [output] as object, but found a concrete value"}

8

OK, and the request itself? (sorry, should have asked for that all at once)

9

Request Header:

PUT /api/beats/tag/output HTTP/1.1
Host: ocdt70573375.office.adroot.bmogc.net:5601
Connection: keep-alive
Content-Length: 101
Origin: http://ocdt70573375.office.adroot.bmogc.net:5601
kbn-xsrf: 6.5.3
kbn-version: 6.5.3
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36
credentials: same-origin
Content-Type: application/json
Accept: application/json
Referer: http://ocdt70573375.office.adroot.bmogc.net:5601/app/kibana
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cookie: sid=Fe26.2**1b8a39fca2dd08605dffbcb6a98a397a3915693a7e4b123e1356f8eaf91d8697*uN_nNkUj0lHBOgcHpN0F9Q*Dq9lyipw2g94uhh1mbBhaR6BlB8vwlYd0Qf_N4LueMwF4caHkV_K8zBcE-bD6eRLHM-JISAJFCOo4nMn0HzSofmXxZOtoPbO6khogDSqnof1AX5Q2ZeOOhCXgUX2q8dW_IYbKQ0K0W3JUcg3y96obg**bf31316af8d2ce559a659620bb513caaea1552c67bd8cb6b61fabcd59834e423*JLbvvEwOlkxqDXiKWDZhnDdxgiYQqmkWBve8r7n7mRY

Request Payload:

{"color":"#00b3a4","configuration_blocks":[{"type":"output","configs":[{"output":"elasticsearch"}]}]}

10

OK. So the issue here seems to be that at some point an output with a bad config format was pushed up. Has anyone attempted to add or edit a tag via the API in your setup? Or added an output.elasticsearch: {...} inside another config?

11

Are you referring to es cluster or the kibana or filebeat config? And how should I check this? It is my own sandbox no one will be touching this except myself.

12

Sorry, I mean in a config block of this or another tag.

13

The only other tag I has was a filebeat input tag.
Maybe there are some zombie tags on the cluster? Is there an API to show all the tags in the cluster?

14

There would be no zombie tags. But if a tag was created in the past containing an output in the “other” field. It’s affects on this issue would continue until it has been reindexed.

15

@Matthew_Apperson Do you mean reindexing .kibana? Or all the live indices?

16

sorry, reindexing .management-beats. This is the index beats management uses

17

Thanks. Do I just DELETE the index? Do I need to restart kibana or ES?

18

The easiest way for you to reindex is probably https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade-inplace.html

Please note we do consider the state your cluster is in right now to be a bug. You can track the status of the fix here - https://github.com/elastic/kibana/pull/27717

19

I do not have any data at all in the cluster.
If I just delete .management-beats will it recreate a brand new in the cluster when I used the beat configuration function?

20

Ah, then I believe that to be correct, yes