Beat setup dashboards don't work in different spaces

Elastic Stack Kibana
1

I'm running the 8.10.4 stack and having trouble settings up the dashboards for various beats. I run a simple setup command using Docker and all the dashboards for a given beat are imported. However, when clicking a link in, say, Sudo commands in the Filebeat System] Syslog dashboard ECS dashboard in a space which is not default, I'm greeted by the "The requested dashboard could not be found." error. The setup command does not report any error and all dashboards are imported into my space. It's just the links that are completely broken.

The setup command I use is:

docker run --net="host" docker.elastic.co/beats/filebeat:8.10.4 setup \
  -E setup.kibana.host=... \
  -E setup.kibana.username=... \
  -E setup.kibana.password=... \
  -E setup.kibana.space.id=admins \
  --dashboards

The only space where the imported dashboards work as advertised is in the "Default" space. The link in the respective space in this case is:

Default: .../app/dashboards#/dashboard/277876d0-fa2c-11e6-bbd3-29c986c96e5a-ecs
Admins: .../s/admins/app/dashboards#/dashboard/277876d0-fa2c-11e6-bbd3-29c986c96e5a-ecs

Admins is the space I'm trying to setup the dashboards in.

What am I doing wrong here?

2

Hi @litsegaard

Try Running without the --dashboards if you do this only the dashboards get loaded to that space there are other assets that are needed... specifically the template and dataview that are needed for the dashboards.

In general I always recommend just running

setup -e without any other args, that way all the needed assets are loaded.

When I ran as you did... same issues... when I ran without the --dashboards i.e. load all assets it worked.

Screenshot 2023-11-12 at 7.50.24 AM
Screenshot 2023-11-12 at 7.50.24 AM3342×1730 450 KB

3

Hi Stephen!

First many thanks for your reply. However, setup -e did not solve the issue for me. All index templates, the data view, index templates and dashboards+viz were successfully imported but the original error remains the same. All of the imported objects were correctly imported into the space ("admins" in this case) BUT as soon as I click 'Sudo commands' for example in the syslog dashboard (admins space) I get an angry "The requested dashboard could not be found." error.

I currently have six spaces for various tenants but only "Default" is working.

There must be something else going on here...

4

Sorry confused. Did you take out the --dashboards

docker run --net="host" docker.elastic.co/beats/filebeat:8.10.4 setup \
  -E setup.kibana.host=... \
  -E setup.kibana.username=... \
  -E setup.kibana.password=... \
  -E setup.kibana.space.id=admins

Does the user you are running setup as have permission to write to this spaces?

I don't have an answer for you as I cannot repeat what you're seeing on my side because it's working fine.

5

Hi Stephen!

(This is me as well - just another account :grinning: )

Affirmative: I just executed 'setup -e' using the 'elastic' super user. It's correct that other users have very limited access to the other spaces as each tenant has a specific role, but I thought the 'elastic' user still had full access to the search/kibana functionality. Everything seems to be imported (dashboards, visualizations, index templates, data view, pipelines...) no errors there, but even if I login using the 'elastic' user and select any other space but 'Default' I have these issues. It is very odd.

Hmmmm, may I ask which user you are using for the beats setup? Are you using, say, a 'filebeat/audit/packet...-setup' (filebeat-setup, auditbeat-setup...) user or are you using 'elastic' as me? I thought (wrongly?) that 'elastic' could do whatever in the cluster having all as a cluster priv but this might be a wrong reasoning by me? The dashboards/visualizations etc are indeed installed in the respective space - it's just the links between dashboards that fail if I'm in a space which is not 'Default'.

I hope this makes sense...

7

Hi Stephen!

Thanks for helping me sort this out!

Allow me to describe what I've done for one of my tenants that fails to work as intended and let's focus on the Filebeat integration:

I've created the following roles (pseudo code):

filebeat_setup
  cluster privs: [ monitor, manage_ilm ]
  indices privs: [ filebeat-* : manage ]

filebeat_reader
  cluster privs: [monitor]
  indices privs: [ filebeat-* : read ]

filebeat_writer
  cluster privs: [monitor, read_ilm, read_pipeline ]
  indices privs: [ filebeat-* : create_doc ]

I have established one role/tenant and a sample tenant role looks like this:

role "tenant x" having the following privileges:

tenant x
  cluster privs: none
  indices privs: [ filebeat-* : read, "tenant indices": all ]
  tenant x space: [
    Discover : all,
    Dashboard: all,
    Visualize: all
  ]

Finally, I have a tenant user x with the following privileges:

Privileges: tenant x, filebeat_reader

I use filebeat_setup to setup all Filebeat related artefacts in my cluster. Although all artefacts are imported I'm unable to follow dashboards links. A click on the "Raw Flow Records" in the "Filebeat Net Flow" dashboard, for example, gives a 404 error. If I look in the saved objects section I can see all imported dashboards and visualizations but they're not "clickable"by my tenant user x.

There must be a Space feature and/or privilege I'm missiong somewhere but I'm out of luck identifying which... :frowning:

8

What happens if you log in as elastic super user then go to that tenant space do the dashboards work?

9

I still experience problems. Let me give you some additional info here.

The initial (Filebeat setup in this case) has been done like this:

docker run --net="host" \
 -v .../ca.crt:/usr/share/filebeat/ca.crt \
 docker.elastic.co/beats/filebeat:8.10.4 \
 setup -e  \
 -E setup.kibana.username=filebeat_setup \
 -E setup.kibana.password=... \
 -E setup.kibana.host="http://localhost:5601" \
 -E output.elasticsearch.username=filebeat_setup \
 -E output.elasticsearch.password=... \
 -E output.elasticsearch.hosts=["https://localhost:9200"] \
 -E output.elasticsearch.ssl.certificate_authorities=["/usr/share/filebeat/ca.crt"]

This command seems to work fine and I get no errors. After this I issue the following for the other tenants:

docker run --net="host" \
 -v .../ca.crt:/usr/share/filebeat/ca.crt \
 docker.elastic.co/beats/filebeat:8.10.4 \
 setup --dashboards  \
 -E setup.kibana.username=filebeat_setup \
 -E setup.kibana.password=... \
 -E setup.kibana.host="http://localhost:5601" \
 -E setup.kibana.space.id=<tenant x> \
 -E output.elasticsearch.username=filebeat_setup \
 -E output.elasticsearch.password=... \
 -E output.elasticsearch.hosts=["https://localhost:9200"] \
 -E output.elasticsearch.ssl.certificate_authorities=["/usr/share/filebeat/ca.crt"]

Again the command executes without any errors. Now, if log in as a user of the "admins" role I may access (say the Filebeat Syslog) dashboard and click the links. However, if i switch to the "tenant x" space, select the same dashboard and then click the links, I get a 404 error.

If I log in as a member of the "tenant x" space, go to the same dashboard as the above, I can open the dashboard but clicking any of the links result int oa 404 error.

I think this shows that I must have done something wrong in my (ansible) setup.
The members of the "admins" role have all cluter privileges and the following Kibana features enabled for the "admins" space (sorry for the long yaml layout):

          kibana:
             features:
              - feature: enterpriseSearch
                privileges:
                  - all
              - feature: discover
                privileges:
                  - all
              - feature: dashboard
                privileges:
                  - all
              - feature: canvas
                privileges:
                  - all
              - feature: maps
                privileges:
                  - all
              - feature: ml
                privileges:
                  - all
              - feature: logs
                privileges:
                  - all
              - feature: graph
                privileges:
                  - all
              - feature: visualize
                privileges:
                  - all
              - feature: infrastructure
                privileges:
                  - all
              - feature: apm
                privileges:
                  - all
              - feature: uptime
                privileges:
                  - all
              - feature: siem
                privileges:
                  - all
              - feature: securitySolutionCases
                privileges:
                  - all
              - feature: slo
                privileges:
                  - all
              - feature: dev_tools
                privileges:
                  - all
              - feature: advancedSettings
                privileges:
                  - all
              - feature: indexPatterns
                privileges:
                  - all
              - feature: fileManagement
                privileges:
                  - all
              - feature: filesSharedImage
                privileges:
                  - all
              - feature: savedObjectsManagement
                privileges:
                  - all   
              - feature: savedObjectsTagging
                privileges:
                  - all
              - feature: osquery
                privileges:
                  - none
              - feature: actions
                privileges:
                  - all
              - feature: generalCases
                privileges:
                  - all
              - feature: guidedOnboardingFeature
                privileges:
                  - all
              - feature: monitoring
                privileges:
                  - all
                  - monitoring_user
             spaces:
              - space: admins

The "tenant x" users have no cluster privileges and the following Kibana features:

          kibana:
            features:
              - feature: enterpriseSearch
                privileges:
                  - none
              - feature: discover
                privileges:
                  - all
              - feature: dashboard
                privileges:
                  - all
              - feature: canvas
                privileges:
                  - none
              - feature: maps
                privileges:
                  - none
              - feature: ml
                privileges:
                  - none
              - feature: logs
                privileges:
                  - none
              - feature: graph
                privileges:
                  - none
              - feature: visualize
                privileges:
                  - all
              - feature: infrastructure
                privileges:
                  - none
              - feature: apm
                privileges:
                  - none
              - feature: uptime
                privileges:
                  - none
              - feature: siem
                privileges:
                  - none
              - feature: securitySolutionCases
                privileges:
                  - none
              - feature: slo
                privileges:
                  - none
              - feature: dev_tools
                privileges:
                  - none
              - feature: advancedSettings
                privileges:
                  - none
              - feature: indexPatterns
                privileges:
                  - all
              - feature: fileManagement
                privileges:
                  - none
              - feature: filesSharedImage
                privileges:
                  - none
              - feature: savedObjectsManagement
                privileges:
                  - none   
              - feature: savedObjectsTagging
                privileges:
                  - none
              - feature: osquery
                privileges:
                  - none
              - feature: actions
                privileges:
                  - none
              - feature: generalCases
                privileges:
                  - none
              - feature: guidedOnboardingFeature
                privileges:
                  - none
            spaces:
              - space: <tenant x>

Currently I'm out of luck understanding what's going on "under the hood" but it feels like I must have missed something in the Kibana sections above or a cluster/kibana privilege.

Again, many thanks for looking into this!

10

I can't debug all your combinations...

If I were you I run this against each space / tenant

docker run --net="host" \
 -v .../ca.crt:/usr/share/filebeat/ca.crt \
 docker.elastic.co/beats/filebeat:8.10.4 \
 setup -e  \
 -E setup.kibana.username=filebeat_setup \
 -E setup.kibana.password=... \
 -E setup.kibana.host="http://localhost:5601" \
 -E output.elasticsearch.username=filebeat_setup \
 -E output.elasticsearch.password=... \
 -E setup.kibana.space.id=<tenant x> \ <!----- Added
 -E output.elasticsearch.hosts=["https://localhost:9200"] \
 -E output.elasticsearch.ssl.certificate_authorities=["/usr/share/filebeat/ca.crt"]

Then log into the System as the elastic users and see if the Dashboards works for each of the spaces / tenants...

Confirm that...

then we can work on getting the privileges correct for each tenant.

Users directed to each tenant / space should be based on the role is directed to their space it should work, I am not really clear on all the controls you are trying to do...

When I read this ... this is typically the privileges for the logged in user... again if you login as elastic user do the dashboards work in each space?

Ohh something just occurred to me.....

Pretty sure you're missing the Data Views which are saved Objects and are NOT loaded with --dashboards

Dashboards work off data views not indices... Check in your tenant spaces if you have Data Views.

11

Fully understand you're not able to work out all permutations :slight_smile: Ok, I followed your instructions and executed the "setup -e" for all tenants, then logged on as the 'elastic' user but the error remains for the "tenant x" spaces but it works in the "admins" tenant so there's definitely a difference with regards to the "admins" and "tenant x" spaces.

What I'm trying to accomplish is to "lock down" everything except the Dashboards (including Visualizations) and Discover modules. The users of those tenants are not allowed to do anything except searchin and discovering in the underlying indexed data - nothing else and this is the reason I'm trying to lock them out from all the other bells and whistles.

Regarding the data views they're there for 'tenant x' - nspecting the "filebeat-*" data view I can see 7290 fields defined together with index pattern and time field so it seems to have been added.

I did some analysis of the difference between the admins/tenant spaces and there's difference in the indices privs between the two:

admins: filebeat-* all
tenant x: filebeat-* read

Hmmmmm.... could this be the culprit here?

12

UPDATE

I added the 'all' privilege on the filebeat-* indices to the 'tenant x' roles but the problem remains :frowning: I'm very reluctant giving the 'tenant x' roles more privileges than necessary as there's a reason they're locked down. I really hope we find a solution to this...

13

I have created many *read" only spaces / roles but I copied the dashboard and dataviews from the default space not trying to load the with filebeat

Not sure your issue... Recommend you try that then work back

14

I did the following:

  • removed all previously imported objects (setup -e) from 'tenant x'
  • went to 'Default'->Saved Objects
  • Selected Copy to Spaces for dashboard 'Syslog logs [Filebeat System] ECS
  • ('Check for existing objects' selected - Overwite!)
  • Copied to 'tenant x'
  • Verified that 'tenant x' has access to 'filebeat-*' data view: yes
  • Logged on as 'tenant x' user and 'elastic'
  • 404 error in both cases

UPDATE

  • (Decided to leave my ansible-road for a while and perform the steps manually)
  • Logged on as 'elastic'
  • Created a new space 'foo' with dashboard, discover, visualization features, saved objects
  • Again copied the syslogfilebeat dashboard (with dependencies)
  • Selected 'foo' as 'elastic' user and selected syslog dashboard
  • 404 error as well

FYI I'm running the Elastic stack in docker (compose) and haven't experienced any other problem apart from this... (so far)

15

Hi @petlit2049 sorry...Not sure what to tell you ... You have, Something else going on...

There is some restriction or privileges issue... As you can see above I simply created a space ran the setup and or I copied the assets.. made sure space/ rolled hass access to the index and it works... As it has for me many times... There is fundamental issue with the setup but I don't see it.

16

Hmmmm... in the freshly created 'foo' space->'Saved Objects' I can see some 'Advanced Settings [8.10.2]' and a 'Global Settings [8.10.2]'. can threy contain any strange settings that could have these effects?

17

Just for reference - which privileges do you apply for, say, the filebeat_setup/reader users and for the users interacting with your imported dashboard(s)?

18

I would forget about all the filebeat setup stuff until you get the basics to.work...

It's late here if I get a chance tomorrow I will do the step by step setup for a read only user role dashboard space

Some how I think you are missing a simple step or over complicating it not sure

19

Believe my Stephen, I really, no really, appreciate your efforts here. Just FYI I just went the export/import (Saved Objects) route to see if there was an issue with the copy route but, sorry to say, export/import resulted in the same error: 404. Looking forward to your findings... :slight_smile:

21

Exactly where how 404

Please show where 404

Show the entire URL etc

Have you cleared all your cookies have you tried in an incognito window ...

22

Hi Stephen! Sorry for not getting back to you - been "swamped". However, I'll look into this during the weekend. Have a nice weekend!