Permissions for beats

David_Fuge · February 14, 2020, 7:00pm

I've read the documentation about accounts for beats and securing beats. However, I still dont understand how to properly limit the accounts used to authenticate my winlogbeat and metricbeats back to my cluster.

I am running on the elastic cloud version, so the built in users aren't available.

Am I just missing a list of the permissions required for beats to ship the data to the cluster? I hate having a super user account enabled on these shippers just to get them to report. However, I can find no good concise list of the correct permissions that are required by the Beats accounts. Where can I look to find these, also why is there not just a standard "beats permissions role" that just allows these beats to ship log data and metric data back to the cluster?

DanRoscigno · February 14, 2020, 10:06pm

Hi David,
It boils down to these steps:
Create the roles as specified in the docs. Here is a screenshot of the setup role I created for Filebeat, Metricbeat, and Heartbeat (I don't have a Windows box, so I can't show that one):

I only show Filebeat and Metricbeat as Heartbeat scrolled off the page.

Here is the writer role:

Here is the setup user:

and the writer user:

Based on the Winlogbeat docs, the above roles should just work if you click on the Add Index Priv and add a line for winlogbeat-* with the privs from https://www.elastic.co/guide/en/beats/winlogbeat/current/feature-roles.html

If you get stuck send the command and error

DanRoscigno · February 14, 2020, 10:06pm

@David_Fuge

If you prefer JSON over the UI this should do it for Metricbeat:

PUT /_security/role/beats_writer
{
  "cluster": [
    "monitor",
    "cluster:admin/ingest/pipeline/get"
  ],
  "indices": [
    {
      "names": [
        "filebeat-*"
      ],
      "privileges": [
        "create_doc"
      ],
      "field_security": {
        "grant": [
          "*"
        ],
        "except": []
      },
      "allow_restricted_indices": false
    },
    {
      "names": [
        "metricbeat-*"
      ],
      "privileges": [
        "create_doc"
      ],
      "field_security": {
        "grant": [
          "*"
        ]
      },
      "allow_restricted_indices": false
    },
    {
      "names": [
        "heartbeat-*"
      ],
      "privileges": [
        "create_doc"
      ],
      "field_security": {
        "grant": [
          "*"
        ]
      },
      "allow_restricted_indices": false
    }
  ],
  "applications": [],
  "run_as": [],
  "metadata": {},
  "transient_metadata": {
    "enabled": true
  }
}

PUT /_security/role/beats_setup
{
  "cluster": [
    "monitor",
    "manage_ilm",
    "manage_ml"
  ],
  "indices": [
    {
      "names": [
        "filebeat-*"
      ],
      "privileges": [
        "manage",
        "read"
      ],
      "field_security": {
        "grant": [
          "*"
        ],
        "except": []
      },
      "allow_restricted_indices": false
    },
    {
      "names": [
        "metricbeat-*"
      ],
      "privileges": [
        "manage"
      ],
      "field_security": {
        "grant": [
          "*"
        ]
      },
      "allow_restricted_indices": false
    },
    {
      "names": [
        "heartbeat-*"
      ],
      "privileges": [
        "manage"
      ],
      "field_security": {
        "grant": [
          "*"
        ]
      },
      "allow_restricted_indices": false
    }
  ],
  "applications": [],
  "run_as": [],
  "metadata": {},
  "transient_metadata": {
    "enabled": true
  }
}


PUT /_security/user/beat-setup
{
  "username": "beat-setup",
  "roles": [
    "beats_setup",
    "kibana_user",
    "ingest_admin",
    "beats_admin"
  ],
  "full_name": "Beat setup user",
  "email": "",
  "metadata": {},
  "enabled": true,
  "password": "a-really-good-password-for-setup"
}

PUT /_security/user/beat-writer
{
  "username": "beat-writer",
  "roles": [
    "beats_writer"
  ],
  "full_name": "Beat writer",
  "email": "",
  "metadata": {},
  "enabled": true,
  "password": "a-really-good-password-for-writer"
}

DanRoscigno · February 14, 2020, 10:42pm

I always use the keystore, here are the Winlogbeat docs:

https://www.elastic.co/guide/en/beats/winlogbeat/current/keystore.html

I am showing the commands I used for Metricbeat, it looks like the Winlogbeat syntax is same.

metricbeat keystore create

metricbeat keystore add ES_SETUP_USER

metricbeat keystore add ES_SETUP_PASSWORD

metricbeat keystore add ES_WRITER_USER

metricbeat keystore add ES_WRITER_PASSWORD

And the setup command:

./metricbeat setup -E setup.ilm.check_exists="true" \
-E cloud.id=BeatsPerms:dXMtXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXZTc2OTZkMzRjMDk4NDhhOTk3YjNkNGUxYWZmJDVmNzU2NGMyMzRhZDQyZDZiNGFiMmNhNDA1ZDEyY2Jk \
-E cloud.auth=\${ES_SETUP_USER}:\${ES_SETUP_PASSWORD}

And the run command:

./metricbeat -E setup.ilm.check_exists="false" \
-E cloud.id=BeatsPerms:dXMtXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXZTc2OTZkMzRjMDk4NDhhOTk3YjNkNGUxYWZmJDVmNzU2NGMyMzRhZDQyZDZiNGFiMmNhNDA1ZDEyY2Jk \
-E cloud.auth=\${ES_WRITER_USER}:\${ES_WRITER_PASSWORD}

David_Fuge · February 15, 2020, 2:54pm

Thanks Dan,

So are you setting up two roles, one to set up the index when the beat is installed for the first time, and another that just ships/writes the data to the index?

I read the documentation but honestly I was a little confused by the permissions and the caveats for cloud vs roll your own. The verbiage gets a little dense, your clarification was great. thank you.

As far as the keystore goes, I dont understand the value of the keystore, if I can just view and change the keystore values with any account? Is it just so the credential is not stored in plain view?

Lastly, in your run commands it's looking like you're invoking some index life cycle management command, is that needed to use the commands to call a key in the keystore (I have ilm set up through kibana).

Thanks again for all your help,

David

David_Fuge · February 15, 2020, 3:07pm

@DanRoscigno forgot to tag you in the last post.

Here is where I get confused in the documentation:

"Administrators who set up Winlogbeat typically need to load mappings, dashboards, and other objects used to index data into Elasticsearch and visualize it in Kibana."

Does this pertain to accounts that are not used by users. Like, I am not delegating this task to a different person, I just want to use a service account to do this function. Is that what this is referencing?

Does that mean that the steps below that are saying that if I want to create a service account for this I need to give those permissions to the role so that the winlogbeat index can be created on the cluster and set up the kibana dashboards?

Further down it says, Grant roles needed for monitoring. What is monitoring? Does that mean sending the log data from the device to the cluster? Or is that for publishing?

It may seem like im being intentionally obtuse but truly I'm not. The wording in the documentation is a little confusing to me. More confusing are these little nuggets entered into the documentation:

"When using ILM, turn off the ILM setup check in the Winlogbeat config file before running Winlogbeat to publish events:"

Huh? I set up lifecycle policies in Kibana, but now I have to disable a startup check. This is confusing, will this break if I don't include this command, what is this doing exactly?

I truly do appreciate your help, this is just confusing for me.

Thanks again,

David

DanRoscigno · February 15, 2020, 4:01pm

I understand why you are asking, I come from ops and I get it; you have to know exactly what is going on so you know what to expect. The setup command creates the index pattern, ingest pipelines (parsing and enrichment), dashboards, machine learning, etc.

Because it takes elevated privs to do all of the above we decided on two roles.

The writer role can only add to an existing index. This is why that priv gets set to false, as ILM is already setup.

Ignore the monitoring role and the reader roles. You don't need to do anything with them, they are reference material.

DanRoscigno · February 15, 2020, 4:02pm

More later, headed out @David_Fuge

David_Fuge · February 15, 2020, 5:03pm

Thank you, @DanRoscigno !! I appreciate it!

DanRoscigno · February 15, 2020, 6:06pm

@David_Fuge

Yup, This is all about the service accounts, which use the roles I detailed earlier

Yeah, confusing. There are docs that go deep into the details on these options, I would suggest getting things working with the instructions I provided and then read these if you want more info:

Question for you: you said that you configured ILM in Kibana. Did you create your own policy by hand? If so then I need to update the command that I gave you for setup, as you do not want the default ILM policies to be created. Probably you will need to edit your .yml files to specify that you want to use a specific policy that already exists. I will have to read up on this as I have not used a policy other than the default! I always:

Run setup and let it create the default policy
Modify the policy to point to my warm nodes and set the rollover that I want
Start my Beats

Another question for you: You mentioned differences between cloud and self managed. Other than using cloud.id and cloud.auth, did you see mention of other differences? I never do anything differently.

David_Fuge · February 16, 2020, 12:23am

@DanRoscigno Thank you again, this is helpful. So as far as ILM goes, I did exactly as you did, I let it create it’s own policy, then edited it in Kibana after the fact for warm/cold/delete.

As far as cloud vs self managed, there are built in accounts in self managed, where as the documentation says they are not in cloud. I think I got confused on that as some of the permissions overlap with the user account names used in self managed.

Super glad you were here to help! I’m feeling more confident about the process now. I’m gonna give it a try tomorrow night for both my win and metric beats.

@DanRoscigno you are the man!

DanRoscigno · February 16, 2020, 1:09am

Ah, I need to read those docs @David_Fuge. Technically the docs are correct, here are the built in accounts I see in a cluster I have running in Docker on my laptop:

elastic
kibana
logstash_system
beats_system
apm_system
remote_monitoring_user

But,
none of these are related to the users needed for Beats setup or publishing.

I will ping you Tuesday to check in on you.

Have fun!

DanRoscigno · February 18, 2020, 11:32pm

@David_Fuge, how are you doing with this?

David_Fuge · February 19, 2020, 1:27am

Hey @DanRoscigno, it's going well. Metricbeat and Winlogbeat successfully deployed using those permissions. I did have an issue with the keystore not working - it wouldn't auth, but the hard coded limited accounts did work out fine. I'm sure I'll start tackling the keystore stuff here shortly. Thanks again for your help.

system · March 18, 2020, 3:27am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.