Need to define a role for Filebet and metricbeat for writing to elasticsearch

Elastic Stack Beats
, ,
1

I am running v7.7 of the stack from a ZIP install on Windows.
But I have the same issue with v7.12 running as docker container on a Mac.

I've searched and searched and either I don't know what to search for, or I don't recognize the answer when I see it.

I have a pair of ES nodes, one for ingest, hot, data the other for warm when using ILM. I am sending data from a filebeat and a metricbeat instance. I can create user, use it in the filebeat.yml and give is some pretty open privileges (like superuser) and it writes the data just fine. But there must be a better defined role for doing this without such open access. So, the question is, what are the best role privileges to assign a user for filebeat and for metricbeat?

Thanks for the help!

2

Did you see this?

I kinda like the API method nice example here, defines in detail what privileges filebeat needs.

You can add multiple roles to the API key like monitor and writer so it can be used for both.

3

I did. I followed the part for "Grant privileges and roles needed for publishing" (I assumed Filebeat was 'publishing' log data to ES) and I do not ingest any data. This is what prompted my question here.

4

Ahh... Perhaps provide a the code of the role you defined and did you look to see if there are any errors in the filebeat log?

I just used the API key here pretty much verbatim and it worked for writing

POST /_security/api_key
{
  "name": "filebeat_host001", 
  "role_descriptors": {
    "filebeat_writer": { 
      "cluster": ["monitor", "read_ilm"],
      "index": [
        {
          "names": ["filebeat-*"],
          "privileges": ["view_index_metadata", "create_doc"]
        }
      ]
    }
  }
}

If you changes any of the index names etc... that will matter.

Pretty sure this will work for writing and monitoring...

POST /_security/api_key
{
  "name": "filebeat_host001",
  "role_descriptors": {
    "filebeat_writer": {
      "cluster": [
        "monitor",
        "read_ilm"
      ],
      "index": [
        {
          "names": [
            "filebeat-*"
          ],
          "privileges": [
            "view_index_metadata",
            "create_doc"
          ]
        }
      ]
    },
    "filebeat_monitoring": {
      "cluster": [
        "monitor"
      ],
      "index": [
        {
          "names": [
            ".monitoring-beats-*"
          ],
          "privileges": [
            "create_index",
            "create"
          ]
        }
      ]
    }
  }
}
5

We don't have TLS/HTTPS, so no API keys for us

This is my Role:

{
      "fb_svc_new" : {
        "cluster" : [
          "monitor",
          "read_ilm"
        ],
        "indices" : [
          {
            "names" : [
              "filebeat-*"
            ],
            "privileges" : [
              "create_doc",
              "view_index_metadata"
            ],
            "allow_restricted_indices" : false
          }
        ],
        "applications" : [ ],
        "run_as" : [ ],
        "metadata" : { },
        "transient_metadata" : {
          "enabled" : true
        }
      }
    }

and, I do have an ERROR in the filebeat log

2021-05-09T12:20:45.661-0500	ERROR	[publisher_pipeline_output]	pipeline/output.go:106	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: failed to create alias: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/create] is unauthorized for user [fb_service]"}],"type":"security_exception","reason":"action [indices:admin/create] is unauthorized for user [fb_service]"},"status":403}: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/create] is unauthorized for user [fb_service]"}],"type":"security_exception","reason":"action [indices:admin/create] is unauthorized for user [fb_service]"},"status":403}

This seems to indicate I need more privileges?

6

Looks like perhaps you need the setup privileges as well not sure if you already ran setup separately.

Looks like it's trying to create the alias

7

ok, adding the setup privileges to the publisher privileges is working. So, to recap.. my role now looks like this:

  "fb_svc_new" : {
    "cluster" : [
      "monitor",
      "read_ilm"
    ],
    "indices" : [
      {
        "names" : [
          "filebeat-*"
        ],
        "privileges" : [
          "create_doc",
          "view_index_metadata",
          "create_index",
          "create",
          "manage",
          "read",
          "write"
        ],
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [ ],
    "run_as" : [ ],
    "metadata" : { },
    "transient_metadata" : {
      "enabled" : true
    }
  }
}

Correct?

8

Well if that works good, I guess I would have expected manage_ilm as defined by the docs I referenced above for setup, but if read_ilm works then good.

9

Oh.. I missed that in the doc. Probably need to make that change as well.

Thanks for all the help!

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.