Elastic/fleet-server cannot create/write to index - insufficient permissions

Elastic Stack Elasticsearch
1

Hi, I am trying to log a variety of custom metrics to Elasticsearch through TCP, each in their own index.
Until now I was working with a custom deployment of the ELK stack, and simply sending these metrics in JSON form through TCP, received by a Logstash pipeline that outputted to the correct ES index, depending on a name field in the message payload. The pipeline looked something like:

input { tcp { port => 1234 } }
filter { json { source => "message" } }
output {
    elasticsearch {
        hosts => "IP:9000"
        index => "%{[name]}"
    }
}

and messages something like:

{
    "name": "some-metric",
    "message": {
        "field1": "123",
        "field2": "456",
    }
}

^ This is for instance getting routed to a some-metric index.

This seemed to work pretty well, but since then I switched to using a hosted Elastic Cloud instance, and the "Custom TCP Logs" integration. By default, this logs everything under 1 data stream, so I figured the way to go is to define an ingest pipeline which does what my previous logstash pipeline was doing, and make the Custom TCP Logs integration use it. My ingest pipeline is composed of 2 processors like so:

[
  {
    "json": {
      "field": "message",
      "add_to_root": true
    }
  },
  {
    "set": {
      "field": "_index",
      "value": "{{{name}}}",
      "media_type": "text/plain"
    }
  }
]

The last part is the best I could come up with, to achieve the injection into the correct index depending on the payload name. However all events get dropped, due to insufficient permissions for the user elastic/fleet-server:

 Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=403): {"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for API key id [XXXXXXXX] of user [elastic/fleet-server] on indices [some-metric], this action is granted by the index privileges [auto_configure,create_index,manage,all]"}, dropping event!

and I have not been able to find this user, or a place where I can modify its permissions.

My questions are:

  1. Is there a way to allow that user to create/write to these indices?
  2. Am I going about this correctly? Is there a more straighforward way to achieve my goal?

Thanks in advance

2

No, it is not possible. The fleet-server service account is only able to write to indices that are managed by fleet, and there is no way to change that.

You're definitely fighting against what Fleet/Agent tries to do.
It is designed to ingest into curated index names, and not be a universal data router in the way that Logstash is often used.
I don't understand why you want to have your logs routed like that, but it's not really a scenario fleet & agent are designed for.

The best suggestion I can make is to change your pipeline to something like:

    "set": {
      "field": "_index",
      "value": "logs-{{{name}}}",
      "media_type": "text/plain"
    }

It's still likely to violate some of the assumptions fleet makes, but it's going to get you closer.

3

Thanks a lot for the answer and suggestion, at least I know what to/not to dig into.

I guess at this point I am just not familiar enough with the concepts to know what the idiomatic way is. All in all, I am looking for a solution on Elastic Cloud that allows simultaneously:

  • easy and secure ingest of many different metrics from many different clients: here the "Custom TCP log" allows for that, and easy enrollment
  • availability of these metrics in dedicated index patterns in Kibana: each of these metrics have a dedicated index mapping, and I can only think of routing these logs to them somehow through the above ingest pipeline. Maybe I am just not using the correct integration, or misusing index mapping/patterns?

Do you have an alternative that sounds more natural?

4

Update: This fails with the same type of permission error, for the indices:admin/auto_create action.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.