Hi, I am trying to log a variety of custom metrics to Elasticsearch through TCP, each in their own index.
Until now I was working with a custom deployment of the ELK stack, and simply sending these metrics in JSON form through TCP, received by a Logstash pipeline that outputted to the correct ES index, depending on a name field in the message payload. The pipeline looked something like:
input { tcp { port => 1234 } }
filter { json { source => "message" } }
output {
elasticsearch {
hosts => "IP:9000"
index => "%{[name]}"
}
}
and messages something like:
{
"name": "some-metric",
"message": {
"field1": "123",
"field2": "456",
}
}
^ This is for instance getting routed to a some-metric index.
This seemed to work pretty well, but since then I switched to using a hosted Elastic Cloud instance, and the "Custom TCP Logs" integration. By default, this logs everything under 1 data stream, so I figured the way to go is to define an ingest pipeline which does what my previous logstash pipeline was doing, and make the Custom TCP Logs integration use it. My ingest pipeline is composed of 2 processors like so:
[
{
"json": {
"field": "message",
"add_to_root": true
}
},
{
"set": {
"field": "_index",
"value": "{{{name}}}",
"media_type": "text/plain"
}
}
]
The last part is the best I could come up with, to achieve the injection into the correct index depending on the payload name. However all events get dropped, due to insufficient permissions for the user elastic/fleet-server:
Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=403): {"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for API key id [XXXXXXXX] of user [elastic/fleet-server] on indices [some-metric], this action is granted by the index privileges [auto_configure,create_index,manage,all]"}, dropping event!
and I have not been able to find this user, or a place where I can modify its permissions.
My questions are:
- Is there a way to allow that user to create/write to these indices?
- Am I going about this correctly? Is there a more straighforward way to achieve my goal?
Thanks in advance