What is best way to secure credentials on filebeat configuration? If i put username/password on yml file, anyone customer can login and read them and login into elasticsearch.
Unless your customers have sudo access to the machine just lock down the file permissions of the sensitive files.
they have root access, they own the servers, we just want to collect logs from them but we dont want them to see the credentials and login into kibana. Any idea?
You could use environment variables for username and password or pass them as params during starting filebeat with the -E flag. But I think in both cases, customers could also read the variables as they could check the content of the environment variables or check the command which was run. As long as they are root, there is not really a way around this.
What if you make the Kibana URL not available to them? You could have an index for each customer means if the customer starts to mess around with the data, he only messes around with his own log data?
I see what you mean, but as you said env variables is not very secured and some smart guy can find out. Having an index per customer is not very viable cause we have more than 5.000 customers and we dont want to create an index and role/creds for each of them.
Is there anyway to configure elasticsearch output credentials used by filebeat to just insert data, but that they cant query/delete the elasticsearch data using api or login into kibana with those creds? I dont see this possible when creating a role.
I thought about using logstash and dont expose ES and KB but i have others scripts that collect more info and use NEST library so it cant use logstash and has to connect to elasticsearch directly.
Is not this a problem in other projects where a user can see creds and connect tot the cluster a query all data they want to? How do people solve this?
I need to check if you give someone create and create_index permissions if then creating queries is still possible: https://www.elastic.co/guide/en/shield/current/shield-privileges.html#privileges-list-indices
So far this problem did not pop, at least I didn't see it. The reason is probably that in most cases the machines and the elasticsearch instances are run by the same group, and the "consumers" only need read access to Kibana.
An other option could be to put a queue like Kafka in between and LS fetches the events from there. So you could separate the access. Also putting LS in the middle sounds like a good idea.
What do you mean by the NEST library?
We use NEST/elasticsearch.net to collect info from REST APIs and index it in Elasticsearch. This approach cant use LS in the middle, so it has to connect directly with ES and pass creds.
OK, let me know if those privileges work. But that will require to buy shield license?
Yes, I assume you have Shield as you mentioned above setting rules.
Ah, NEST = .net client 
How do you handle there the credentials?
Now they are hardcoded into a .net binary and we can ofuscate it so they cant decompile and see them, it makes it more difficult.
I guess we will have to use both approach for .net client use direct connection to ES with creds hardcoded into binary and for filebeat use logstash output with certificates and no creds.
As we dont require LS for parsing/enriching logs ( just maybe for date timestamp) i wanted to know if we could simplify and skip LS and just use ingest node for date timestamp but exposing creds maybe a risk.
Or use the bulk es input of logstash for the client (not sure if that works). Currently I don't have more ideas. Anyone else?
Using Shield you can create a role for the Filebeat user that restricts it to the least amount of privileges required.
Filebeat needs to be able to create indexes, write data, and manage index templates. If you assign those privs then data cannot be queried by that user. Additionally if you configure Filebeat to not manage the index templates then you can remove that privilege as well.
If this is an application that is generally available to untrusted users then hardcoding the same credentials into the binary that is being distributed sounds like a bad idea. These credentials will be able to create indexes and write to the cluster which opens you to DoS attacks.
But we need at least write privileges to index new docs in the cluster. So you are still open to DoS attacks if someone can see the restricted Shield creds on filebeat.yml or decompiling the binary, they can run scripts to continue create new fake documentes and DoS the cluster. Is there any solution to avoid this?
It depends a bit on what you are trying to accomplish. You could assign each customer their own unique credentials. This would allow you to revoke a single customer's credentials if they become compromised which will help to limit the scope of a problem if it should occur.
But in order to know that the credentials have been compromised or that someone is doing bad things you would need to be monitoring this service you are providing. You could use Shield auditing and some Watcher queries to help keep an on things.
Depending on what you are building, it might make sense to proxy the traffic through something that can apply rate limits per customer to prevent abuse or to just monitor traffic levels per customer (maybe use Packetbeat behind the proxy to make measurements).
Also you could not grant Filebeat the ability to create indexes if you create them yourself ahead of time (like a cron + curl).
We cant assign each customer a different creds. We have more than 5.000 and they just deploy our software as virtual appliance that comes preconfigured with everything and this is a transparent thing like a telemetry, we can't ask them to customize and edit filebeat.yml to put some new creds, it will also not scale for us to operate in that way.
Also for the NEST binary i dont see other solution than hardconding creds in binary as it doesnt support to send to logstash, kafka or other queues.
How are you attributing metrics and Filebeat data to a specific customer?
with a fields.asset_id. Each appliance has an unique id and we add it as a variable field in the .yml config. This enviroment variable get set when the service start running a command line. So we dont need to customize for each customer.
I would limit the privileges of this user to only allow writes to the current daily index.
- Automate pre-creating the next daily index.
- Automate marking the previous daily index readonly (index.blocks.read_only=true) after some grace period to allow offline/disconnected nodes to report older data.
- Set up some monitoring, rate limiting, IP blacklisting. If it's on the Internet then it's bound to be discovered and abused at some point.
I encourage you to think through what you would do if the credentials encoded in your binary are abused.
If the asset IDs are long/random/private then you could treat it as a shared secret and do away with encoding a static username/password in the binary. Reverse proxy the requests to Shield and validate the asset ID at the proxy. Or implement a custom Shield realm to validate the asset IDs.
Ok. Thanks for the discussion and ideas. If anyone else come up with other ideas feel free to add )
This topic was automatically closed after 21 days. New replies are no longer allowed.