I'm playing around with the new Elastic agent (7.9) and using the Ingest Manager BETA. Created several agent configs with several integrations and can them easily re-assign to an agent. Standalone the configuration goes well when using a self-signed certificate (insecure session) on my ECK sandbox (NodePort without cert-manager), but when using fleet management I couldn't figure out how to add this additional setting to disable certifcate_verification (ssl.verification_mode: none).
Is there a way how to pass these custom settings from Ingest Manager UI or Ingest Manager configuration, besides the kibana and elasticsearch urls ?
I already seen that it creates an action_store.yml file, which contains the pushed configuration, but this need to be updated. If I update this file by hand and restart the elastic-agent it works and data is coming into elasticsearch. But this is not a long-term solution.
Would be great that you can still "advanced" edit the agent configuration that is pushed or just set some advanced parameters. For example when enrolling the elastic agent I can ignore self-signed certificates (allow insecure sessions) with the -i option.
easiest method i found was that instead of disabling tls checking, put your self-signed ca certificate in the clients' trusted ca store (e.g. /etc/ssl/certs).
i think certificate management absolutely has to be part of the config push system - there seems to be talk of that on github - whether its just telling the beats which file to check for client/server/ca keys or actually sending new certs.
i seem to have over-estimated the agent config deployment setup as i can't find any way to define and push a complete filebeat.yml from ingest-manager to agents.
i'd like to see a step-by-step install process that elastic employees are using to install/configure this stuff, as the publicly documented process simply doesn't work and if employees can't reproduce issues due to using a different process, its pointless doing a beta. i mean where is it written in big letters "forget doing this on self-signed certs"? how many bug reports did that cause?
the first beta rpm didn't even work, that being a surprise to employees is really worrying, but then they only seem to test on macos and docker not all the other platforms you support. got no CI?
If you do not want to worry about certificate management on your end, I recommend you to use Elastic Cloud as there on the Stack side all is set up for you.
For the case of pushing a complete filebeat.yml from Ingest Manager, this is not supported and not planned to be supported as the config format from Filebeat to Agent sligthly changed. There are ideas around pushing partially raw agent.yml files in the future. If you miss a feature here, feel free to file a feature request on Github.
Thanks for the update. cloud isn't an option for me.
So what are agent configs for if not for configuring the stuff that's usually in yaml files? it seems to just be to configure which integrations are enabled?
If you are running the Elastic Agent in standalone mode, you can use any input configuration you want. If you run in management mode with Fleet at the moment you can only use the predefined integrations and the log integration to tail any log file. There is definitively more to come also in managed mode, but that is what is available in our first beta.
That is also what I expect from it. Fleet management mode is mostly interesting when endpoint protection / SIEM is needed on large-scale ( mainly windows desktop/server some linux) environments and maintained/controlled by the security department (say SOC). Here the management of all integrations and endpoints is important from a functional view. Also smaller/simpler setups can profit from the fleet management mode. Of Course you can make more predefined integrations available for user-experience, but the question is do you want to solve all external management problems with elastic ? To be honest then you get more and more into the area of big all-in-one enterprise suites
In larger / enterprise environments I still see the need for a config management / orchestration tool (like Ansible or containers(K8S) using ArgoCD) that will deploy both software and configuration with finegrain configuration that is part of the application deployment from a GitOps perspective . Which in these cases have a more technical view on their needs of management like DevOps teams. Agents that are running in standalone management mode are still visible (for visibility) in the Ingress Management UI I think ?
Still think that these features are helpful to support both simple and advanced rollouts.
@arnold79 At the moment agent in standalone don't show up in the UI, but agree it would be nice if they can. This is definitively an interesting idea that it would connect for visibility and probably monitoring but all the management is local done through Ansible and friends. We kind of know about the agents already today if the agents ship data because of the agent.id in the data but we don't make anything of it yet.
Maybe somewhat offtopic, but please dont forget to build in some kind of fail-safe for whenever a certificate expires (this can be ca certificate or client certificate). We had this already happen to us that the ca certificate expired and we lost all endpoint agents and had to manually reconfigure all of them to connect again.
Hi @ruflin . Thanks for the response. If you need a Github issue for the agent visibility use case of standalone management mode. Please let me know, then I will create one. Always in to improving a product .
Think most of the responses are off-topic, but still valuable for further development.
@arnold79 An issue for the standalone visibility in Github would be great, so I don't have to file it and we know who to ping for more details Can you then also paste the link here for reference?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
.