Hi,
I'm trying to test the new Elastic Agent on a ECK deployed stack with security enabled, so Elastic Agent ca work with it.
My yaml files:
elasticsearch.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elasticsearch
spec:
version: 7.9.0
nodeSets:
- name: default
count: 1
config:
node.master: true
node.data: true
node.ingest: true
podTemplate:
metadata:
labels:
app: elasticsearch
spec:
initContainers:
- name: sysctl
securityContext:
privileged: true
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
containers:
- name: elasticsearch
resources:
requests:
memory: 2Gi
cpu: 0.5
limits:
memory: 2Gi
cpu: 1
env:
- name: ES_JAVA_OPTS
value: "-Xms1g -Xmx1g"
http:
service:
spec:
type: LoadBalancer
tls:
selfSignedCertificate:
subjectAltNames:
- ip: 10.11.0.246
- dns: elk-k3s
kibana.yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana
spec:
version: 7.9.0
count: 1
elasticsearchRef:
name: "elasticsearch"
secureSettings:
- secretName: kibana-saved-objects-encrypted-key
http:
service:
spec:
type: LoadBalancer
tls:
selfSignedCertificate:
subjectAltNames:
- ip: 10.11.0.246
- dns: elk-k3s
podTemplate:
metadata:
labels:
app: kibana
spec:
containers:
- name: kibana
resources:
limits:
memory: 2Gi
cpu: 2
This configuration is working right for testing some features, but I have problems testing Elastic Agent.
The Agent Configuration:
The Fleet Enrollment token:
Ingest Manager settings:
Is not clear for me here what are the correct values to use for "Kibana URL" and "Elasticsearch URL" when I'm using a ECK deployed stack. Both ElasticSearch and Kibana are exposed to outside Kubernetes Cluster in their original ports, 9200 and 5601 (the cluster have only one Kubernetes Node), with Security Enabled. And that is a main question for me: I supposed that the Elastic Agent is enrolling it to Kibana Fleet using the Fleet installation token, but how is the Elastic Agent connecting to ElasticSearch and Kibana to send logs, being that ElasticSearch and Kibana need user and password authentication?
I'm registering the Elastic Agent with the "Enroll and Fleet" instructions (with --insecure flag, because the certificate is self signed), in the same host where I have deployed ELK stack with ECK, but outside Kubernetes, in the host directly, installed the deb package, for testing:
elastic-agent enroll --insecure https://10.11.0.246:5601 *bigsecrettokenfromenrollmenttokens*
systemctl enable elastic-agent
systemctl start elastic-agent
After run these tree commands, I see the agent Running for a while, but after aprox a minute, the agent go Offline, even after restarting the service with "systemctl restart elastic-agent" (I see "Offline" in Fleet Kibana page only, even the elastic-agent service at OS level is running):
I don't see any in Datasets:
Running a "journalctl -u elastic-agent" in the OS (Ubuntu 20.04) I see:
Aug 28 18:38:34 elk-k3s elastic-agent[2664321]: 2020-08-28T18:38:34.441Z DEBUG application/fleet_gateway.go:162 FleetGateway is sleeping, next update in 1s
Aug 28 18:38:35 elk-k3s elastic-agent[2664321]: 2020-08-28T18:38:35.463Z DEBUG application/fleet_gateway.go:142 FleetGateway calling Checkin API
Aug 28 18:38:35 elk-k3s elastic-agent[2664321]: 2020-08-28T18:38:35.468Z DEBUG kibana/client.go:170 Request method: POST, path: /api/ingest_manager/fleet/agents/529228ed-7c5b-437c-8330-4a471d7b2ca2/checkin
root@elk-k3s:/var/log/elastic-agent# tail elasticagent.txt
Aug 28 18:38:34 elk-k3s elastic-agent[2664321]: 2020-08-28T18:38:34.118Z DEBUG kibana/client.go:170 Request method: POST, path: /api/ingest_manager/fleet/agents/529228ed-7c5b-437c-8330-4a471d7b2ca2/checkin
Aug 28 18:38:34 elk-k3s elastic-agent[2664321]: 2020-08-28T18:38:34.431Z DEBUG application/action_dispatcher.go:81 Dispatch 1 actions of types: *fleetapi.ActionConfigChange
Aug 28 18:38:34 elk-k3s elastic-agent[2664321]: 2020-08-28T18:38:34.432Z DEBUG application/handler_action_policy_change.go:23 handlerConfigChange: action 'action_id: fe39f6ea-e567-4592-9bf9-0b30621585ec, type: CONFIG_CHANGE' received
Aug 28 18:38:34 elk-k3s elastic-agent[2664321]: 2020-08-28T18:38:34.438Z DEBUG application/handler_action_policy_change.go:34 handlerConfigChange: emit configuration for action action_id: fe39f6ea-e567-4592-9bf9-0b30621585ec, type: CONFIG_CHANGE
Aug 28 18:38:34 elk-k3s elastic-agent[2664321]: 2020-08-28T18:38:34.439Z DEBUG application/emitter.go:39 Transforming configuration into a tree
Aug 28 18:38:34 elk-k3s elastic-agent[2664321]: 2020-08-28T18:38:34.439Z DEBUG application/action_dispatcher.go:93 Failed to dispatch action 'action_id: fe39f6ea-e567-4592-9bf9-0b30621585ec, type: CONFIG_CHANGE', error: could not create the AST from the configuration: missing field accessing 'inputs'
Aug 28 18:38:34 elk-k3s elastic-agent[2664321]: 2020-08-28T18:38:34.440Z ERROR application/fleet_gateway.go:159 failed to dispatch actions, error: could not create the AST from the configuration: missing field accessing 'inputs'
Aug 28 18:38:34 elk-k3s elastic-agent[2664321]: 2020-08-28T18:38:34.441Z DEBUG application/fleet_gateway.go:162 FleetGateway is sleeping, next update in 1s
Aug 28 18:38:35 elk-k3s elastic-agent[2664321]: 2020-08-28T18:38:35.463Z DEBUG application/fleet_gateway.go:142 FleetGateway calling Checkin API
Aug 28 18:38:35 elk-k3s elastic-agent[2664321]: 2020-08-28T18:38:35.468Z DEBUG kibana/client.go:170 Request method: POST, path: /api/ingest_manager/fleet/agents/529228ed-7c5b-437c-8330-4a471d7b2ca2/checkin
The fleet.yml created by the enroll process:
agent:
id: 529228ed-7c5b-437c-8330-4a471d7b2ca2
fleet:
enabled: true
access_api_key: *bigsecrettokenfromenrollmenttokens*
kibana:
protocol: https
host: 10.11.0.246:5601
timeout: 1m30s
ssl:
verification_mode: none
renegotiation: never
reporting:
threshold: 10000
check_frequency_sec: 30
agent:
id: ""
The elastic-agent.yml created by the enroll process:
fleet:
enabled: true
At my understanding, even if the "Kibana URL" and "Elasticsearch URL" URLs are not the correct ones (please help me with this information too if you can), the Fleet Manager is not "pushing" the right configuration to the agent, but is not clear for me why.
Can anybody point me in the right direction to solve this issue? Because I really need and want to test the new Elastic Agent, mostly the EndPoint Security Feature.
Thanks in advance.
Regards,
Alejandro