Elastic-agent with system module does not send any data to elasticsearch

[zozo6015 (Peter)]

Hello,

I have deployed a simple elastic-agent with a system system module where I wanted to have the /var/log/syslog ad messages parsed and send metrics (pretty default). But the module does not send any data into the elasticsearch. The output works ok as the locallly deployed integration with pretty much the same settings works just right. Any idea what can be the issue?

[stephenb (Stephen Brown)]

Hi @zozo6015

We need a lot more details...

What version of the Agent / Elastic Stack?

Standalone or Fleet Managed?

Please share the agent Policy if stand alone

@zozo6015

But the module does not send any data into the elasticsearch. The output works ok as the locallly deployed integration

What does this mean? I am confused does it work for a local elasticsearch but not remote?

Can you describe your architecture? what is "local" and what is "remote"

Have you validated connectivity to the remote cluster?

What is the remote Cluster is it Elastic Cloud? if so you have to set the port to 443 or 9243 because otherwise it will default to 9200 and it wont work...

As much detail as you can provide.. perhaps someone can help.

[zozo6015 (Peter)]

Both elasticsearch and agent are the latest version (8.5). Self hosted everything. Using Fleet server. I have installed integration on the fleet server as well and the fleet server gets the data, but integration on remote servers (not the same server as the fleet server) doesn't send any data into elasticsearch.

Architecture looks like the following elasticsearch cluster <--- fleet server(with integrations)<-- application server(fleet agent running on them).
elasticsearch is running on port 9200 with defalt transport and http ssl certificates. Connectivity is ensured using ss.ca-cert-fingerprint (tested, connectivity works fine.)

Also attaching the two different policies:

policies - Pastebin.com

[stephenb (Stephen Brown)]

@zozo6015

doesn't send any data into elasticsearch.

Apologies I am pretty much completely confused... Think we are close but some basic questions.

Are you trying to use / install Fleet Managed Agents? Did you follow these instructions to install the Local / Self Managed Fleet Server? here

And you aware of the difference between
Fleet vs Stand Alone Agents

The policy above looks to the be the "Local Agent Policy which Has the Fleet Server Policy / + Many other integrations" not the remote agent policy which is what I thought we were talking about... you can not just deploy that to other remote hosts and have that work.

I assume since you are using Fleet you want Fleet Managed Agents? Correct?

If So how are you installing / configuring the remote agents are you using the Add Agent Instructions from within Kibana?

You should not be "manually installing policies" etc on the remote host if you want Fleet Managed Agents.

This is My Full Self Managed Setup up With Elasticsearch, Kibana and the Fleet Server Self Signed Certs all on 1 host. Then if I want to install agents other host I use the Add Agents from the Kibana UI.

You will just run the commands given to install the agent then it will automatically download the policies etc... you do not edit and manually install them

tar xzvf elastic-agent-8.5.2-linux-x86_64.tar.gz
cd elastic-agent-8.5.2-linux-x86_64
sudo ./elastic-agent install --url=https://fleet.mydomain.net:8220 --enrollment-token=TsafsdfsadfasdfQMDdBNlk2ZGc6cFl2QXpXbWdTYWFxckZwZlpoX3VoZw==

[zozo6015 (Peter)]

@stephenb

BTW This is My Full Self Managed Setup up With Elasticsearch, Kibana and the Fleet Server Self Signed Certs all on 1 host. Then if I want to install agents other host I use the Add Agents from the Kibana UI.

Yes I have added the agent using the the wizard showed. All was ok until at the end of the wizard saying wait for new data to come and that never comes. As you see in the image bellow I have two agents the elastic-search works just fine (it is the fleet server itself), while the egy1vpsie00 does not gets any data.

[stephenb (Stephen Brown)]

Ok good. I was confused when you sent me that whole policy.

@zozo6015

All was ok until at the end of the wizard saying wait for new data to come and that never comes.

Yeah that needs to go green...

You have tested connectivity from the remote host to the host / ports for you Fleet / Elasticsearch?

Curious did you bind elasticsearch to the network?

Did you log into that remote host and You can curl elasticsearch from the remote host?

Have you tested all the connectivity to elasticsearch and fleet?

There are instructions in the trouble shooting guide.

I would start with the trouble shooting Guide.

Most common issues, connectivity and certs ...

Troubleshoot common problems | Fleet and Elastic Agent Guide [8.11] | Elastic

elastic-agent status

curl -f http://<fleet-server-url>:8220/api/status

curl -u elastic http://<elasticsearch-server-url>:9200

[zozo6015 (Peter)]

@stephenb

elastic-agent status

Applications:
  * filebeat               (HEALTHY)
                           Running
  * metricbeat             (HEALTHY)
                           Running
  * filebeat_monitoring    (HEALTHY)
                           Running
  * metricbeat_monitoring  (HEALTHY)
                           Running

curl -f https://fleet.elasticdns.ml:8220/api/status -k
{"name":"fleet-server","status":"HEALTHY"}
 
root@egy1vpsie00:~# curl -u elastic https://10.100.11.204:9200 -u elastic -k
Enter host password for user 'elastic':
{
  "name" : "elastic-search",
  "cluster_name" : "vpsie-logs",
  "cluster_uuid" : "8Aeby0suQsGd4Xoic_fVwg",
  "version" : {
    "number" : "8.5.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "a846182fa16b4ebfcc89aa3c11a11fd5adf3de04",
    "build_date" : "2022-11-17T18:56:17.538630285Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

as you can see all seam to be up and working but logs/metrics are not getting pushed from the remote agent.

[stephenb (Stephen Brown)]

But it never went green?

I would uninstall and try again.

Remove add the policy again.

You added the System Module? Where are you looking for the metrics?

I don't have a simple answer... The good... you seem to be doing all the right things.. the bad.. it is not working....

Does the Agent Metrics show up in the Agent Dashboard?

[stephenb (Stephen Brown)]

Is it just the system integration or all those other integrations as well?

[zozo6015 (Peter)]

Not sure what you saying that was never green. The agents were always showing green.

[zozo6015 (Peter)]

As you can see the agent's Dashnoard is empty.

[stephenb (Stephen Brown)]

@zozo6015

All was ok until at the end of the wizard saying wait for new data to come and that never comes.

I meant this I should not have said Green ... you get a check mark / says completed.... if that did not happen then the install / connection / configuration did not completely finish.

[stephenb (Stephen Brown)]

Regarding the dashboard there is a bug with the CPU the others should show data... so you are not sending data... even if the agent Show Green / Healthy (which I admit is weird / not right)

Is there anything in the agent logs? probably not...

Did you uninstall and try to reinstall

I would uninstall from the command line on the remote host...

Then try to re-install the agent

I have to step away for the rest of today good luck!

[zozo6015 (Peter)]

@stephenb

I have to step away for the rest of today good luck!

I have uninstalled it and reinstalled it. As usual it hangs in here. Nothing in the agent's logs.

[zozo6015 (Peter)]

This is all I can see in the agent-logs.

[zozo6015 (Peter)]

I have run a tcpdump on the elasticsearch server and I can see traffic on the tcp level that is coming into the elasticsearch machine from that remote server but the traffic is not showing up into the elasticsearch. I will look into this more. On the other hand if I install metricbeat standalone app on the same machine with the same output setup I will get the data just right into the elasticsearch.

[stephenb (Stephen Brown)]

Try Reinstalling the System Integration Assets...

Please no screen shots of text to hard to see... paste with </> button

Those logs look like the shutdown what does the startup look like?

Try another Host?

Run the Diagnostics bundle

elastic-agent diagnostics collect

Something unusual is going on...

I know this does not directly help you but I just installed another host in my subnet...
Self Signed cert I needed to use the --insecure flag.... self signed cert.,

sudo ./elastic-agent install --url=https://fleet.bvader.net:8220 --enrollment-token=TnsadfasdfasdfsadfQMDdBNlk2ZGc6cFl2QXpXbWdTYWFxckZwZlpoX3VoZw== --insecure

Then I got the Check Mark... took about 30 sec.

So your next step is run the diagnostics

elastic-agent diagnostics collect

Then unzip and go into the configs and logs...

I suspect the underlying metricbeat is not connecting with elasticsearch or something like that

looks for the connection logs I suspect it will be failing

metricbeat-20221124-1.ndjson:{"log.level":"info","@timestamp":"2022-11-24T19:19:05.454Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(elasticsearch(https://10.168.0.12:9200))","service.name":"metricbeat","ecs.version":"1.6.0"}
metricbeat-20221124-1.ndjson:{"log.level":"info","@timestamp":"2022-11-24T19:19:05.513Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(elasticsearch(https://10.168.0.12:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}

You can look at the policies etc... lots of stuff...

from the directory
grep 10.168.0.12 config/*

gotta run good luck...

Something basic going on .. should not be this hard... I will check back later if I can

NOTE: Just found a small bug I think (that is not your issue) The Agent CPU process for the Agent Running Fleet is not properly reported. I am seeing the CPU etc for the new Agent I installed remotely.

This is for the new host I just deployed on

Sorry you are having so much trouble.. it should be easier than this ... for you.

[zozo6015 (Peter)]

I went through the diagnostics configs and one thing I could observe that on the output the elasticsearch server is https://localhost:9200 instead of the configured IP address of the policy in fleet settings. Is there a way to overwrite these settings? Basically except the ssl.ca_trusted_fingerprint everything in the output configuration is wrong.

[zozo6015 (Peter)]

Was my mistake as I overlooked the elasticsearch ip address. Once I have connected it started to work. Thank you for your help. And Happy Thanksgiving.

[stephenb (Stephen Brown)]

@zozo6015

I went through the diagnostics configs and one thing I could observe that on the output the elasticsearch server is https://localhost:9200 instead of the configured IP address of the policy in fleet settings. Is there a way to overwrite these settings? Basically except the ssl.ca_trusted_fingerprint everything in the output configuration is wrong.

Yup that is a problem for sure....

I did not have that problem...

What does your elasticsearch.yml look like .. .especially this network host settings....