Elastic Agent 7.9.1 doesn´t send data on Windows

[hofrichterovak (Cate)]

Hello,

I have problem with Elastic Agent 7.9.1 on Windows. Agent doesn´t generate any data. I don´t see generated dataset on tab Datasets in Kibana. GET metrics-/_search; GET logs-/_search is without data too.

At the beginning I had problem with starting Elastic Agent (https://github.com/elastic/beats/issues/20404). I needed to restart computer to start Elastic Agent service.

I started it this way:

1. ./install-service-elastic-agent.ps1
2. I opened app Services, there was newly added 'elastic-agent'. Status of 'elastic-agent' was Stopped. When I started service here manually, it generated error.
3. I restarted computer
4. I opened app Services. Status of 'elastic-agent' was Running
5. I enrolled agent '.\elastic-agent enroll http://localhost:5601 xxxxxxxxxxxxxxxx --insecure'
6. ./install-service-elastic-agent.ps1
7. I had active agent in Kibana Fleet

On Linux agent works.

Thank you,
Katerina

[ruflin]

Could you check the logs in the ./data/ directory to see if it contains any errors?

[hofrichterovak (Cate)]

Hello,

In ./data I see this error:

 {"log.level":"info","@timestamp":"2020-09-16T11:01:43.997+0200","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2020-09-16T11:01:43+02:00: type: 'STATE': sub_type: 'RUNNING' message: Application: endpoint-security--7.9.1[019dd1bb-921d-4f22-813f-df9803c84529]: State changed to DEGRADED: Missed last check-in","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-16T11:01:52.344+0200","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2020-09-16T11:01:52+02:00: type: 'STATE': sub_type: 'RUNNING' message: Application: endpoint-security--7.9.1[019dd1bb-921d-4f22-813f-df9803c84529]: State changed to RUNNING: ","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-16T11:06:52.721+0200","log.origin":{"file.name":"kibana/client.go","file.line":170},"message":"Request method: POST, path: /api/ingest_manager/fleet/agents/019dd1bb-921d-4f22-813f-df9803c84529/checkin","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-09-16T11:06:56.727+0200","log.origin":{"file.name":"application/fleet_gateway.go","file.line":176},"message":"Could not communicate with Checking API will retry, error: fail to checkin to fleet: Post \"http://localhost:5601/api/ingest_manager/fleet/agents/019dd1bb-921d-4f22-813f-df9803c84529/checkin?\": dial tcp 127.0.0.1:5601: connectex: No connection could be made because the target machine actively refused it.","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-16T11:16:13.880+0200","log.origin":{"file.name":"kibana/client.go","file.line":170},"message":"Request method: POST, path: /api/ingest_manager/fleet/agents/019dd1bb-921d-4f22-813f-df9803c84529/checkin","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-09-16T11:16:17.886+0200","log.origin":{"file.name":"application/fleet_gateway.go","file.line":176},"message":"Could not communicate with Checking API will retry, error: fail to checkin to fleet: Post \"http://localhost:5601/api/ingest_manager/fleet/agents/019dd1bb-921d-4f22-813f-df9803c84529/checkin?\": dial tcp [::1]:5601: connectex: No connection could be made because the target machine actively refused it.","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-16T11:25:47.176+0200","log.origin":{"file.name":"kibana/client.go","file.line":170},"message":"Request method: POST, path: /api/ingest_manager/fleet/agents/019dd1bb-921d-4f22-813f-df9803c84529/checkin","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-09-16T11:25:51.179+0200","log.origin":{"file.name":"application/fleet_gateway.go","file.line":176},"message":"Could not communicate with Checking API will retry, error: fail to checkin to fleet: Post \"http://localhost:5601/api/ingest_manager/fleet/agents/019dd1bb-921d-4f22-813f-df9803c84529/checkin?\": dial tcp [::1]:5601: connectex: No connection could be made because the target machine actively refused it.","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-16T11:33:10.783+0200","log.origin":{"file.name":"kibana/client.go","file.line":170},"message":"Request method: POST, path: /api/ingest_manager/fleet/agents/019dd1bb-921d-4f22-813f-df9803c84529/checkin","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-09-16T11:33:14.787+0200","log.origin":{"file.name":"application/fleet_gateway.go","file.line":176},"message":"Could not communicate with Checking API will retry, error: fail to checkin to fleet: Post \"http://localhost:5601/api/ingest_manager/fleet/agents/019dd1bb-921d-4f22-813f-df9803c84529/checkin?\": dial tcp 127.0.0.1:5601: connectex: No connection could be made because the target machine actively refused it.","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-16T11:39:15.091+0200","log.origin":{"file.name":"kibana/client.go","file.line":170},"message":"Request method: POST, path: /api/ingest_manager/fleet/agents/019dd1bb-921d-4f22-813f-df9803c84529/checkin","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-09-16T11:39:19.095+0200","log.origin":{"file.name":"application/fleet_gateway.go","file.line":176},"message":"Could not communicate with Checking API will retry, error: fail to checkin to fleet: Post \"http://localhost:5601/api/ingest_manager/fleet/agents/019dd1bb-921d-4f22-813f-df9803c84529/checkin?\": dial tcp 127.0.0.1:5601: connectex: No connection could be made because the target machine actively refused it.","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-16T11:42:09.355+0200","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2020-09-16T11:42:09+02:00: type: 'STATE': sub_type: 'RUNNING' message: Application: endpoint-security--7.9.1[019dd1bb-921d-4f22-813f-df9803c84529]: State changed to DEGRADED: Missed last check-in","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-16T11:42:15.358+0200","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2020-09-16T11:42:15+02:00: type: 'STATE': sub_type: 'RUNNING' message: Application: endpoint-security--7.9.1[019dd1bb-921d-4f22-813f-df9803c84529]: State changed to RUNNING: ","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-16T11:44:57.708+0200","log.origin":{"file.name":"kibana/client.go","file.line":170},"message":"Request method: POST, path: /api/ingest_manager/fleet/agents/019dd1bb-921d-4f22-813f-df9803c84529/checkin","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-09-16T11:45:01.712+0200","log.origin":{"file.name":"application/fleet_gateway.go","file.line":176},"message":"Could not communicate with Checking API will retry, error: fail to checkin to fleet: Post \"http://localhost:5601/api/ingest_manager/fleet/agents/019dd1bb-921d-4f22-813f-df9803c84529/checkin?\": dial tcp [::1]:5601: connectex: No connection could be made because the target machine actively refused it.","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-16T11:52:09.309+0200","log.origin":{"file.name":"kibana/client.go","file.line":170},"message":"Request method: POST, path: /api/ingest_manager/fleet/agents/019dd1bb-921d-4f22-813f-df9803c84529/checkin","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-09-16T11:52:13.313+0200","log.origin":{"file.name":"application/fleet_gateway.go","file.line":176},"message":"Could not communicate with Checking API will retry, error: fail to checkin to fleet: Post \"http://localhost:5601/api/ingest_manager/fleet/agents/019dd1bb-921d-4f22-813f-df9803c84529/checkin?\": dial tcp [::1]:5601: connectex: No connection could be made because the target machine actively refused it.","ecs.version":"1.5.0"}

Thank you,
Katerina

[ruflin]

This is strange as the enrollment worked. Any chance you could share what is shown in the Kibana / Elasticsearch logs? I would expect some logs also there on why it is "actively refused".

[ruflin]

To double check, is your Kibana instance running on localhost:5601?

[hofrichterovak (Cate)]

Hello,

yes, Kibana is running on localhost:5601. Elasticsearch, Kibana´s logs look fine. I don´t see there errors.

K.

[ruflin]

@Michal_Pristas Could you chime in here? I wonder in which case KB returns the error above? Permissions?

[Michal_Pristas]

this is a network exception happening when agent tries to connect to a machine where nothing is listening on the port.

maybe kibana was not initialized just yet? just guessing. for me it looks like this. is there some kind of firewall this might be also the case

[windkill]

Same problem, U resovled it?

[windkill]

@ruflin @Michal_Pristas Maybe Same problem. I check network connect，it looks fine.

My windows agent log

I also see logs in Elastic-Agent\data\logs\default

[Michal_Pristas]

it looks like agent is ok connecting to fleet (kibana) (which was not possible with the original post) but beats has some issues with ES connection. i will try to setup win machine with this stack to play with it a bit

[hofrichterovak (Cate)]

Hello, Problems still persist. Kibana is started and the firewall is completely turned off.

[windkill]

Thanks for your help. En... I just made a mistake, I used wrong elasticsearch.yml which is backup-older, and "# network.host: 0.0.0.0" is inside..... After I reconfigure elasticsearch.yml, Ingest Manager works fine.

@hofrichterovak I think we have different problems, but my mistake maybe it will help you。
I suggest you check your configuration file again.

[Michal_Pristas]

also we;ve seen some issues if metricbeat/filebeat was present before. was this fresh install @hofrichterovak

[hofrichterovak (Cate)]

Hello, yes. First I had installed elasticsearch, filebeat, and then I finally added the Elastic Agent