8.2.3 Agent unhealthy, when "Network Packet Capture" integration is enabled in agent policy

buzzdeee · June 16, 2022, 11:27am

I created a elastic cloud trial instance yesterday, but my Windows client is unhealthy.
elastic-agent status output shows:


Status: FAILED
Message: (no message)
Applications:
  * osquerybeat        (HEALTHY)
                       Running
  * packetbeat         (HEALTHY)
                       Running
  * endpoint-security  (HEALTHY)
                       Protecting with policy {4e3326b6-237c-4ba7-9f65-b30f646605f3}
  * filebeat           (FAILED)
                       1 error occurred:
                       * 1 error: Error creating runner from config: missing required field accessing 'hosts'


  * filebeat_monitoring    (HEALTHY)
                           Running
  * metricbeat_monitoring  (HEALTHY)
                           Running
  * metricbeat             (HEALTHY)
                           Running

and diagnostics shows:

elastic-agent  id: 762aad65-f9a4-42ff-b408-2c0b83e76245                version: 8.2.3
               build_commit: f44953023f48ff11f9e5eb6d7194d741955e1083  build_time: 2022-06-09 01:04:56 +0000 UTC  snapshot_build: false
Applications:
  *  name: filebeat_monitoring    route_key: default
     process: filebeat            id: 011233b5-696c-4403-af38-dc3db4db1224          ephemeral_id: 0ddfabf4-d7c2-4735-93a5-63e7e07c8113  elastic_license: true
     version: 8.2.3               commit: 7826dc5e91c6e6d2487e05d3a8298f49041cd5c2  build_time: 2022-06-08 15:51:33 +0000 UTC           binary_arch: amd64
     hostname: BERLINER55         username: NT AUTHORITY\SYSTEM                     user_id: S-1-5-18                                   user_gid: S-1-5-18
  *  name: metricbeat_monitoring  route_key: default
     process: metricbeat          id: c8bdeb8c-399e-49f4-8362-f2955d14d246          ephemeral_id: a5a64bab-846e-4a49-89e9-6fec45e5cfbb  elastic_license: true
     version: 8.2.3               commit: 7826dc5e91c6e6d2487e05d3a8298f49041cd5c2  build_time: 2022-06-08 15:49:40 +0000 UTC           binary_arch: amd64
     hostname: BERLINER55         username: NT AUTHORITY\SYSTEM                     user_id: S-1-5-18                                   user_gid: S-1-5-18
  *  name: metricbeat             route_key: default
     process: metricbeat          id: c8bdeb8c-399e-49f4-8362-f2955d14d246          ephemeral_id: a5a64bab-846e-4a49-89e9-6fec45e5cfbb  elastic_license: true
     version: 8.2.3               commit: 7826dc5e91c6e6d2487e05d3a8298f49041cd5c2  build_time: 2022-06-08 15:49:40 +0000 UTC           binary_arch: amd64
     hostname: BERLINER55         username: NT AUTHORITY\SYSTEM                     user_id: S-1-5-18                                   user_gid: S-1-5-18
  *  name: osquerybeat            route_key: default
     process: osquerybeat         id: 63d4b038-dd23-4b38-a600-4b6cc8207829          ephemeral_id: 32d8d73c-ca99-4ba6-8082-bd3c887cab28  elastic_license: true
     version: 8.2.3               commit: 7826dc5e91c6e6d2487e05d3a8298f49041cd5c2  build_time: 2022-06-08 15:43:52 +0000 UTC           binary_arch: amd64
     hostname: BERLINER55         username: NT AUTHORITY\SYSTEM                     user_id: S-1-5-18                                   user_gid: S-1-5-18
  *  name: packetbeat             route_key: default
     process: packetbeat          id: c9672290-db27-4d94-acfa-14b01072b4d8          ephemeral_id: e6cef367-62ec-4fb5-84a4-36b4ad24b8bf  elastic_license: true
     version: 8.2.3               commit: 7826dc5e91c6e6d2487e05d3a8298f49041cd5c2  build_time: 2022-06-08 15:57:21 +0000 UTC           binary_arch: amd64
     hostname: BERLINER55         username: NT AUTHORITY\SYSTEM                     user_id: S-1-5-18                                   user_gid: S-1-5-18
  *  name: endpoint-security      route_key: default
     error: Get "http://npipe/": open \\.\pipe\default-endpoint-security: The system cannot find the file specified.
  *  name: filebeat        route_key: default
     process: filebeat     id: 011233b5-696c-4403-af38-dc3db4db1224          ephemeral_id: 0ddfabf4-d7c2-4735-93a5-63e7e07c8113  elastic_license: true
     version: 8.2.3        commit: 7826dc5e91c6e6d2487e05d3a8298f49041cd5c2  build_time: 2022-06-08 15:51:33 +0000 UTC           binary_arch: amd64
     hostname: BERLINER55  username: NT AUTHORITY\SYSTEM                     user_id: S-1-5-18                                   user_gid: S-1-5-18

diagnostics actually looks good, but I wonder about the endpoint-security error message.

Anyone knows what's going on?

buzzdeee · June 16, 2022, 11:37am

Just saw, on the test Mac client, I see the same:

Status: FAILED
Message: (no message)
Applications:
  * metricbeat             (HEALTHY)
                           Running
  * metricbeat_monitoring  (HEALTHY)
                           Running
  * filebeat               (FAILED)
                           1 error occurred:
                           * 1 error: Error creating runner from config: missing required field accessing 'hosts'


  * endpoint-security    (HEALTHY)
                         Protecting with policy {4e3326b6-237c-4ba7-9f65-b30f646605f3}
  * osquerybeat          (HEALTHY)
                         Running
  * packetbeat           (HEALTHY)
                         Running
  * filebeat_monitoring  (HEALTHY)
                         Running
elastic-agent  id: 47fb4fb1-840e-4837-986e-b34b583de0e3                version: 8.2.3
               build_commit: f44953023f48ff11f9e5eb6d7194d741955e1083  build_time: 2022-06-09 01:04:55 +0000 UTC  snapshot_build: false
Applications:
  *  name: endpoint-security  route_key: default
     error: Get "http://unix/": dial unix /Library/Elastic/Agent/data/tmp/default/endpoint-security/endpoint-security.sock: connect: no such file or directory
  *  name: osquerybeat            route_key: default
     process: osquerybeat         id: a4e27933-0879-45f1-b489-73ae5ce4c9c7          ephemeral_id: f6b678d9-b6af-4bfb-92db-cdade9ff2cec  elastic_license: true
     version: 8.2.3               commit: 7826dc5e91c6e6d2487e05d3a8298f49041cd5c2  build_time: 2022-06-08 15:43:55 +0000 UTC           binary_arch: amd64
     hostname: berliner99.local   username: root                                    user_id: 0                                          user_gid: 0
  *  name: packetbeat             route_key: default
     process: packetbeat          id: ccbf8fb8-1bef-47ed-a7cc-92f724c5ffc1          ephemeral_id: 596f8479-d6b9-4011-9cda-ec8e7983641b  elastic_license: true
     version: 8.2.3               commit: 7826dc5e91c6e6d2487e05d3a8298f49041cd5c2  build_time: 2022-06-08 15:49:49 +0000 UTC           binary_arch: amd64
     hostname: berliner99.local   username: root                                    user_id: 0                                          user_gid: 0
  *  name: filebeat_monitoring    route_key: default
     process: filebeat            id: a7065d5c-9f71-4c08-a734-d9381e4ad686          ephemeral_id: 63498967-3960-4b64-9231-3aeb37a7f719  elastic_license: true
     version: 8.2.3               commit: 7826dc5e91c6e6d2487e05d3a8298f49041cd5c2  build_time: 2022-06-08 15:51:33 +0000 UTC           binary_arch: amd64
     hostname: berliner99.local   username: root                                    user_id: 0                                          user_gid: 0
  *  name: metricbeat             route_key: default
     process: metricbeat          id: eb92465f-72db-494e-bc23-ae0a15767a18          ephemeral_id: d2db2f04-2219-45d5-9371-fedbe53d2b2d  elastic_license: true
     version: 8.2.3               commit: 7826dc5e91c6e6d2487e05d3a8298f49041cd5c2  build_time: 2022-06-08 15:49:40 +0000 UTC           binary_arch: amd64
     hostname: berliner99.local   username: root                                    user_id: 0                                          user_gid: 0
  *  name: metricbeat_monitoring  route_key: default
     process: metricbeat          id: eb92465f-72db-494e-bc23-ae0a15767a18          ephemeral_id: d2db2f04-2219-45d5-9371-fedbe53d2b2d  elastic_license: true
     version: 8.2.3               commit: 7826dc5e91c6e6d2487e05d3a8298f49041cd5c2  build_time: 2022-06-08 15:49:40 +0000 UTC           binary_arch: amd64
     hostname: berliner99.local   username: root                                    user_id: 0                                          user_gid: 0
  *  name: filebeat               route_key: default
     process: filebeat            id: a7065d5c-9f71-4c08-a734-d9381e4ad686          ephemeral_id: 63498967-3960-4b64-9231-3aeb37a7f719  elastic_license: true
     version: 8.2.3               commit: 7826dc5e91c6e6d2487e05d3a8298f49041cd5c2  build_time: 2022-06-08 15:51:33 +0000 UTC           binary_arch: amd64
     hostname: berliner99.local   username: root                                    user_id: 0                                          user_gid: 0

buzzdeee · June 16, 2022, 12:03pm

On a Linux client it looked a little different, but then I removed the "Network Packet Capture" integration from the policies, and they all three became healthy.

Seems to be something odd with that integration.

TiagoQueiroz · June 17, 2022, 9:27am

Indeed it's a problem with the "Network Packet Capture" integration, more specifically with its Redis capture.

If you want to use the integration, just disable the Redis capture, to do so go to the integration settings, then on "Capture network traffic" click on "Change defaults", scroll down until you find "Redis", then turn it off.

That should solve the problem and make Filebeat start.

I'll report the bug.

buzzdeee · June 17, 2022, 3:09pm

Hi @TiagoQueiroz

thanks for your reply, that helped.
In my testing, I enabled everything, even though I don't need all.
With only the pieces enabled that I need, agents are healthy.

cheers,
Sebastian

andrewkroh · July 6, 2022, 1:35pm

It's a problem with the way Agent generates the config for Filebeat. The Agent misidentifies the redis config as something that belongs to Filebeat instead of Packetbeat.

github.com/elastic/elastic-agent

Filebeat spec is incorporating Packetbeat inputs into its config

opened 10:42PM - 13 May 22 UTC

andrewkroh

bug Team:Elastic-Agent-Data-Plane 8.5-candidate

When an agent policy contains the Packetbeat redis input, Elastic Agent is gener…ating config for Filebeat that includes a redis log input. The policy for Packetbeat uses input `type: packet` with a data_stream of `type: redis`. For example: ```yaml # agent policy inputs: - type: packet streams: - data_stream: dataset: network_traffic.redis type: logs type: redis ports: - 6379 ``` The impact is that this causes Filebeat to report UNHEALTHY status if Packetbeat is deployed at the same time. A workaround is to disable redis collection in the Network Packet Capture integration. Here is a patch to the Elastic Agent testdata that reproduces the bug using unit tests: <details> ```diff diff --git a/internal/pkg/agent/program/testdata/single_config-packetbeat.yml b/internal/pkg/agent/program/testdata/single_config-packetbeat.yml index f800d0bd2..4ea37b1fb 100644 --- a/internal/pkg/agent/program/testdata/single_config-packetbeat.yml +++ b/internal/pkg/agent/program/testdata/single_config-packetbeat.yml @@ -23,6 +23,13 @@ inputs: data_stream: dataset: packet.icmp type: logs + - data_stream: + dataset: network_traffic.redis + type: logs + id: packet-network_traffic.redis-387bdc6a-0acb-4ef2-9552-c21e524a2d21 + ports: + - 6379 + type: redis output: elasticsearch: hosts: diff --git a/internal/pkg/agent/program/testdata/single_config.yml b/internal/pkg/agent/program/testdata/single_config.yml index 16a03f9a7..140a61f79 100644 --- a/internal/pkg/agent/program/testdata/single_config.yml +++ b/internal/pkg/agent/program/testdata/single_config.yml @@ -104,6 +104,13 @@ inputs: data_stream: dataset: packet.icmp type: logs + - data_stream: + dataset: network_traffic.redis + type: logs + id: packet-network_traffic.redis-387bdc6a-0acb-4ef2-9552-c21e524a2d21 + ports: + - 6379 + type: redis - id: endpoint-id type: endpoint name: endpoint-1 ``` </details> ### Workarounds You can disable the redis protocol in the network packet capture integration. <img width="717" alt="Screen Shot 2022-05-13 at 18 41 58" src="https://user-images.githubusercontent.com/4565752/168398092-a23997f5-9723-448e-b7b0-9fee64ca8195.png"> <img width="715" alt="Screen Shot 2022-05-13 at 18 42 12" src="https://user-images.githubusercontent.com/4565752/168398099-0e8b16c0-685e-41b6-87e1-017956a1fb17.png">

system · August 3, 2022, 3:36pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.