Elastic agent does not send data

Pavel_Penka · June 29, 2020, 8:38pm

Hello,

I am trying to use ingest management and elastic agent on Windows Server 2012 against elastic cloud instance. Unfortunately, I am ending with the error bellow and no data is sent to elastic.

Can someone please help me with this?

Thank you very much.

Pavel

2020-06-29T22:28:44+02:00 INFO  stateresolver.go:47     New State ID is _-m5mkG0
2020-06-29T22:28:44+02:00 INFO  stateresolver.go:48     Converging state requires execution of 3 step(s)
2020-06-29T22:28:44+02:00 DEBUG operator.go:236 operator is looking for filebeat--7.8.0 in app collection: map[]
2020-06-29T22:28:44+02:00 INFO  operation_fetch.go:65   filebeat.7.8.0 already exists in C:\Program Files\Elastic-Agent\
data\downloads\filebeat-7.8.0-windows-x86_64.zip. Skipping operation operation-fetch
2020-06-29T22:28:44+02:00 INFO  operator.go:217 operation 'operation-fetch' skipped for filebeat.7.8.0
2020-06-29T22:28:44+02:00 INFO  operator.go:217 operation 'operation-verify' skipped for filebeat.7.8.0
2020-06-29T22:28:44+02:00 DEBUG operator.go:221 running operation 'operation-install' for filebeat.7.8.0
2020-06-29T22:28:45+02:00 ERROR reporter.go:47  2020-06-29T22:28:45+02:00: type: 'ERROR': sub_type: 'CONFIG' message: Ap
plication: filebeat[e1c9a6cf-852d-477e-b17e-e19fc6da241e]: operation-install: exit status 1
2020-06-29T22:28:45+02:00 DEBUG action_dispatcher.go:93 Failed to dispatch action 'action_id: a46ce0db-9abc-440d-9f39-95
d1eaf01826, type: CONFIG_CHANGE', error: operator: failed to execute step sc-run, error: operation-install: exit status
1: operation-install: exit status 1
        operator: failed to execute step sc-run, error: operation-install: exit status 1: operation-install: exit status
 1
        operation-install: exit status 1
        exit status 1
2020-06-29T22:28:45+02:00 ERROR fleet_gateway.go:163    failed to dispatch actions, error: operator: failed to execute s
tep sc-run, error: operation-install: exit status 1: operation-install: exit status 1
        operator: failed to execute step sc-run, error: operation-install: exit status 1: operation-install: exit status
 1
        operation-install: exit status 1
        exit status 1
2020-06-29T22:28:45+02:00 DEBUG fleet_gateway.go:166    FleetGateway is sleeping, next update in 30s

EricDavisX · June 30, 2020, 1:34am

Hello! I'm new to SDH issues, and new-ish to the ingest product but wanted to give some minimal help if I could. I'm sure others on the team will join in soon.

It is possible this is the same issue I've logged here:

github.com/elastic/beats

Elastic Agent install fails on Win 2012 Server as: filebeat[xyz] operation-install: exit status 1

opened 07:32PM - 27 May 20 UTC

closed 12:48AM - 11 Aug 20 UTC

EricDavisX

bug Ingest Management:beta1

The new Alpha Elastic Agent is supported on Win 2012 I believe, I tried running …it and it enrolled ok, but when running it, it throws an error, which is bubbled up into the Fleet Ingest Manager UI as: Application: filebeat[f8419071-5c7d-4c54-8a0e-5cf4c9a9eec5]: operation-install: exit status 1 I'm testing on BC3 of 7.8 on Elastic Cloud Staging, and using a hand compiled 8.0 Agent from 5/27. Agent OS Version: Windows Server 2012 R2 Standard Build 9600 (2013) - I'm afraid I only have a copy of this in vSphere, behind the Endgame VPN at the moment. We can work on how to get the team access or buy another image to test with for public use. Attached are 2 screenshots and the cmd prompt logs from the machine itself ![windows-2012-error](https://user-images.githubusercontent.com/12970373/83064053-29001600-a02f-11ea-8679-33378aee803c.png) ![windows-server-2012](https://user-images.githubusercontent.com/12970373/83064056-2998ac80-a02f-11ea-8a58-0b5b436a204c.png) [error-logs-win-2012.txt](https://github.com/elastic/beats/files/4691296/error-logs-win-2012.txt) specific error lines in the console output: 2020-05-27T15:15:45-04:00 ERROR reporter.go:47 2020-05-27T15:15:45-04:00: type: 'ERROR': sub_type: 'CONFIG' message: Application: metricbeat[f8419071-5c7d-4c54 -8a0e-5cf4c9a9eec5]: operation-install: exit status 1 2020-05-27T15:15:45-04:00 DEBUG action_dispatcher.go:93 Failed to dispatch actio n 'action_id: e770d274-7d40-4fe5-b7e2-37fe3b7f8035, type: CONFIG_CHANGE', error: operator: failed to execute step sc-run, error: operation-install: exit status 1: operation-install: exit status 1 operator: failed to execute step sc-run, error: operation-install: exit status 1: operation-install: exit status 1 operation-install: exit status 1 exit status 1 2020-05-27T15:15:45-04:00 ERROR fleet_gateway.go:163 failed to dispatch actio ns, error: operator: failed to execute step sc-run, error: operation-install: ex it status 1: operation-install: exit status 1 operator: failed to execute step sc-run, error: operation-install: exit status 1: operation-install: exit status 1 operation-install: exit status 1 exit status 1

We are prioritizing and hoping to make progress shortly.

is it possible for the immediate term for you to evaluate Ingest Manager and fleet with a different flavor of Windows? Win 7, Win 10, Win 8.1, Win 2019 are among the os versions we explicitly confirmed, per our support matrix listing for 7.8.

While I ask that.. I can ask more about the issue at hand, too:
One of the first things we would seek to trouble shoot is whether or not the host can communicate with Kibana. Can you confirm that using a 'ping' command or curl or similar returns successfully? If not the problem is in the networking / communication there.

If that works, perhaps you could post the configuration you are using when starting the Agent (are you trying to follow 'stand-alone' agent mode usage) or you can post the configuration yaml from the Ingest Manager UI in Kibana if you are following the Fleet-controlled Agent usage.

Regards.

Pavel_Penka · June 30, 2020, 9:48am

Hello!

Thank you for your answer, the issue you sent looks the same as the one I have. I am looking for the version 7.9.x where it should be fixed. Is there any chance to get the fixed version before? For the other questions, I could give it a try on WIn 2019, maybe Win 10.

To the issue, host can definitely communicate with Kibana, it is listed in fleet as online/error and has the activity log with items in it. The configuration is the fleet mode one, yaml attached.

Thank you,

id: a7e9af70-ba3e-11ea-977e-b370193d06b6
revision: 7
outputs:
  default:
    type: elasticsearch
    hosts:
      - >-
        https://a1d50438db3047cf8c6e2f1bdfbe407f.northeurope.azure.elastic-cloud.com:443
datasources:
  - id: aeebd050-ba3e-11ea-977e-b370193d06b6
    name: system-1
    enabled: true
    package:
      name: system
      version: 0.3.0
    namespace: default
    use_output: default
    inputs:
      - type: logs
        enabled: true
        streams:
          - id: logs-system.auth
            enabled: true
            dataset: system.auth
            exclude_files:
              - .gz$
            paths:
              - /var/log/auth.log*
              - /var/log/secure*
            multiline:
              pattern: ^\s
              match: after
            processors:
              - add_locale: null
              - add_fields:
                  fields:
                    ecs.version: 1.5.0
                  target: ''
          - id: logs-system.syslog
            enabled: true
            dataset: system.syslog
            exclude_files:
              - .gz$
            paths:
              - /var/log/messages*
              - /var/log/syslog*
            multiline:
              pattern: ^\s
              match: after
            processors:
              - add_locale: null
              - add_fields:
                  fields:
                    ecs.version: 1.5.0
                  target: ''
      - type: system/metrics
        enabled: true
        streams:
          - id: system/metrics-system.cpu
            enabled: true
            dataset: system.cpu
            period: 10s
            cpu.metrics:
              - percentages
              - normalized_percentages
            metricsets:
              - cpu
          - id: system/metrics-system.diskio
            enabled: true
            dataset: system.diskio
            period: 10s
            diskio.include_devices: null
            metricsets:
              - diskio
          - id: system/metrics-system.fsstat
            enabled: true
            dataset: system.fsstat
            period: 1m
            metricsets:
              - fsstat
            processors:
              - drop_event.when.regexp:
                  system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
          - id: system/metrics-system.load
            enabled: true
            dataset: system.load
            period: 10s
            metricsets:
              - load
          - id: system/metrics-system.memory
            enabled: true
            dataset: system.memory
            period: 10s
            metricsets:
              - memory
          - id: system/metrics-system.network
            enabled: true
            dataset: system.network
            period: 10s
            network.interfaces: null
            metricsets:
              - network
          - id: system/metrics-system.process
            enabled: true
            dataset: system.process
            process.include_top_n.by_memory: 5
            period: 10s
            processes:
              - .*
            process.include_top_n.by_cpu: 5
            process.cgroups.enabled: true
            process.cmdline.cache.enabled: true
            metricsets:
              - process
          - id: system/metrics-system.process_summary
            enabled: true
            dataset: system.process_summary
            period: 10s
            metricsets:
              - process_summary
          - id: system/metrics-system.socket_summary
            enabled: true
            dataset: system.socket_summary
            period: 10s
            metricsets:
              - socket_summary
          - id: system/metrics-system.uptime
            enabled: true
            dataset: system.uptime
            period: 10s
            metricsets:
              - uptime
  - id: 1d7c48c0-bab6-11ea-977e-b370193d06b6
    name: log-1
    enabled: true
    package:
      name: log
      version: 0.1.0
    namespace: default
    use_output: default
    inputs:
      - type: logs
        enabled: true
        streams:
          - id: logs-generic
            type: log
            enabled: true
            dataset: generic
            paths:
              - 'd:/data/logfiles/service.*.log'
settings:
  monitoring:
    enabled: true
    use_output: default
    logs: true
    metrics: true

EricDavisX · July 2, 2020, 2:10pm

Hello - we're reviewing the issue in depth on our end and when fixed we can share the 7.9 build, I cannot provide any time estimate on that. Thank you for the patience and for asking! Best regards.

Pavel_Penka · July 3, 2020, 7:41am

Great, thank you very much.

system · July 31, 2020, 9:41am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

EricDavisX · August 6, 2020, 1:36pm

Hello Pavel_Penka, hope you are well. We found the issue to be that our test version of Win 2012 had a very old version of Powershell, and the commands we were using were not compatible with it. We've updated the Agent and successfully tested on Win 2012, so I hope the 7.9 release will work for you as well. If there is anything that doesn't work further, please do report a ticket in the elastic/beats repo for us (or here again if you desire). Recommend using this location to try it out, or wait for the 7.9 GA - https://staging.elastic.co/7.9.0-aed29770/summary-7.9.0.html

Thanks again for the interest and usage.