API logs not coming after setting up Elastic Agent , Beats logs are coming

Elastic Stack Elastic Agent
1

Hi All,

We have setup an Elastic Agent to send logs from an Application API to Kafka.
We are able to establish the connection using the Elastic Agent. But instead of the API logs, we are getting Beat logs (Filebeat and Metricbeat).

Below is the Elastic-agent.yml configuration :

agent:
  logging:
    files:
      keepfiles: 7
      name: elastic-agent
      path: /var/log/elastic-agent/
      permissions: 420
    level: info
    to_files: true

inputs:
  - type: httpjson
    id: api-logs-input
    schedule: '@every 1m'
    config:
      url: https://.com/monitoring/logs?source=am-everything
      method: GET
      headers:
        Authorization: Bearer 2--1:c--1
      response_format: json
    use_output: default

outputs:
  default:
    type: kafka
    hosts:
      - b-1.---:9094
      - b-2.---:9094
      - b-3.---:9094
    topic: test_app_topic
    ssl:
      enabled: true
      truststore_location: /etc/pki/tls/certs/kafka.client.truststore.jks
      truststore_password: ----

Could anyone please help on this?

Regards

2

It probably means this is not working. Have you tried to run the elastic-agent status and inspect commands as I suggested in the other thread?

Have you looked in the agent logs ...

/opt/Elastic/Agent/elastic-agent status
/opt/Elastic/Agent/elastic-agent inspect
/opt/Elastic/Agent/elastic-agent logs
3

Hi @stephenb -

Thanks for the reply.
Yes , we have tried running the two commands that you shared.

/opt/Elastic/Agent/elastic-agent status

┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
└─ elastic-agent
   └─ status: (HEALTHY) Running

/opt/Elastic/Agent/elastic-agent inspect

agent:
  logging:
    files:
      keepfiles: 7
      name: elastic-agent
      path: /var/log/elastic-agent/
      permissions: 420
    level: info
    to_files: true
inputs:
- config:
    headers:
      Authorization: Bearer 2----1:c----1
    method: GET
    response_format: json
    url: https://demo.com/monitoring/logs?source=am-everything
  id: api-logs-input
  schedule: '@every 1m'
  type: httpjson
  use_output: default
outputs:
  default:
    hosts:
    - b-1.com:9094
    - b-2.com:9094
    - b-3.com:9094
    ssl:
      enabled: true
      truststore_location: /etc/pki/tls/certs/kafka.client.truststore.jks
      truststore_password: ----
    topic: test_app_topic
    type: kafka
agent:
  logging:
    files:
      keepfiles: 7
      name: elastic-agent
      path: /var/log/elastic-agent/
      permissions: 420
    level: info
    to_files: true
inputs:
- config:
    headers:
      Authorization: Bearer 2----1:c----1
    method: GET
    response_format: json
    url: https://demo.com/monitoring/logs?source=am-everything
  id: api-logs-input
  schedule: '@every 1m'
  type: httpjson
  use_output: default
outputs:
  default:
    hosts:
    - b-1.com:9094
    - b-2.com:9094
    - b-3.com:9094
    ssl:
      enabled: true
      truststore_location: /etc/pki/tls/certs/kafka.client.truststore.jks
      truststore_password: ----
    topic: test_app_topic
    type: kafka

/opt/Elastic/Agent/elastic-agent logs

{"log.level":"info","@timestamp":"2024-12-17T22:06:42.689Z","message":"Non-zero metrics in the last 30s","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cpu":{"system":{"ticks":3970},"total":{"ticks":11880,"time":{"ms":10},"value":11880},"user":{"ticks":7910,"time":{"ms":10}}},"handles":{"limit":{"hard":4096,"soft":4096},"open":19},"info":{"ephemeral_id":"0722f2ff-715f-4ca6-837f-4fd7f46ab226","uptime":{"ms":23460048},"version":"8.15.2"},"memstats":{"gc_next":75751488,"memory_alloc":36659448,"memory_total":966782768,"rss":129503232},"runtime":{"goroutines":68}},"filebeat":{"events":{"active":0,"added":4,"done":4},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0},"write":{"latency":{"histogram":{"count":0,"max":0,"mean":0,"median":0,"min":0,"p75":0,"p95":0,"p99":0,"p999":0,"stddev":0}}}},"pipeline":{"clients":1,"events":{"active":0,"filtered":4,"total":4},"queue":{"filled":{"bytes":0,"events":0,"pct":0},"max_bytes":0,"max_events":3200}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0,"15":0.05,"5":0.01,"norm":{"1":0,"15":0.0125,"5":0.0025}}}}},"log.logger":"monitoring","log.origin":{"file.line":192,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"service.name":"filebeat","ecs.version":"1.6.0"}

Could you please suggest on what needs to be done on this issue?

Regards

4

There should be considerably more logs than just that one log line, can you confirm?

It may be helpful to send the agent monitoring data (the Filebeat and metricbeat data you're seeing) to a different topic to reduce the amount of noise while troubleshooting: Configure outputs for standalone Elastic Agents | Fleet and Elastic Agent Guide [8.17] | Elastic

5

Hi @strawgate ,

Yes, there are 10 logs generated after using the command
" /opt/Elastic/Agent/elastic-agent logs" . Below are a few of them:

{"log.level":"info","@timestamp":"2024-12-18T00:23:12.456Z","message":"Non-zero metrics in the last 30s","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.logger":"monitoring","log.origin":{"file.line":192,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"service.name":"filebeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cpu":{"system":{"ticks":4240,"time":{"ms":10}},"total":{"ticks":12010,"time":{"ms":10},"value":12010},"user":{"ticks":7770}},"handles":{"limit":{"hard":4096,"soft":4096},"open":14},"info":{"ephemeral_id":"c2951da8-b83d-4085-adbb-8332d63f513a","uptime":{"ms":31650068},"version":"8.15.2"},"memstats":{"gc_next":73427976,"memory_alloc":34428624,"memory_total":972370520,"rss":122867712},"runtime":{"goroutines":42}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"write":{"latency":{"histogram":{"count":0,"max":0,"mean":0,"median":0,"min":0,"p75":0,"p95":0,"p99":0,"p999":0,"stddev":0}}}},"pipeline":{"clients":0,"events":{"active":0},"queue":{"filled":{"bytes":0,"events":0,"pct":0},"max_bytes":0,"max_events":3200}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0,"15":0.05,"5":0.01,"norm":{"1":0,"15":0.0125,"5":0.0025}}}}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T00:23:12.552Z","message":"Non-zero metrics in the last 30s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cpu":{"system":{"ticks":4530},"total":{"ticks":18410,"value":18410},"user":{"ticks":13880}},"handles":{"limit":{"hard":4096,"soft":4096},"open":22},"info":{"ephemeral_id":"dd0b456d-43ab-4bc3-9a4d-845a7b2dd273","uptime":{"ms":31650072},"version":"8.15.2"},"memstats":{"gc_next":145075088,"memory_alloc":70381360,"memory_total":977882992,"rss":161382400},"runtime":{"goroutines":91}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":7}},"output":{"events":{"acked":6,"active":0,"batches":1,"total":6},"write":{"latency":{"histogram":{"count":0,"max":0,"mean":0,"median":0,"min":0,"p75":0,"p95":0,"p99":0,"p999":0,"stddev":0}}}},"outputs":{"kafka":{"bytes_read":355,"bytes_write":5717}},"pipeline":{"clients":7,"events":{"active":0,"published":6,"total":6},"queue":{"acked":6,"added":{"events":6},"consumed":{"events":6},"filled":{"bytes":0,"events":0,"pct":0},"max_bytes":0,"max_events":3200,"removed":{"events":6}}}},"metricbeat":{"http":{"json":{"events":6,"success":6}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0,"15":0.05,"5":0.01,"norm":{"1":0,"15":0.0125,"5":0.0025}}}}},"log.logger":"monitoring","log.origin":{"file.line":192,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T00:23:12.689Z","message":"Non-zero metrics in the last 30s","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cpu":{"system":{"ticks":5340,"time":{"ms":10}},"total":{"ticks":15930,"time":{"ms":10},"value":15930},"user":{"ticks":10590}},"handles":{"limit":{"hard":4096,"soft":4096},"open":19},"info":{"ephemeral_id":"0722f2ff-715f-4ca6-837f-4fd7f46ab226","uptime":{"ms":31650049},"version":"8.15.2"},"memstats":{"gc_next":75762112,"memory_alloc":35962328,"memory_total":1279940672,"rss":129662976},"runtime":{"goroutines":68}},"filebeat":{"events":{"active":0,"added":4,"done":4},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0},"write":{"latency":{"histogram":{"count":0,"max":0,"mean":0,"median":0,"min":0,"p75":0,"p95":0,"p99":0,"p999":0,"stddev":0}}}},"pipeline":{"clients":1,"events":{"active":0,"filtered":4,"total":4},"queue":{"filled":{"bytes":0,"events":0,"pct":0},"max_bytes":0,"max_events":3200}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0,"15":0.05,"5":0.01,"norm":{"1":0,"15":0.0125,"5":0.0025}}}}},"log.logger":"monitoring","log.origin":{"file.line":192,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T00:23:13.027Z","message":"Non-zero metrics in the last 30s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cpu":{"system":{"ticks":4550},"total":{"ticks":18720,"time":{"ms":20},"value":18720},"user":{"ticks":14170,"time":{"ms":20}}},"handles":{"limit":{"hard":4096,"soft":4096},"open":19},"info":{"ephemeral_id":"e581ee85-ad88-49cf-a420-30332d8b4538","uptime":{"ms":31650069},"version":"8.15.2"},"memstats":{"gc_next":156200752,"memory_alloc":75454600,"memory_total":979246232,"rss":165187584},"runtime":{"goroutines":73}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":4}},"output":{"events":{"acked":4,"active":0,"batches":1,"total":4},"write":{"latency":{"histogram":{"count":0,"max":0,"mean":0,"median":0,"min":0,"p75":0,"p95":0,"p99":0,"p999":0,"stddev":0}}}},"outputs":{"kafka":{"bytes_read":284,"bytes_write":5740}},"pipeline":{"clients":4,"events":{"active":0,"published":4,"total":4},"queue":{"acked":4,"added":{"events":4},"consumed":{"events":4},"filled":{"bytes":0,"events":0,"pct":0},"max_bytes":0,"max_events":3200,"removed":{"events":4}}}},"metricbeat":{"beat":{"stats":{"events":4,"success":4}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0,"15":0.05,"5":0.01,"norm":{"1":0,"15":0.0125,"5":0.0025}}}}},"log.logger":"monitoring","log.origin":{"file.line":192,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"service.name":"metricbeat","ecs.version":"1.6.0"}
6

Please review the logs on the disk under /var/log/elastic-agent

7

Hi @strawgate - The same logs are coming in "/var/log/elastic-agent/" as in " /opt/Elastic/Agent/elastic-agent logs ".

8

You will need to share considerably more logs than those four and there should be many more than ten logs in that directory.

Can you restart elastic agent and share all the logs from the first minute after restarting elastic agent?

9

Hi @strawgate

I have attached the Elastic agent logs from the restart of the service.
Could not find any abnormality apart from "unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock".

Could you please check if you are able to get anything?

{"log.level":"info","@timestamp":"2024-12-18T13:56:03.427Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":647},"message":"Component state changed beat/metrics-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '11055'","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.430Z","message":"BeatV2Manager.unitListen UnitChanged.ID(beat/metrics-monitoring-metrics-monitoring-beats), UnitChanged.Type(added), UnitChanged.Trigger(4): added/feature_change_triggered","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":506,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.430Z","message":"BeatV2Manager.unitListen UnitChanged.ID(beat/metrics-monitoring), UnitChanged.Type(added), UnitChanged.Trigger(4): added/feature_change_triggered","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":506,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.479Z","message":"Home path: [/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components] Config path: [/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components] Data path: [/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/run/http/metrics-monitoring] Logs path: [/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components/logs]","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0","log.origin":{"file.line":828,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.479Z","message":"Beat ID: 69a59614-ac32-48a7-98da-67ba036d285e","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.origin":{"file.line":836,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.483Z","message":"Output reload is enabled, the beat will restart as needed on change of output config","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0","log.logger":"centralmgmt","log.origin":{"file.line":204,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.NewV2AgentManagerWithClient"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.483Z","message":"Set gc percentage to: 100","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0","log.origin":{"file.line":890,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.484Z","message":"running under elastic-agent, per-beat lockfiles disabled","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.origin":{"file.line":443,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.485Z","message":"Starting stats endpoint","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"api","log.origin":{"file.line":69,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-12-18T13:56:03.485Z","message":"add_cloud_metadata: received error failed with http status code 404","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":190,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.485Z","message":"Syscall filter successfully installed","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"seccomp","log.origin":{"file.line":125,"file.name":"seccomp/seccomp.go","function":"github.com/elastic/beats/v7/libbeat/common/seccomp.loadFilter"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.485Z","message":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/akSPbdqgaHaTY0_J01-dsfYK6JpMz2zn.sock (configured: unix:///opt/Elastic/Agent/data/tmp/akSPbdqgaHaTY0_J01-dsfYK6JpMz2zn.sock)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0","log.logger":"api","log.origin":{"file.line":71,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1"},"service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"20{"log.level":"info","@timestamp":"2024-12-18T13:56:02.534Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.watchCmd","file.name":"cmd/watch.go","file.line":68},"message":"Upgrade Watcher started","process.pid":11026,"agent.version":"8.15.2","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.535Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.watchCmd","file.name":"cmd/watch.go","file.line":76},"message":"update marker not present at '/opt/Elastic/Agent/data'","ecs.version":"1.6.0"}
[root@d1entsttlsr007 elastic-agent]# cat elastic-agent-20241218.ndjson
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.476Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.InvokeWatcher","file.name":"upgrade/rollback.go","file.line":149},"message":"Starting upgrade watcher","log":{"source":"elastic-agent"},"path":"/opt/Elastic/Agent/elastic-agent","args":["/opt/Elastic/Agent/elastic-agent","watch","--path.config","/opt/Elastic/Agent","--path.home","/opt/Elastic/Agent"],"env":[],"dir":"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.478Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.InvokeWatcher","file.name":"upgrade/rollback.go","file.line":163},"message":"Upgrade Watcher invoked","log":{"source":"elastic-agent"},"agent.upgrade.watcher.process.pid":11026,"agent.process.pid":11019,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.480Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":284},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.481Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":65},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.499Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":71},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"inputs":["entity-analytics","o365audit","kubernetes/metrics","packet","cloudbeat/cis_eks","endpoint","benchmark","cel","redis","unix","uwsgi/metrics","apm","pf-elastic-symbolizer","audit/file_integrity","gcp-pubsub","synthetics/browser","syncgateway/metrics","rabbitmq/metrics","netflow","mysql/metrics","azure/metrics","containerd/metrics","gcp/metrics","memcached/metrics","nginx/metrics","container","salesforce","logstash/metrics","system/metrics","statsd/metrics","journald","filestream","haproxy/metrics","iis/metrics","zookeeper/metrics","cometd","etw","tcp","synthetics/tcp","cloudbeat/asset_inventory_aws","pf-host-agent","cloudfoundry","lumberjack","mqtt","linux/metrics","prometheus/metrics","stan/metrics","http_endpoint","apache/metrics","kafka/metrics","cloudbeat","osquery","audit/system","log","enterprisesearch/metrics","postgresql/metrics","redis/metrics","cloudfoundry/metrics","etcd/metrics","cloudbeat/cis_gcp","pf-elastic-collector","azure-eventhub","gcs","winlog","synthetics/http","docker/metrics","elasticsearch/metrics","awsfargate/metrics","cloudbeat/cis_aws","aws-cloudwatch","docker","httpjson","kafka","syslog","activemq/metrics","nats/metrics","beat/metrics","mongodb/metrics","aws/metrics","jolokia/metrics","sql/metrics","traefik/metrics","cloudbeat/cis_azure","oracle/metrics","aws-s3","azure-blob-storage","udp","websocket","synthetics/icmp","kibana/metrics","windows/metrics","http/metrics","cloudbeat/cis_k8s","cloudbeat/vuln_mgmt_aws","fleet-server","audit/auditd","mssql/metrics","vsphere/metrics"],"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.499Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/capabilities.LoadFile","file.name":"capabilities/capabilities.go","file.line":48},"message":"Capabilities file not found in /opt/Elastic/Agent/capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.499Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":77},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.499Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":92},"message":"Loading baseline config from /opt/Elastic/Agent/elastic-agent.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.595Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.NewManager","file.name":"runtime/manager.go","file.line":180},"message":"GRPC comms socket listening at localhost:6789","log":{"source":"elastic-agent"},"address":"localhost:6789","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.595Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":148},"message":"Parsed configuration and determined agent is managed locally","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.597Z","log.logger":"control","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/control/v2/server.(*Server).Start","file.name":"server/server.go","file.line":88},"message":"GRPC control socket listening at unix:///opt/Elastic/Agent/elastic-agent.sock","log":{"source":"elastic-agent"},"address":"unix:///opt/Elastic/Agent/elastic-agent.sock","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.597Z","log.logger":"control","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":343},"message":"Created control socket symlink /run/elastic-agent.sock -> /opt/Elastic/Agent/elastic-agent.sock; allowing unix:///run/elastic-agent.sock connection","log":{"source":"elastic-agent"},"path":"/opt/Elastic/Agent/elastic-agent.sock","link":"/run/elastic-agent.sock","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.597Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.(*periodic).work","file.name":"application/periodic.go","file.line":93},"message":"Configuration changes detected","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.598Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).Run","file.name":"runtime/manager.go","file.line":246},"message":"Starting grpc control protocol listener on port 6789 with max_message_size 104857600","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.599Z","log.logger":"composable.providers.docker","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/docker.(*dynamicProvider).Run","file.name":"docker/docker.go","file.line":44},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.599Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":123},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.599Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:true, MetricsPeriod:\"\", LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc001738e10), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:\"\", APIKey:\"\", SecretToken:\"\", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.599Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":90},"message":"creating monitoring API with cfg api.Config{Enabled:true, Host:\"http://localhost:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.599Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.600Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.705Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1272},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.782Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":627},"message":"Spawned new component httpjson-default: Starting: spawned pid '11035'","log":{"source":"elastic-agent"},"component":{"id":"httpjson-default","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.782Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":634},"message":"Spawned new unit httpjson-default-httpjson: Starting: spawned pid '11035'","log":{"source":"elastic-agent"},"component":{"id":"httpjson-default","state":"STARTING"},"unit":{"id":"httpjson-default-httpjson","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.783Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":634},"message":"Spawned new unit httpjson-default: Starting: spawned pid '11035'","log":{"source":"elastic-agent"},"component":{"id":"httpjson-default","state":"STARTING"},"unit":{"id":"httpjson-default","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.893Z","message":"Home path: [/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components] Config path: [/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components] Data path: [/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/run/httpjson-default] Logs path: [/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components/logs]","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.origin":{"file.line":828,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.893Z","message":"Beat ID: 2a39fc69-5660-4a30-a81b-ac830ba2da6d","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":836,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.897Z","message":"Output reload is enabled, the beat will restart as needed on change of output config","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"centralmgmt","log.origin":{"file.line":204,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.NewV2AgentManagerWithClient"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.897Z","message":"Set gc percentage to: 100","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.origin":{"file.line":890,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.899Z","message":"running under elastic-agent, per-beat lockfiles disabled","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.origin":{"file.line":443,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.899Z","message":"Starting stats endpoint","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"ecs.version":"1.6.0","log.logger":"api","log.origin":{"file.line":69,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.899Z","message":"Syscall filter successfully installed","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.logger":"seccomp","log.origin":{"file.line":125,"file.name":"seccomp/seccomp.go","function":"github.com/elastic/beats/v7/libbeat/common/seccomp.loadFilter"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.899Z","message":"Beat info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.logger":"beat","log.origin":{"file.line":1385,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"filebeat","system_info":{"beat":{"path":{"config":"/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components","data":"/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/run/httpjson-default","home":"/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components","logs":"/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components/logs"},"type":"filebeat","uuid":"2a39fc69-5660-4a30-a81b-ac830ba2da6d"},"ecs.version":"1.6.0"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.899Z","message":"Build info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"service.name":"filebeat","system_info":{"build":{"commit":"26daf71e4ec87172523af7f0e916cba9f79dc0d0","libbeat":"8.15.2","time":"2024-09-19T09:24:35.000Z","version":"8.15.2"},"ecs.version":"1.6.0"},"log.logger":"beat","log.origin":{"file.line":1394,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-12-18T13:56:02.899Z","message":"add_cloud_metadata: received error failed with http status code 404","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.origin":{"file.line":190,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.899Z","message":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/VVUqAPuu8p5QR6A6OTFimXP6SXZxshT_.sock (configured: unix:///opt/Elastic/Agent/data/tmp/VVUqAPuu8p5QR6A6OTFimXP6SXZxshT_.sock)","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.logger":"api","log.origin":{"file.line":71,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.899Z","message":"Go runtime info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"service.name":"filebeat","system_info":{"ecs.version":"1.6.0","go":{"arch":"amd64","max_procs":4,"os":"linux","version":"go1.22.6"}},"log.logger":"beat","log.origin":{"file.line":1397,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-12-18T13:56:02.900Z","message":"add_cloud_metadata: received error failed with http status code 404","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":190,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-12-18T13:56:02.900Z","message":"add_cloud_metadata: received error failed with http status code 404","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.origin":{"file.line":190,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-12-18T13:56:02.900Z","message":"add_cloud_metadata: received error failed with http status code 404","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":190,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.900Z","message":"Host info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.logger":"beat","log.origin":{"file.line":1403,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"filebeat","system_info":{"ecs.version":"1.6.0","host":{"architecture":"x86_64","boot_time":"2024-07-18T01:02:29+01:00","containerized":false,"id":"69f417b1a15a420eb9acfd36802ff697","ip":["127.0.0.1","::1","10.178.2.12","fe80::477:70ff:fefe:3fdb"],"kernel_version":"3.10.0-1160.118.1.el7.x86_64","mac":["06:77:70:fe:3f:db"],"name":"d1entsttlsr007.europe.easyjet.local","native_architecture":"","os":{"codename":"Maipo","family":"redhat","major":7,"minor":7,"name":"Red Hat Enterprise Linux Server","patch":0,"platform":"rhel","type":"linux","version":"7.7 (Maipo)"},"timezone":"GMT","timezone_offset_sec":0}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.901Z","message":"Process info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.logger":"beat","log.origin":{"file.line":1432,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"filebeat","system_info":{"ecs.version":"1.6.0","process":{"capabilities":{"ambient":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"]},"cwd":"/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/run/httpjson-default","exe":"/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components/agentbeat","name":"agentbeat","pid":11035,"ppid":11019,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2024-12-18T13:56:02.530Z"}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.901Z","message":"Setup Beat: filebeat; Version: 8.15.2","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":341,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.902Z","message":"Output is configured through Central Management","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.origin":{"file.line":373,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.902Z","message":"add_cloud_metadata: hosting provider type detected as openstack, metadata={\"cloud\":{\"availability_zone\":\"eu-west-1a\",\"instance\":{\"id\":\"i-0528118076d76fe10\",\"name\":\"ip-10-178-2-12.europe.easyjet.local\"},\"machine\":{\"type\":\"c5.xlarge\"},\"provider\":\"openstack\",\"service\":{\"name\":\"Nova\"}}}","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":104,"file.name":"add_cloud_metadata/add_cloud_metadata.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.905Z","message":"Beat name: d1entsttlsr007.europe.easyjet.local","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"publisher","log.origin":{"file.line":105,"file.name":"pipeline/module.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.905Z","message":"Enabled modules/filesets: ","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.origin":{"file.line":136,"file.name":"fileset/modules.go","function":"github.com/elastic/beats/v7/filebeat/fileset.newModuleRegistry"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"modules","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.905Z","message":"Starting metrics logging every 30s","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.origin":{"file.line":150,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"monitoring","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.905Z","message":"filebeat start running.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.origin":{"file.line":540,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.905Z","message":"Finished loading transaction log file for '/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/run/httpjson-default/registry/filebeat'. Active transaction id=0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.origin":{"file.line":134,"file.name":"memlog/store.go","function":"github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.openStore"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-12-18T13:56:02.905Z","message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.origin":{"file.line":331,"file.name":"beater/filebeat.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.906Z","message":"States Loaded from registrar: 0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.logger":"registrar","log.origin":{"file.line":107,"file.name":"registrar/registrar.go","function":"github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).loadStates"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.906Z","message":"Loading Inputs: 0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.logger":"crawler","log.origin":{"file.line":71,"file.name":"beater/crawler.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Start"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.906Z","message":"Loading and starting Inputs completed. Enabled inputs: 0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"crawler","log.origin":{"file.line":106,"file.name":"beater/crawler.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Start"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.913Z","log.logger":"component.runtime.httpjson-default","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).CheckinV2","file.name":"runtime/manager.go","file.line":715},"message":"control checkin v2 protocol has chunking enabled","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.914Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":647},"message":"Component state changed httpjson-default (STARTING->HEALTHY): Healthy: communicating with pid '11035'","log":{"source":"elastic-agent"},"component":{"id":"httpjson-default","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.915Z","message":"BeatV2Manager.unitListen UnitChanged.ID(httpjson-default-httpjson), UnitChanged.Type(added), UnitChanged.Trigger(4): added/feature_change_triggered","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":506,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.915Z","message":"BeatV2Manager.unitListen UnitChanged.ID(httpjson-default), UnitChanged.Type(added), UnitChanged.Trigger(4): added/feature_change_triggered","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.origin":{"file.line":506,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"centralmgmt.V2-manager","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.933Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).checkAndLogUpdate","file.name":"coordinator/coordinator.go","file.line":1478},"message":"component model updated","log":{"source":"elastic-agent"},"changes":{"components":{"count":4},"outputs":{}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:02.933Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1272},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.065Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":627},"message":"Spawned new component filestream-monitoring: Starting: spawned pid '11045'","log":{"source":"elastic-agent"},"component":{"id":"filestream-monitoring","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.065Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":634},"message":"Spawned new unit filestream-monitoring-filestream-monitoring-agent: Starting: spawned pid '11045'","log":{"source":"elastic-agent"},"component":{"id":"filestream-monitoring","state":"STARTING"},"unit":{"id":"filestream-monitoring-filestream-monitoring-agent","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.065Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":634},"message":"Spawned new unit filestream-monitoring: Starting: spawned pid '11045'","log":{"source":"elastic-agent"},"component":{"id":"filestream-monitoring","state":"STARTING"},"unit":{"id":"filestream-monitoring","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.172Z","message":"Home path: [/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components] Config path: [/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components] Data path: [/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/run/filestream-monitoring] Logs path: [/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components/logs]","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":828,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.172Z","message":"Beat ID: c0933cde-64a6-4079-8667-cc152b847d5c","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":836,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.175Z","message":"Output reload is enabled, the beat will restart as needed on change of output config","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"centralmgmt","log.origin":{"file.line":204,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.NewV2AgentManagerWithClient"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.175Z","message":"Set gc percentage to: 100","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":890,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.177Z","message":"running under elastic-agent, per-beat lockfiles disabled","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":443,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.177Z","message":"Starting stats endpoint","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"ecs.version":"1.6.0","log.logger":"api","log.origin":{"file.line":69,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.178Z","message":"Syscall filter successfully installed","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":125,"file.name":"seccomp/seccomp.go","function":"github.com/elastic/beats/v7/libbeat/common/seccomp.loadFilter"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"seccomp","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.178Z","message":"Beat info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"beat","log.origin":{"file.line":1385,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"filebeat","system_info":{"beat":{"path":{"config":"/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components","data":"/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/run/filestream-monitoring","home":"/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components","logs":"/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components/logs"},"type":"filebeat","uuid":"c0933cde-64a6-4079-8667-cc152b847d5c"},"ecs.version":"1.6.0"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.178Z","message":"Build info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"beat","log.origin":{"file.line":1394,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"filebeat","system_info":{"build":{"commit":"26daf71e4ec87172523af7f0e916cba9f79dc0d0","libbeat":"8.15.2","time":"2024-09-19T09:24:35.000Z","version":"8.15.2"},"ecs.version":"1.6.0"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.178Z","message":"Go runtime info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"beat","log.origin":{"file.line":1397,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"filebeat","system_info":{"ecs.version":"1.6.0","go":{"arch":"amd64","max_procs":4,"os":"linux","version":"go1.22.6"}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.178Z","message":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/xTEtpJ7117ppc6OYvJCaYHbDW8mLjXGe.sock (configured: unix:///opt/Elastic/Agent/data/tmp/xTEtpJ7117ppc6OYvJCaYHbDW8mLjXGe.sock)","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"api","log.origin":{"file.line":71,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.179Z","message":"Host info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"beat","log.origin":{"file.line":1403,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"filebeat","system_info":{"ecs.version":"1.6.0","host":{"architecture":"x86_64","boot_time":"2024-07-18T01:02:29+01:00","containerized":false,"id":"69f417b1a15a420eb9acfd36802ff697","ip":["127.0.0.1","::1","10.178.2.12","fe80::477:70ff:fefe:3fdb"],"kernel_version":"3.10.0-1160.118.1.el7.x86_64","mac":["06:77:70:fe:3f:db"],"name":"d1entsttlsr007.europe.easyjet.local","native_architecture":"","os":{"codename":"Maipo","family":"redhat","major":7,"minor":7,"name":"Red Hat Enterprise Linux Server","patch":0,"platform":"rhel","type":"linux","version":"7.7 (Maipo)"},"timezone":"GMT","timezone_offset_sec":0}},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-12-18T13:56:03.179Z","message":"add_cloud_metadata: received error failed with http status code 404","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":190,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-12-18T13:56:03.179Z","message":"add_cloud_metadata: received error failed with http status code 404","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":190,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-12-18T13:56:03.179Z","message":"add_cloud_metadata: received error failed with http status code 404","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":190,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.179Z","message":"Process info","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":1432,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo"},"service.name":"filebeat","system_info":{"ecs.version":"1.6.0","process":{"capabilities":{"ambient":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"]},"cwd":"/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/run/filestream-monitoring","exe":"/opt/Elastic/Agent/data/elastic-agent-8.15.2-621bbc/components/agentbeat","name":"agentbeat","pid":11045,"ppid":11019,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2024-12-18T13:56:02.810Z"}},"log.logger":"beat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.179Z","message":"Setup Beat: filebeat; Version: 8.15.2","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":341,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-12-18T13:56:03.180Z","message":"Output is configured through Central Management","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":373,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater"},"ecs.version":"1.6.0"}
10

Are your API requests working?

You may need to enable the request tracer to see if there is any issue.

No Idea how you do that on Stand Alone agents, I only use managed agents, but the documentation for the input is here.

Look for the request.tracer.enable and request.tracer.filename settings.

11

Yes, I am able to CURL the API logs in this machine. But unable to get any of these logs to Kafka.

I have tried using both these settings : request.tracer.enable and request.tracer.filename . Is this syntactically correct?

inputs:
- config:
    headers:
      x-api-key: 2---
      x-api-secret: c---d1
    method: GET
    response_format: json
    url: https://abc.forgeblocks.com/monitoring/logs?source=am-everything
  processors:
  - foreach:
      field: result
      processor:
      - decode_json_fields:
          add_error_key: true
          fields:
          - payload
          overwrite_keys: true
          target: ""
      - rename:
          fields:
          - from: timestamp
            to: '@timestamp'
          ignore_missing: true
      - add_fields:
          fields:
            log_source: api-logs
          target: ""
  **request.tracer.enabled: true**
**  request.tracer.filename: /var/log/elastic-agent/http-request-trace-*.ndjson**
  schedule: '@every 1m'
  type: httpjson
  use_output: default
12

The logs shared appear to just be for the first 2 seconds of agent running. Can you share the logs from at least the first minute?

Preferably the first 2 minutes as your interval on your httpjson input is 1 minute

13

I have sent that to you in a message, separately. As it is not allowing us to paste all the contents here.

14

It looks like you are configuring this as if it were filebeat but the Elastic Agent standalone configuration is different. The reference configuration for standalone agent looks like this:

inputs:
  # Collect custom data from REST API's: Collect custom data from REST API's
  - id: generic-httpjson
    type: httpjson
    streams:
      # Custom API Input: Collect custom data from REST API's
      - id: httpjson-httpjson.generic
        data_stream:
          dataset: httpjson.generic
        config_version: 2
        interval: 1m
        request.url: 'https://server.example.com:8089/api'
        request.method: GET
        request.tracer.filename: ../../logs/httpjson/http-request-trace-*.ndjson
        request.tracer.maxbackups: 5
        tags:
          - forwarded
        publisher_pipeline.disable_host: true
        # auth.basic.user: <USERNAME> # Basic Auth Username: The username to be used with Basic Auth headers
        # auth.basic.password: <PASSWORD> # Basic Auth Password: The password to be used with Basic Auth headers
        # pipeline: <PIPELINE>
        #  # Ingest Pipeline: The Ingest Node pipeline ID to be used by the integration.
        # request.encode_as: <REQUEST_ENCODE_AS> # Request Encode As: ContentType used for encoding the request body. If set it will force the encoding in the specified format regardless of the Content-Type header value.
        # request.timeout: <REQUEST_TIMEOUT> # Request Timeout: Duration before declaring that the HTTP client connection has timed out. Valid time units are ns, us, ms, s, m, h. Default is "30"s.
        # request.proxy_url: <REQUEST_PROXY_URL> # Request Proxy: This specifies proxy configuration in the form of `http[s]://<user>:<password>@<server name/ip>:<port>`.
        # request.retry.max_attempts: <REQUEST_RETRY_MAX_ATTEMPTS> # Request Retry Max Attempts: The maximum number of retries for the HTTP client. Default is "5".
        # request.retry.wait_min: <REQUEST_RETRY_WAIT_MIN> # Request Retry Wait Min: The minimum time to wait before a retry is attempted. Default is "1s".
        # request.retry.wait_max: <REQUEST_RETRY_WAIT_MAX> # Request Retry Wait Max: The maximum time to wait before a retry is attempted. Default is "60s".
        # request.redirect.forward_headers: <REQUEST_REDIRECT_FORWARD_HEADERS> # Request Redirect Forward Headers: When set to true request headers are forwarded in case of a redirect. Default is "false".
        # request.redirect.headers_ban_list:
        #  - <REQUEST_REDIRECT_HEADERS_BAN_LIST> # Request Redirect Headers Ban List: When Redirect Forward Headers is set to true, all headers except the ones defined in this list will be forwarded. All headers are forwarded by default.
        # request.redirect.max_redirects: <REQUEST_REDIRECT_MAX_REDIRECTS> # Request Redirect Max Redirects: The maximum number of redirects to follow for a request. Default is "10".
        # request.rate_limit.limit: <REQUEST_RATE_LIMIT_LIMIT> # Request Rate Limit: The value of the response that specifies the total limit. It is defined with a Go template value.
        # request.rate_limit.reset: <REQUEST_RATE_LIMIT_RESET> # Request Rate Limit Reset: The value of the response that specifies the epoch time when the rate limit will reset. It is defined with a Go template value.
        # request.rate_limit.remaining: <REQUEST_RATE_LIMIT_REMAINING> # Request Rate Limit Remaining: The value of the response that specifies the remaining quota of the rate limit. It is defined with a Go template value.
        # response.decode_as: <RESPONSE_DECODE_AS>
        #  # Response decode settings: ContentType used for decoding the response body. Supported values: application/json, application/x-ndjson. By default it will use what is in the response Content-Type header.
        # response.request_body_on_pagination: <RESPONSE_REQUEST_BODY_ON_PAGINATION>
        #  # Include request body on Pagination: If set to true, the values in request.body are sent with pagination requests.
        # processors:

If you go into Fleet and make an Agent Policy with the Custom API integration configured how you want it configured you can click Actions -> View Policy to view the configuration that is provided to the agent:

Screenshot 2024-12-18 at 10.07.05 AM
Screenshot 2024-12-18 at 10.07.05 AM1300×866 80.1 KB

1 Like
15

Thank you so much @strawgate and @leandrojmp. Setting up the config in the above format helped me get the logs through to Kafka .

2 Likes