Unable to load data from filebeat using input of AWS S3

Elastic Stack Beats
1

Hi Team,

We are unable to load data from filebeat using input of AWS S3. We have followed the below article for setting the filebeat configuration.

Post configuration of filebeat, Please let us know on how to create dashboard for viewing and analyzing the data on kibana.

I am new to Elasticsearch. Any help/guidance will be highly appreciated.

Regards
Vivek

2

Hi Team,

Please provide me guidance on configuring S3 input for file beat to load data to elasticsearch.

Regards
Vivek

3

You need to share your filebeat configuration and any logs that indicates an error or the issue you are having.

4

Hi,

Kindly find the filebeat configuration below .

filebeat.inputs:
- type: aws-s3
  queue_url: https://sqs.us-east-1.amazonaws.com/QueueforElasticSearch
  access_key_id: 'XXXXXXXXXXXXXXX'
  secret_access_key: 'XXXXXXXXXXXXXXX'
  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id
  # Change to true to enable this input configuration.
  enabled: false
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  #exclude_lines: ['^DBG']
  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  #include_lines: ['^ERR', '^WARN']
  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #prospector.scanner.exclude_files: ['.gz$']
  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1
filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml
  # Set to true to enable config reloading
  reload.enabled: false
  # Period on which files under path should be checked for changes
  #reload.period: 10s
setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false
setup.kibana:
  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"
  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  # Protocol - either `http` (default) or `https`.
  #protocol: "https"
  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"
  # The Logstash hosts
  #hosts: ["localhost:5044"]
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"
  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
    # Set to true to enable instrumentation of filebeat.
    #enabled: false
    # Environment in which filebeat is running on (eg: staging, production, etc.)
    #environment: ""
    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200
    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:
    # Secret token for the APM Server(s).
    #secret_token:

===============================================

The below is kibana.yml configuration file.

server.port: 5601
server.host: "localhost"
logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file
pid.file: /run/kibana/kibana.pid
elasticsearch.hosts: ['https://localhost:9200']
elasticsearch.username: kibana_system
elasticsearch.password: XXXXXXXXXXX
elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1663157228267.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://localhost:9200'], ca_trusted_fingerprint: c458aa0c9be0d5241e6c364f7082da6305e96ab22c4ae60beaf345de0b64429f}]

======================================================

Below is the sample kibana logs.

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T14:20:03.601+00:00","message":"Running Fleet Usage telemetry send task","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"b085775168ca8607"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T14:20:04.093+00:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"e7807064fc6c6574"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T14:20:09.719+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"7a3e8fa969076e7e"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T14:35:12.794+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"3e0a9d1f97f2c3f8"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T14:50:12.866+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"f27f54e36013f348"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T15:05:12.915+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"f1aa78b9521141c7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T15:20:03.820+00:00","message":"Running Fleet Usage telemetry send task","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"82a080df9d7d23c2"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T15:20:04.445+00:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"12b6c38a7a9d0430"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T15:20:12.866+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"1733adface67062c"}}

=================================================

Kindly review these and let me know what is missing here.

Regards
Vivek

5

You have enabled as false, you need to change this to true to enable this input as mentioned in the configuration file.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.