# Unable to load data from filebeat using input of AWS S3 **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [March 8, 2023, 11:55am UTC](https://discuss.elastic.co/t/unable-to-load-data-from-filebeat-using-input-of-aws-s3/327268 "2023-03-08T11:55:02Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![vivekd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivekd/32/118194_2.png) [@vivekd](https://discuss.elastic.co/u/vivekd)\ **Post date:** [March 8, 2023, 11:55am UTC](https://discuss.elastic.co/t/unable-to-load-data-from-filebeat-using-input-of-aws-s3/327268/1 "2023-03-08T11:55:02Z")
Hi Team, We are unable to load data from filebeat using input of AWS S3. We have followed the below article for setting the filebeat configuration. > **[AWS S3 input | Filebeat Reference \[8.6\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html)** Post configuration of filebeat, Please let us know on how to create dashboard for viewing and analyzing the data on kibana. I am new to Elasticsearch. Any help/guidance will be highly appreciated. Regards Vivek ---
**Author:** ![vivekd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivekd/32/118194_2.png) [@vivekd](https://discuss.elastic.co/u/vivekd)\ **Post date:** [March 14, 2023, 12:05pm UTC](https://discuss.elastic.co/t/unable-to-load-data-from-filebeat-using-input-of-aws-s3/327268/2 "2023-03-14T12:05:18Z")
Hi Team, Please provide me guidance on configuring S3 input for file beat to load data to elasticsearch. Regards Vivek ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 14, 2023, 1:09pm UTC](https://discuss.elastic.co/t/unable-to-load-data-from-filebeat-using-input-of-aws-s3/327268/3 "2023-03-14T13:09:52Z")
You need to share your `filebeat` configuration and any logs that indicates an error or the issue you are having. ---
**Author:** ![vivekd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivekd/32/118194_2.png) [@vivekd](https://discuss.elastic.co/u/vivekd)\ **Post date:** [March 14, 2023, 3:38pm UTC](https://discuss.elastic.co/t/unable-to-load-data-from-filebeat-using-input-of-aws-s3/327268/4 "2023-03-14T15:38:50Z")
Hi, Kindly find the filebeat configuration below . ```auto filebeat.inputs: - type: aws-s3 queue_url: https://sqs.us-east-1.amazonaws.com/QueueforElasticSearch access_key_id: 'XXXXXXXXXXXXXXX' secret_access_key: 'XXXXXXXXXXXXXXX' # Unique ID among all inputs, an ID is required. id: my-filestream-id # Change to true to enable this input configuration. enabled: false # Paths that should be crawled and fetched. Glob based paths. paths: #- /var/log/*.log #- c:\programdata\elasticsearch\logs\* # Exclude lines. A list of regular expressions to match. It drops the lines that are # matching any regular expression from the list. # Line filtering happens after the parsers pipeline. If you would like to filter lines # before parsers, use include_message parser. #exclude_lines: ['^DBG'] # Include lines. A list of regular expressions to match. It exports the lines that are # matching any regular expression from the list. # Line filtering happens after the parsers pipeline. If you would like to filter lines # before parsers, use include_message parser. #include_lines: ['^ERR', '^WARN'] # Exclude files. A list of regular expressions to match. Filebeat drops the files that # are matching any regular expression from the list. By default, no files are dropped. #prospector.scanner.exclude_files: ['.gz$'] # Optional additional fields. These fields can be freely picked # to add additional information to the crawled log files for filtering #fields: # level: debug # review: 1 filebeat.config.modules: # Glob pattern for configuration loading path: ${path.config}/modules.d/*.yml # Set to true to enable config reloading reload.enabled: false # Period on which files under path should be checked for changes #reload.period: 10s setup.template.settings: index.number_of_shards: 1 #index.codec: best_compression #_source.enabled: false setup.kibana: # Kibana Host # Scheme and port can be left out and will be set to the default (http and 5601) # In case you specify and additional path, the scheme is required: http://localhost:5601/path # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601 #host: "localhost:5601" # Kibana Space ID # ID of the Kibana Space into which the dashboards should be loaded. By default, # the Default Space will be used. #space.id: output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"] # Protocol - either `http` (default) or `https`. #protocol: "https" # Authentication credentials - either API key or username/password. #api_key: "id:api_key" #username: "elastic" #password: "changeme" # The Logstash hosts #hosts: ["localhost:5044"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" processors: - add_host_metadata: when.not.contains.tags: forwarded - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~ # Set to true to enable instrumentation of filebeat. #enabled: false # Environment in which filebeat is running on (eg: staging, production, etc.) #environment: "" # APM Server hosts to report instrumentation results to. #hosts: # - http://localhost:8200 # API Key for the APM Server(s). # If api_key is set then secret_token will be ignored. #api_key: # Secret token for the APM Server(s). #secret_token: ``` =============================================== The below is kibana.yml configuration file. ```auto server.port: 5601 server.host: "localhost" logging: appenders: file: type: file fileName: /var/log/kibana/kibana.log layout: type: json root: appenders: - default - file pid.file: /run/kibana/kibana.pid elasticsearch.hosts: ['https://localhost:9200'] elasticsearch.username: kibana_system elasticsearch.password: XXXXXXXXXXX elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1663157228267.crt] xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://localhost:9200'], ca_trusted_fingerprint: c458aa0c9be0d5241e6c364f7082da6305e96ab22c4ae60beaf345de0b64429f}] ``` ====================================================== Below is the sample kibana logs. ```auto {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T14:20:03.601+00:00","message":"Running Fleet Usage telemetry send task","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"b085775168ca8607"}} {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T14:20:04.093+00:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"e7807064fc6c6574"}} {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T14:20:09.719+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"7a3e8fa969076e7e"}} {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T14:35:12.794+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"3e0a9d1f97f2c3f8"}} {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T14:50:12.866+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"f27f54e36013f348"}} {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T15:05:12.915+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"f1aa78b9521141c7"}} {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T15:20:03.820+00:00","message":"Running Fleet Usage telemetry send task","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"82a080df9d7d23c2"}} {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T15:20:04.445+00:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"12b6c38a7a9d0430"}} {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-14T15:20:12.866+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2388},"trace":{"id":"012a4bb8f7b889b8ac3984dd45fb9fb2"},"transaction":{"id":"1733adface67062c"}} ``` ================================================= Kindly review these and let me know what is missing here. Regards Vivek ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 14, 2023, 3:40pm UTC](https://discuss.elastic.co/t/unable-to-load-data-from-filebeat-using-input-of-aws-s3/327268/5 "2023-03-14T15:40:29Z")
> [@vivekd](#): > > ```auto > # Change to true to enable this input configuration. > enabled: false > > ``` You have `enabled` as _false_, you need to change this to _true_ to enable this input as mentioned in the configuration file. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 11, 2023, 3:40pm UTC](https://discuss.elastic.co/t/unable-to-load-data-from-filebeat-using-input-of-aws-s3/327268/6 "2023-04-11T15:40:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.