# Elastic Cloud and Filebeat setup **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [October 5, 2021, 1:29pm UTC](https://discuss.elastic.co/t/elastic-cloud-and-filebeat-setup/285927 "2021-10-05T13:29:27Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Sakshee](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@Sakshee](https://discuss.elastic.co/u/Sakshee)\ **Post date:** [October 5, 2021, 1:29pm UTC](https://discuss.elastic.co/t/elastic-cloud-and-filebeat-setup/285927/1 "2021-10-05T13:29:27Z")
Hello All, I am trying to send cloud Watch logs from a filebeat server to Elastic Cloud. I am getting following warnings. And not able to see any logs on Elastic cloud Kibana. ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/8/88a1f755dc7fd58b8e8e30b14d9d07a2a8e097cb.png) Logs from Filebeat: ```auto 2021-10-05T12:56:59.465Z WARN [elasticsearch] elasticsearch/client.go:405 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x0, ext:63732644280, loc:(*time.Location)(nil)}, Meta:{"_id":"35615348924624437956339893240882599504693814538127212554","raw_index":"dev_rds-alias"}, Fields:{"agent":{"ephemeral_id":"63312bd5-4ef8-402a-8bd2-d6668f07c6d9","hostname":"ip-10-31-32-28.ec2.internal","id":"a02d2186-81fa-4564-94ca-adcbd3751774","name":"ip-10-31-32-28.ec2.internal","type":"filebeat","version":"7.14.0"},"awscloudwatch":{"ingestion_time":"2020-08-10T08:18:00.000Z","log_group":"/aws/rds/cluster/payway-dev-aurora-db/general","log_stream":"tf-20200810080249874600000003"},"cloud":{"provider":"aws","region":"us-east-1"},"ecs":{"version":"1.10.0"},"event":{"id":"35615348924624437956339893240882599504693814538127212554","ingested":"2021-10-05T12:56:42.165Z"},"host":{"name":"ip-10-31-32-28.ec2.internal"},"input":{"type":"aws-cloudwatch"},"log.file.path":"/aws/rds/cluster/payway-dev-aurora-db/general/tf-20200810080249874600000003","message":"2020-08-10T08:18:00.143325Z 6 Query\tSELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status;","mime_type":"text/plain; charset=utf-8"}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"Field [log.file] must be an object; but it's configured as [flattened] in dynamic template [null]"} 2021-10-05T12:56:59.465Z WARN [elasticsearch] elasticsearch/client.go:405 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x0, ext:63734214870, loc:(*time.Location)(nil)}, Meta:{"_id":"35650374248836791604351777452586324382930128185918488624","raw_index":"dev_rds-alias"}, Fields:{"agent":{"ephemeral_id":"63312bd5-4ef8-402a-8bd2-d6668f07c6d9","hostname":"ip-10-31-32-28.ec2.internal","id":"a02d2186-81fa-4564-94ca-adcbd3751774","name":"ip-10-31-32-28.ec2.internal","type":"filebeat","version":"7.14.0"},"awscloudwatch":{"ingestion_time":"2020-08-28T12:34:39.000Z","log_group":"/aws/rds/cluster/regapi-aurora-sl/general","log_stream":"regapi-aurora-sl"},"cloud":{"provider":"aws","region":"us-east-1"},"ecs":{"version":"1.10.0"},"event":{"id":"35650374248836791604351777452586324382930128185918488624","ingested":"2021-10-05T12:56:42.166Z"},"host":{"name":"ip-10-31-32-28.ec2.internal"},"input":{"type":"aws-cloudwatch"},"log.file.path":"/aws/rds/cluster/regapi-aurora-sl/general/regapi-aurora-sl","message":"\t\t 1 Query\tset local oscar_local_only_replica_host_status=0","mime_type":"text/plain; charset=utf-8"}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"Field [log.file] must be an object; but it's configured as [flattened] in dynamic template [null]"} 2021-10-05T12:56:59.465Z WARN [elasticsearch] elasticsearch/client.go:405 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x0, ext:63740149735, loc:(*time.Location)(nil)}, Meta:{"_id":"35782726160989469051069842639628427225448656220337668152","raw_index":"dev_rds-alias"}, Fields:{"agent":{"ephemeral_id":"63312bd5-4ef8-402a-8bd2-d6668f07c6d9","hostname":"ip-10-31-32-28.ec2.internal","id":"a02d2186-81fa-4564-94ca-adcbd3751774","name":"ip-10-31-32-28.ec2.internal","type":"filebeat","version":"7.14.0"},"awscloudwatch":{"ingestion_time":"2020-11-05T05:08:59.000Z","log_group":"/aws/rds/cluster/regapi-aurora-sl/error","log_stream":"regapi-aurora-sl"},"cloud":{"provider":"aws","region":"us-east-1"},"ecs":{"version":"1.10.0"},"event":{"id":"35782726160989469051069842639628427225448656220337668152","ingested":"2021-10-05T12:56:42.166Z"},"host":{"name":"ip-10-31-32-28.ec2.internal"},"input":{"type":"aws-cloudwatch"},"log.file.path":"/aws/rds/cluster/regapi-aurora-sl/error/regapi-aurora-sl","message":"2020-11-05 05:08:55 12212 [Note] Plugin 'FEDERATED' is disabled.","mime_type":"text/plain; charset=utf-8"}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"Field [log.file] must be an object; but it's configured as [flattened] in dynamic template [null]"} ``` Can anyone help me with this ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 11, 2021, 1:11am UTC](https://discuss.elastic.co/t/elastic-cloud-and-filebeat-setup/285927/2 "2021-10-11T01:11:50Z")
Welcome to our community! 😃 > [@Sakshee](#): > > `"type":"mapper_parsing_exception","reason":"Field [log.file] must be an object; but it's configured as [flattened] in dynamic template [null]"}` That's why. Can you share your Filebeat config? ---
**Author:** ![Sakshee](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@Sakshee](https://discuss.elastic.co/u/Sakshee)\ **Post date:** [October 22, 2021, 1:03pm UTC](https://discuss.elastic.co/t/elastic-cloud-and-filebeat-setup/285927/3 "2021-10-22T13:03:11Z")
Hey @warkolm, i was able to resolve that issue, I am stuck at another now I have created the indexes as "dev-alias-00001" Configured the rollover for index, along with the template and ILM Policy. It is rolling over the index to dev-alias-00002 from dev-alias-00001. But filebeat is still using dev-alias-00001 to send the data How can I make file beat aware about the new index Am I missing something here ? ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/6/a6df39f0983ae6f3c62cc3170891f534dc252fcb.png) It should be using index 0002 for writing the new data Pasting my filebeat .yml ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/7/57604e9591e6d6798f20b5e2d787d2ef6b9a27a6.png) ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [October 22, 2021, 1:46pm UTC](https://discuss.elastic.co/t/elastic-cloud-and-filebeat-setup/285927/4 "2021-10-22T13:46:59Z")
Based on ur config, u have`index: ...` set to the specific index not an alias. For ur ILM policy, what is the ILM Alias? ---
**Author:** ![Sakshee](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@Sakshee](https://discuss.elastic.co/u/Sakshee)\ **Post date:** [October 22, 2021, 2:00pm UTC](https://discuss.elastic.co/t/elastic-cloud-and-filebeat-setup/285927/5 "2021-10-22T14:00:10Z")
@legoguy1000 , sharing my config here, i did not quite catch the **salad name** what do you mean by that ? Filebeat: Config: ```auto output.elasticsearch: username: "aws_logstash" password: " ******" index: "%{[raw_index]}-%{[version]}" hosts: [" *******"] ssl.enabled: true bulk_max_size: 5000 worker: 6 monitoring: enabled: true cloud.id: ' *****' cloud.auth: ' *******' processors: - detect_mime_type: field: message target: mime_type - if: equals: mime_type: "application/json" then: - script: lang: javascript id: no_dot source: > function process(event) { event.Put("message", event.Get("message").split("\".\":{},").join("").split(".").join("_")) } - decode_json_fields: fields: ["message"] process_array: false max_depth: 1 target: "message_json" expand_keys: false setup: template.enabled: false ilm.enabled: false dashboards.enabled: false #setup: # template.enabled: true # template.name: "test-eks" # template.pattern: "eks_dev*" # ilm.enabled: auto # ilm.pattern: "{now/M{yyyy.MM}}-000001" # ilm.policy.name: "Test_ILM_Policy" # dashboards.enabled: false logging: #level: debug to_files: true files: path: /var/log/filebeat name: filebeat keepfiles: 10 permissions: 0644 filebeat.inputs: index: "eks_dev-alias-00000001" log_group_arn: "arn:aws:logs:us-east-1:143306358216:log-group:/aws/eks/EKS_dev/cluster:*" role_arn: "arn:aws:iam::143306358216:role/FilebeatPoc" scan_frequency: 1m start_position: end tags: ["eks"] ##################################### Index template settings { "index": { "lifecycle": { "name": "development_env_policy", "rollover_alias": "dev_eks-alias" }, "mapping": { "total_fields": { "limit": "1000" }, "ignore_malformed": "true" }, "refresh_interval": "5s", "number_of_shards": "3", "number_of_replicas": "1" } } ``` ################### ILM: ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/5/05e89c9e2d03ef1d00a347ac97187da48abdd408.png) ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [October 22, 2021, 2:11pm UTC](https://discuss.elastic.co/t/elastic-cloud-and-filebeat-setup/285927/6 "2021-10-22T14:11:37Z")
Stupid swipe auto correct. I meant the ILM alias name. It looks like ur alias is `dev_eks-alias` so that is what should be set as the `index: dev_eks-alias` since FIlebeat isn't setting up the ILM policy. ---
**Author:** ![Sakshee](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@Sakshee](https://discuss.elastic.co/u/Sakshee)\ **Post date:** [October 22, 2021, 2:16pm UTC](https://discuss.elastic.co/t/elastic-cloud-and-filebeat-setup/285927/7 "2021-10-22T14:16:35Z")
@legoguy1000 , yes i kept it like that initially, when i configured ILM policy using that index it was throwing this error on rollover `illegal_argument_exception: index name [eks_dev-alias] does not match pattern '^.*-\d+$'` It expects a number string at the end of index name so I added "00001" and it worked at first rollover ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [October 22, 2021, 2:51pm UTC](https://discuss.elastic.co/t/elastic-cloud-and-filebeat-setup/285927/8 "2021-10-22T14:51:44Z")
What do u mean when u configured ILM policy using that? you shouldn't have to do any more config of the ILM policy as it already exists. U just need to send to that alias. The index names should be able to be anything, what did your config look like when used `eks_dev-alias` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 19, 2021, 2:52pm UTC](https://discuss.elastic.co/t/elastic-cloud-and-filebeat-setup/285927/9 "2021-11-19T14:52:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.