Elastic Cloud and Filebeat setup

Sakshee · October 5, 2021, 1:29pm

Hello All,

I am trying to send cloud Watch logs from a filebeat server to Elastic Cloud. I am getting following warnings. And not able to see any logs on Elastic cloud Kibana.

Logs from Filebeat:


2021-10-05T12:56:59.465Z        WARN    [elasticsearch] elasticsearch/client.go:405     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x0, ext:63732644280, loc:(*time.Location)(nil)}, Meta:{"_id":"35615348924624437956339893240882599504693814538127212554","raw_index":"dev_rds-alias"}, Fields:{"agent":{"ephemeral_id":"63312bd5-4ef8-402a-8bd2-d6668f07c6d9","hostname":"ip-10-31-32-28.ec2.internal","id":"a02d2186-81fa-4564-94ca-adcbd3751774","name":"ip-10-31-32-28.ec2.internal","type":"filebeat","version":"7.14.0"},"awscloudwatch":{"ingestion_time":"2020-08-10T08:18:00.000Z","log_group":"/aws/rds/cluster/payway-dev-aurora-db/general","log_stream":"tf-20200810080249874600000003"},"cloud":{"provider":"aws","region":"us-east-1"},"ecs":{"version":"1.10.0"},"event":{"id":"35615348924624437956339893240882599504693814538127212554","ingested":"2021-10-05T12:56:42.165Z"},"host":{"name":"ip-10-31-32-28.ec2.internal"},"input":{"type":"aws-cloudwatch"},"log.file.path":"/aws/rds/cluster/payway-dev-aurora-db/general/tf-20200810080249874600000003","message":"2020-08-10T08:18:00.143325Z    6 Query\tSELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status;","mime_type":"text/plain; charset=utf-8"}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"Field [log.file] must be an object; but it's configured as [flattened] in dynamic template [null]"}
2021-10-05T12:56:59.465Z        WARN    [elasticsearch] elasticsearch/client.go:405     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x0, ext:63734214870, loc:(*time.Location)(nil)}, Meta:{"_id":"35650374248836791604351777452586324382930128185918488624","raw_index":"dev_rds-alias"}, Fields:{"agent":{"ephemeral_id":"63312bd5-4ef8-402a-8bd2-d6668f07c6d9","hostname":"ip-10-31-32-28.ec2.internal","id":"a02d2186-81fa-4564-94ca-adcbd3751774","name":"ip-10-31-32-28.ec2.internal","type":"filebeat","version":"7.14.0"},"awscloudwatch":{"ingestion_time":"2020-08-28T12:34:39.000Z","log_group":"/aws/rds/cluster/regapi-aurora-sl/general","log_stream":"regapi-aurora-sl"},"cloud":{"provider":"aws","region":"us-east-1"},"ecs":{"version":"1.10.0"},"event":{"id":"35650374248836791604351777452586324382930128185918488624","ingested":"2021-10-05T12:56:42.166Z"},"host":{"name":"ip-10-31-32-28.ec2.internal"},"input":{"type":"aws-cloudwatch"},"log.file.path":"/aws/rds/cluster/regapi-aurora-sl/general/regapi-aurora-sl","message":"\t\t    1 Query\tset local oscar_local_only_replica_host_status=0","mime_type":"text/plain; charset=utf-8"}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"Field [log.file] must be an object; but it's configured as [flattened] in dynamic template [null]"}
2021-10-05T12:56:59.465Z        WARN    [elasticsearch] elasticsearch/client.go:405     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x0, ext:63740149735, loc:(*time.Location)(nil)}, Meta:{"_id":"35782726160989469051069842639628427225448656220337668152","raw_index":"dev_rds-alias"}, Fields:{"agent":{"ephemeral_id":"63312bd5-4ef8-402a-8bd2-d6668f07c6d9","hostname":"ip-10-31-32-28.ec2.internal","id":"a02d2186-81fa-4564-94ca-adcbd3751774","name":"ip-10-31-32-28.ec2.internal","type":"filebeat","version":"7.14.0"},"awscloudwatch":{"ingestion_time":"2020-11-05T05:08:59.000Z","log_group":"/aws/rds/cluster/regapi-aurora-sl/error","log_stream":"regapi-aurora-sl"},"cloud":{"provider":"aws","region":"us-east-1"},"ecs":{"version":"1.10.0"},"event":{"id":"35782726160989469051069842639628427225448656220337668152","ingested":"2021-10-05T12:56:42.166Z"},"host":{"name":"ip-10-31-32-28.ec2.internal"},"input":{"type":"aws-cloudwatch"},"log.file.path":"/aws/rds/cluster/regapi-aurora-sl/error/regapi-aurora-sl","message":"2020-11-05 05:08:55 12212 [Note] Plugin 'FEDERATED' is disabled.","mime_type":"text/plain; charset=utf-8"}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"Field [log.file] must be an object; but it's configured as [flattened] in dynamic template [null]"}

Can anyone help me with this ?

warkolm · October 11, 2021, 1:11am

Welcome to our community!

That's why.
Can you share your Filebeat config?

Sakshee · October 22, 2021, 1:03pm

Hey @warkolm, i was able to resolve that issue, I am stuck at another now

I have created the indexes as "dev-alias-00001"
Configured the rollover for index, along with the template and ILM Policy.
It is rolling over the index to dev-alias-00002 from dev-alias-00001.
But filebeat is still using dev-alias-00001 to send the data

How can I make file beat aware about the new index
Am I missing something here ?

It should be using index 0002 for writing the new data

Pasting my filebeat .yml

legoguy1000 · October 22, 2021, 1:46pm

Based on ur config, u haveindex: ... set to the specific index not an alias. For ur ILM policy, what is the ILM Alias?

Sakshee · October 22, 2021, 2:00pm

@legoguy1000 , sharing my config here, i did not quite catch the salad name what do you mean by that ?
Filebeat: Config:

output.elasticsearch:
  username: "aws_logstash"
  password: "******"
  index: "%{[raw_index]}-%{[version]}"
  hosts: ["*******"]
  ssl.enabled: true
  bulk_max_size: 5000  
  worker: 6

monitoring:
  enabled: true
  cloud.id: '*****'
  cloud.auth: '*******'

processors:
  - detect_mime_type:
      field: message
      target: mime_type
  - if:
      equals:
        mime_type: "application/json"
    then:
      - script:
          lang: javascript
          id: no_dot
          source: >
            function process(event) {
              event.Put("message", event.Get("message").split("\".\":{},").join("").split(".").join("_"))
            }
      - decode_json_fields:
          fields: ["message"]
          process_array: false
          max_depth: 1
          target: "message_json"
          expand_keys: false


setup:
  template.enabled: false
  ilm.enabled: false
  dashboards.enabled: false

#setup:
#  template.enabled: true
#  template.name: "test-eks"
#  template.pattern: "eks_dev*"
#  ilm.enabled: auto
#  ilm.pattern: "{now/M{yyyy.MM}}-000001"
#  ilm.policy.name: "Test_ILM_Policy"
#  dashboards.enabled: false

logging:
  #level: debug
  to_files: true
  files:
    path: /var/log/filebeat
    name: filebeat
    keepfiles: 10
    permissions: 0644


filebeat.inputs:

    index: "eks_dev-alias-00000001"
    log_group_arn: "arn:aws:logs:us-east-1:143306358216:log-group:/aws/eks/EKS_dev/cluster:*"
    role_arn: "arn:aws:iam::143306358216:role/FilebeatPoc"
    scan_frequency: 1m
    start_position: end
    tags: ["eks"]
#####################################

Index template settings
{
  "index": {
    "lifecycle": {
      "name": "development_env_policy",
      "rollover_alias": "dev_eks-alias"
    },
    "mapping": {
      "total_fields": {
        "limit": "1000"
      },
      "ignore_malformed": "true"
    },
    "refresh_interval": "5s",
    "number_of_shards": "3",
    "number_of_replicas": "1"
  }
}

###################
ILM:

legoguy1000 · October 22, 2021, 2:11pm

Stupid swipe auto correct. I meant the ILM alias name. It looks like ur alias is dev_eks-alias so that is what should be set as the index: dev_eks-alias since FIlebeat isn't setting up the ILM policy.

Sakshee · October 22, 2021, 2:16pm

@legoguy1000 , yes i kept it like that initially, when i configured ILM policy using that index it was throwing this error on rollover
illegal_argument_exception: index name [eks_dev-alias] does not match pattern '^.*-\d+$'

It expects a number string at the end of index name so I added "00001"
and it worked at first rollover

legoguy1000 · October 22, 2021, 2:51pm

What do u mean when u configured ILM policy using that? you shouldn't have to do any more config of the ILM policy as it already exists. U just need to send to that alias. The index names should be able to be anything, what did your config look like when used eks_dev-alias

system · November 19, 2021, 2:52pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.