Filebeat azure module error code 409 BlobAlreadyExists

Hi, I'm getting this error from filebeat:

Apr 05 15:14:43 SOMETHING filebeat[410843]:    / ____/   _____  ____  / /_/ / / /_  __/ /_  _____

Apr 05 15:14:43 SOMETHING filebeat[410843]:   / __/ | | / / _ \/ __ \/ __/ /_/ / / / / __ \/ ___/

Apr 05 15:14:43 SOMETHING filebeat[410843]:  / /___ | |/ /  __/ / / / /_/ __  / /_/ / /_/ (__  )

Apr 05 15:14:43 SOMETHING filebeat[410843]: /_____/ |___/\___/_/ /_/\__/_/ /_/\__,_/_.___/____/

Apr 05 15:14:43 SOMETHING filebeat[410843]: => processing events, ctrl+c to exit

Apr 05 15:14:43 SOMETHING /usr/share/filebeat/bin/filebeat[410843]: 2021/04/05 15:14:43 ==> REQUEST/RESPONSE (Try=1/6.55729ms, OpTime=6.613691ms) -- REQUEST ERROR

                                                                         PUT https://SOMETHING.windows.net/filebeat-activitylogs-insights-operational-logs/0?timeout=61

                                                                         Authorization: REDACTED

                                                                         Content-Length: [80]

                                                                         If-None-Match: [*]

                                                                         User-Agent: [Azure-Storage/0.7 (go1.15.8; linux)]

                                                                         X-Ms-Blob-Cache-Control: []

                                                                         X-Ms-Blob-Content-Disposition: []

                                                                         X-Ms-Blob-Content-Encoding: []

                                                                         X-Ms-Blob-Content-Language: []

                                                                         X-Ms-Blob-Content-Type: []

                                                                         X-Ms-Blob-Type: [BlockBlob]

                                                                         X-Ms-Client-Request-Id: [2892b0df-ad1a-4e82-7db0-8a757a788fc8]

                                                                         X-Ms-Version: [2018-11-09]

                                                                         x-ms-date: [Mon, 05 Apr 2021 15:14:43 GMT]

                                                                         --------------------------------------------------------------------------------

                                                                         RESPONSE Status: 409 The specified blob already exists.

                                                                         Content-Length: [220]

                                                                         Content-Type: [application/xml]

                                                                         Date: [Mon, 05 Apr 2021 15:14:42 GMT]

                                                                         Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]

                                                                         X-Ms-Error-Code: [BlobAlreadyExists]

                                                                         X-Ms-Request-Id: [4199160b-e01e-000c-0b2e-2ad423000000]

                                                                         X-Ms-Version: [2018-11-09]

                                                                         --------------------------------------------------------------------------------

                                                                         ERROR:

                                                                      -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azbl>

                                                                      ===== RESPONSE ERROR (ServiceCode=BlobAlreadyExists) =====

                                                                      Description=The specified blob already exists.

                                                                      RequestId:4199160b-e01e-000c-0b2e-2ad423000000

                                                                      Time:2021-04-05T15:14:43.7015200Z, Details:

                                                                         Code: BlobAlreadyExists

                                                                         PUT https://SOMETHING.windows.net/filebeat-activitylogs-insights-operational-logs/0?timeout=61

                                                                         Authorization: REDACTED

                                                                         Content-Length: [80]

                                                                         If-None-Match: [*]

                                                                         User-Agent: [Azure-Storage/0.7 (go1.15.8; linux)]

                                                                         X-Ms-Blob-Cache-Control: []

                                                                         X-Ms-Blob-Content-Disposition: []

OS: Centos8
Filebeat version: 7.12.0

azure.yml:



- module: azure
  # All logs
  activitylogs:
    enabled: true
    var:
      # eventhub name containing the activity logs, overwrite he default value if the logs are exported in a different eventhub
      eventhub: "SOMETHING"
      # consumer group name that has access to the event hub, we advise creating a dedicated consumer group for the azure module
      consumer_group: "$Default"
      # the connection string required to communicate with Event Hubs, steps to generate one here https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string
      connection_string: "Endpoint=sb://SOMETHING.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=KEY"
      # the name of the storage account the state/offsets will be stored and updated
      storage_account: "ACOUNT"
      # the storage account key, this key will be used to authorize access to data in your storage account
      storage_account_key: "KEY"

filebeat.yml



filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s


output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5547"]

I don't know why I can't edit my original post (I could edit it the first time tough), I'm adding more information.

Apr 05 15:14:43 something filebeat[410843]:    / ____/   _____  ____  / /_/ / / /_  __/ /_  _____

Apr 05 15:14:43 something filebeat[410843]:   / __/ | | / / _ \/ __ \/ __/ /_/ / / / / __ \/ ___/

Apr 05 15:14:43 something filebeat[410843]:  / /___ | |/ /  __/ / / / /_/ __  / /_/ / /_/ (__  )

Apr 05 15:14:43 something filebeat[410843]: /_____/ |___/\___/_/ /_/\__/_/ /_/\__,_/_.___/____/

Apr 05 15:14:43 something filebeat[410843]: => processing events, ctrl+c to exit

Apr 05 15:14:43 something /usr/share/filebeat/bin/filebeat[410843]: 2021/04/05 15:14:43 ==> REQUEST/RESPONSE (Try=1/6.55729ms, OpTime=6.613691ms) -- REQUEST ERROR

                                                                         PUT https://something.blob.core.windows.net/filebeat-activitylogs-insights-operational-logs/0?timeout=61

                                                                         Authorization: REDACTED
journalctl -f -l -u filebeat

-- Logs begin at Thu 2021-03-11 14:08:09 UTC. --

Apr 07 12:50:17 something filebeat[446353]:     ______                 __  __  __      __

Apr 07 12:50:17 something filebeat[446353]:    / ____/   _____  ____  / /_/ / / /_  __/ /_  _____

Apr 07 12:50:17 something filebeat[446353]:   / __/ | | / / _ \/ __ \/ __/ /_/ / / / / __ \/ ___/

Apr 07 12:50:17 something filebeat[446353]:  / /___ | |/ /  __/ / / / /_/ __  / /_/ / /_/ (__  )

Apr 07 12:50:17 something filebeat[446353]: /_____/ |___/\___/_/ /_/\__/_/ /_/\__,_/_.___/____/

Apr 07 12:50:17 masecopselk filebeat[446353]: => processing events, ctrl+c to exit

Apr 07 12:50:19 something filebeat[446353]: 2021-04-07T12:50:19.213Z        INFO        [publisher_pipeline_output]        pipeline/output.go:143        Connecting to backoff(async(tcp://localhost:5547))

Apr 07 12:50:19 something filebeat[446353]: 2021-04-07T12:50:19.213Z        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer

Apr 07 12:50:19 something filebeat[446353]: 2021-04-07T12:50:19.213Z        INFO        [publisher]        pipeline/retry.go:223          done

Apr 07 12:50:19 something filebeat[446353]: 2021-04-07T12:50:19.214Z        INFO        [publisher_pipeline_output]        pipeline/output.go:151        Connection to backoff(async(tcp://localhost:5547)) established

Apr 07 12:50:47 something filebeat[446353]: 2021-04-07T12:50:47.861Z        INFO        [monitoring]        log/log.go:144        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"memory":{"id":"filebeat.service","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":38805504}}}},"cpu":{"system":{"ticks":80,"time":{"ms":82}},"total":{"ticks":340,"time":{"ms":351},"value":0},"user":{"ticks":260,"time":{"ms":269}}},"handles":{"limit":{"hard":262144,"soft":1024},"open":21},"info":{"ephemeral_id":"eb437817-03bb-4a4e-aedb-ab69e513c3fc","uptime":{"ms":30054}},"memstats":{"gc_next":22715792,"memory_alloc":18722008,"memory_sys":76628992,"memory_total":77063184,"rss":85762048},"runtime":{"goroutines":73}},"filebeat":{"events":{"added":24,"done":24},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":1,"scans":1},"output":{"events":{"acked":24,"active":0,"batches":1,"total":24},"read":{"bytes":6},"type":"logstash","write":{"bytes":10178}},"pipeline":{"clients":1,"events":{"active":0,"published":24,"retry":24,"total":24},"queue":{"acked":24}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":8},"load":{"1":0.09,"15":0.03,"5":0.05,"norm":{"1":0.0113,"15":0.0038,"5":0.0063}}}}}}

I read this:

It seems that the blob exists but it's blocked and it's not possible to edit/delete it while filebeat is running, so we stopped the filbeat service and we deleted this blob, we restarted filebeat and it worked, but when we stop and start the service again we get the same error again.