Azure-blob-storage input: fatal error: concurrent map iteration and map write

Elastic Stack Beats
1

I am getting an error using filebeat with azure-blob-storage input plugin. I am using filebeat version 8.8.1 and the following configuration:

- type: azure-blob-storage
  id: <id>
  enabled: true
  account_name: <account_name>
  max_workers: 1
  poll: true
  poll_interval: 15s
  auth.shared_credentials.account_key: <account_key>
  containers:
  - name: <container_name_1>
  - name: <container_name_2>
  - name: <container_name_3>

Filebeats panic after a few minutes with the following log:

log 
fatal error: concurrent map iteration and map write

goroutine 75 [running]:
github.com/elastic/go-structform.extObjVisitor.OnInt64Object({{0x558f5d84a960?, 0xc0073fac00?}}, 0xc0005bddd0)
        github.com/elastic/go-structform@v0.0.10/map.go:114 +0xac
github.com/elastic/go-structform/gotype.foldMapInt64(0x558f5d188840?, {0x558f5d188840?, 0xc0005bddd0?})
        github.com/elastic/go-structform@v0.0.10/gotype/fold_map.go:83 +0x3d
github.com/elastic/go-structform/gotype.liftFold.func1(0x558f5d43d800?, {0x558f5d188840?, 0xc0005bde28?, 0x558f5f8121a0?})
        github.com/elastic/go-structform@v0.0.10/gotype/fold_reflect.go:605 +0xda
github.com/elastic/go-structform/gotype.makeFieldFold.func1(0xc0040a3e60, {0x558f5d43d800?, 0xc0005bde00?, 0x558f59c892e4?})
        github.com/elastic/go-structform@v0.0.10/gotype/fold_reflect.go:311 +0xa9
github.com/elastic/go-structform/gotype.makeFieldsFold.func1(0xc00461d878?, {0x558f5d43d800?, 0xc0005bde00?, 0x16?})
        github.com/elastic/go-structform@v0.0.10/gotype/fold_reflect.go:198 +0x89
github.com/elastic/go-structform/gotype.makeStructFold.func1(0xc0040a3e60, {0x558f5d43d800?, 0xc0005bde00?, 0xc00461d8a0?})
        github.com/elastic/go-structform@v0.0.10/gotype/fold_reflect.go:188 +0x83
github.com/elastic/go-structform/gotype.makePointerFold.func1(0xc0040a3e60, {0x558f5ceddb80?, 0xc0005bde00?, 0x558f59366234?})
        github.com/elastic/go-structform@v0.0.10/gotype/fold_reflect.go:124 +0x115
github.com/elastic/go-structform/gotype.foldAnyReflect(0x558f5ceddb80?, {0x558f5ceddb80?, 0xc0005bde00?, 0xc00461d970?})
        github.com/elastic/go-structform@v0.0.10/gotype/fold_reflect.go:568 +0x9d
github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc0040a3e60, {0x558f5ceddb80?, 0xc0005bde00})
        github.com/elastic/go-structform@v0.0.10/gotype/fold.go:116 +0x27f
github.com/elastic/go-structform/gotype.(*Iterator).Fold(...)
        github.com/elastic/go-structform@v0.0.10/gotype/fold.go:93
github.com/elastic/beats/v7/libbeat/common/transform/typeconv.(*Converter).Convert(0xc004be38e0, {0x558f5cefc140, 0xc0000ef558}, {0x558f5ceddb80, 0xc0005bde00})
        github.com/elastic/beats/v7/libbeat/common/transform/typeconv/typeconv.go:108 +0x13a
github.com/elastic/beats/v7/libbeat/common/transform/typeconv.Convert({0x558f5cefc140, 0xc0000ef558}, {0x558f5ceddb80, 0xc0005bde00})
        github.com/elastic/beats/v7/libbeat/common/transform/typeconv/typeconv.go:130 +0xcb
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*updateOp).Execute(0xc007248840, 0x0?)
        github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/publish.go:136 +0x199
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.newInputACKHandler.func1(0x1fffffffffffff?, {0xc00021e0a0?, 0x1c?, 0x0?})
        github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/input.go:199 +0x9f
github.com/elastic/beats/v7/libbeat/common/acker.(*eventDataACKer).onACK(0xc00059e1c0, 0x558f593974a0?, 0x1c)
        github.com/elastic/beats/v7/libbeat/common/acker/acker.go:257 +0x134
github.com/elastic/beats/v7/libbeat/common/acker.(*trackingACKer).ACKEvents(0xc0006541e0, 0x1c)
        github.com/elastic/beats/v7/libbeat/common/acker/acker.go:206 +0x203
github.com/elastic/beats/v7/libbeat/common/acker.ackerList.ACKEvents(...)
        github.com/elastic/beats/v7/libbeat/common/acker/acker.go:294
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*bufferingEventLoop).processACK.func1()
        github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/eventloop.go:515 +0x24
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*bufferingEventLoop).processACK(0xc000e9a5a0, {0x0, 0x0}, 0x800)
        github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/eventloop.go:525 +0x1c5
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).handleBatchSig(0xc000e8aa60)
        github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/ackloop.go:73 +0xa4
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).run(0xc000e8aa60)
        github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/ackloop.go:52 +0x192
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func2()
        github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/broker.go:201 +0x65
created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue
        github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue/broker.go:199 +0x52d

...

I have tried to apply the same changes addressing the concurrency issue to the azure-blob-storage input plugin which were applied to the gcs plugin:

[filebeat][gcs] - Added missing locks for safe concurrency by ShourieG · Pull Request #34914 · elastic/beats (github.com)

[8.8](backport #35605) [filebeat][gcs] Fix for concurrency issues and context timeouts in the GCS input by mergify[bot] · Pull Request #35666 · elastic/beats (github.com)

But the issue still persists.

1 Like
2

Hi @kdobmayer,
Thanks for bringing this up.
There are some concurrency issues present in the azure blob storage input atm. There are fixes planned for this here Improve Azure blob storage input · Issue #35126 · elastic/beats · GitHub . However this is due for 8.11.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.