Filebeat throwing exception with increased value of max_bytes

Elastic Stack Beats
1

I have a problem where I need to read a big xml file (say about 50 MB) through Filebeat and then send this whole content as a single event to logstash which will then use some filters such as xml , mutate etc and finally dump the json in elasticsearch. To do this I am using multiline with filebeat with increased values for multiline.max_lines and also since the file is bigger than 10 MB, i m using about 100 MB for max_bytes. However when i run filebeat i get the following exceptions :

       runtime: VirtualAlloc of 84803584 bytes failed with errno=487  
      fatal error: runtime: cannot map pages in arena address space  

      runtime stack:  
      runtime.throw(0x97485e, 0x30)  
              /usr/local/go/src/runtime/panic.go:566 +0x7f  
      runtime.sysMap(0x48d90000, 0x50e0000, 0x26fd01, 0xc617d8)  
              /usr/local/go/src/runtime/mem_windows.go:116 +0xf9  
      runtime.( mheap).sysAlloc(0xc50d40, 0x50e0000, 0x1)  
              /usr/local/go/src/runtime/malloc.go:407 +0x123  
      runtime.( mheap).grow(0xc50d40, 0x2870, 0x0)  
              /usr/local/go/src/runtime/mheap.go:726 +0x52  
      runtime.( mheap).allocSpanLocked(0xc50d40, 0x286d, 0x0)  
              /usr/local/go/src/runtime/mheap.go:630 +0x4d5  
      runtime.( mheap).alloc_m(0xc50d40, 0x286d, 0x0, 0x1, 0xc546000)  
              /usr/local/go/src/runtime/mheap.go:515 +0x116  
      runtime.( mheap).alloc.func1()  
              /usr/local/go/src/runtime/mheap.go:579 +0x3d  
      runtime.systemstack(0x26ff10)  
              /usr/local/go/src/runtime/asm_386.s:339 +0x88  
      runtime.( mheap).alloc(0xc50d40, 0x286d, 0x0, 0x3a90001, 0x3a9dac0)  
              /usr/local/go/src/runtime/mheap.go:580 +0x5a  
      runtime.largeAlloc(0x50da000, 0x451000, 0x44cd0000)  
              /usr/local/go/src/runtime/malloc.go:774 +0x94  
      runtime.mallocgc.func1()  
              /usr/local/go/src/runtime/malloc.go:669 +0x31  
      runtime.systemstack(0x1d4b4a00)  
              /usr/local/go/src/runtime/asm_386.s:323 +0x5e  
      runtime.mstart()  
              /usr/local/go/src/runtime/proc.go:1079

below is the filebeat.yml

filebeat.yml

filebeat.prospectors:


- input_type: log
  paths:
    - D:\sdn_advanced\sdn_advanced.xml

  close_eof: true

  multiline.pattern: '^<?xml'

  multiline.negate: true

  multiline.match: after
  multiline.max_lines: 1000000
  
  #something around 100 MB
  max_bytes: 104857600  


#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]


logging:
  to_files: true
  files:
    path: D:/Softwares/Filebeat/logs
  level: debug
  selectors: ["*"]
2

That looks like you are getting out of memory. How much free memory do you have on the system and what is the total size of the files you are ingesting? Filebeat is going to potentially buffer thousands of events in memory for batching, so for extremely large events you probably need to drastically reduce the size of all queues.

3

Yes this looks like an OOM issue. However I think there is sufficient memory available on the system . below is the output of systeminfo command .

Total Physical Memory: 8,098 MB
Available Physical Memory: 1,995 MB
Virtual Memory: Max Size: 16,195 MB
Virtual Memory: Available: 8,074 MB

Also as per filebeat.yml shared above, I am having only a single file and I want this big xml into a single event. As per your comment, @tudor which configuration needs to be changed for reducing queue sizes

4

If you expect a single event, then it's maybe not the queue sizes. Can you run filebeat with -d "*" and post the log here? I'd be curious to see if it gets past multiline or not.

5

@tudor attaching the log containing the error. This error occurs when the max_bytes configured is around 100 MB . However it runs fine when the value for max_bytes is default (10 MB).

runtime: VirtualAlloc of 165609472 bytes failed with errno=487
fatal error: runtime: cannot map pages in arena address space

runtime stack:
runtime.throw(0x97485e, 0x30)
/usr/local/go/src/runtime/panic.go:566 +0x7f
runtime.sysMap(0x50ee0000, 0x9df0000, 0xcfd01, 0xc617d8)
/usr/local/go/src/runtime/mem_windows.go:116 +0xf9
runtime.(*mheap).sysAlloc(0xc50d40, 0x9df0000, 0x1)
/usr/local/go/src/runtime/malloc.go:407 +0x123
runtime.(*mheap).grow(0xc50d40, 0x4ef8, 0x0)
/usr/local/go/src/runtime/mheap.go:726 +0x52
runtime.(*mheap).allocSpanLocked(0xc50d40, 0x4ef7, 0x40e054)
/usr/local/go/src/runtime/mheap.go:630 +0x4d5
runtime.(*mheap).alloc_m(0xc50d40, 0x4ef7, 0x0, 0x1, 0x18158000)
/usr/local/go/src/runtime/mheap.go:515 +0x116
runtime.(*mheap).alloc.func1()
/usr/local/go/src/runtime/mheap.go:579 +0x3d
runtime.systemstack(0xcfe8c)
/usr/local/go/src/runtime/asm_386.s:339 +0x88
runtime.(*mheap).alloc(0xc50d40, 0x4ef7, 0x0, 0x33bf0001, 0x33bf9680)
/usr/local/go/src/runtime/mheap.go:580 +0x5a
runtime.largeAlloc(0x9dee000, 0x451000, 0x4907a000)
/usr/local/go/src/runtime/malloc.go:774 +0x94
runtime.mallocgc.func1()
/usr/local/go/src/runtime/malloc.go:669 +0x31
runtime.systemstack(0xc4bd40)
/usr/local/go/src/runtime/asm_386.s:323 +0x5e
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1079

goroutine 42 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_386.s:277 fp=0x227cf5e0 sp=0x227cf5dc
runtime.mallocgc(0x9dee000, 0x0, 0x6512000, 0x4907a000)
/usr/local/go/src/runtime/malloc.go:670 +0x7b2 fp=0x227cf648 sp=0x227cf5e0
runtime.growslice(0x8ca960, 0x4907a000, 0x1f96000, 0x1f96000, 0x1f96001, 0x0, 0x0, 0x0)
/usr/local/go/src/runtime/slice.go:126 +0x183 fp=0x227cf688 sp=0x227cf648
os.(*File).writeConsole(0x12e520d0, 0x2c19c0a9, 0xe08783, 0xe09f57, 0x2d9e82c, 0x0, 0x0)
/usr/local/go/src/os/file_windows.go:361 +0x5a8 fp=0x227cfaf8 sp=0x227cf688
os.(*File).write(0x12e520d0, 0x2a206000, 0x2d9e82c, 0x2da0000, 0x0, 0x0, 0x0)
/usr/local/go/src/os/file_windows.go:398 +0xc0 fp=0x227cfb30 sp=0x227cfaf8
os.(*File).Write(0x12e520d0, 0x2a206000, 0x2d9e82c, 0x2da0000, 0x2d9e82b, 0x0, 0x0)
/usr/local/go/src/os/file.go:142 +0x6f fp=0x227cfb64 sp=0x227cfb30
log.(*Logger).Output(0x12e37b60, 0x4, 0x3ae00000, 0x2d9e801, 0x0, 0x0)
/usr/local/go/src/log/log.go:166 +0x2c5 fp=0x227cfbdc sp=0x227cfb64
github.com/elastic/beats/libbeat/logp.send(0x4, 0x7, 0x955f41, 0x5, 0x95bf0d, 0xb, 0x227cfd40, 0x1, 0x1)
/go/src/github.com/elastic/beats/libbeat/logp/log.go:85 +0x172 fp=0x227cfca8 sp=0x227cfbdc
github.com/elastic/beats/libbeat/logp.debugMessage(0x3, 0x958e76, 0x7, 0x95bf0d, 0xb, 0x227cfd40, 0x1, 0x1)
/go/src/github.com/elastic/beats/libbeat/logp/log.go:76 +0xd5 fp=0x227cfce8 sp=0x227cfca8
github.com/elastic/beats/libbeat/logp.Debug(0x958e76, 0x7, 0x95bf0d, 0xb, 0x227cfd40, 0x1, 0x1)
/go/src/github.com/elastic/beats/libbeat/logp/log.go:95 +0x59 fp=0x227cfd0c sp=0x227cfce8
github.com/elastic/beats/libbeat/publisher.(*client).filterEvent(0x12eebd60, 0x12fc60e0, 0x0)
/go/src/github.com/elastic/beats/libbeat/publisher/client.go:214 +0x282 fp=0x227cfd68 sp=0x227cfd0c
github.com/elastic/beats/libbeat/publisher.(*client).PublishEvents(0x12eebd60, 0x1ce377d0, 0x1, 0x3, 0x1ce37810, 0x3, 0
3, 0x1)
/go/src/github.com/elastic/beats/libbeat/publisher/client.go:143 +0x1ef fp=0x227cfeac sp=0x227cfd68
github.com/elastic/beats/filebeat/publisher.(*syncLogPublisher).Publish(0x12ee7260, 0x0, 0x0)
/go/src/github.com/elastic/beats/filebeat/publisher/sync.go:62 +0x23b fp=0x227cff98 sp=0x227cfeac
github.com/elastic/beats/filebeat/publisher.(*syncLogPublisher).Start.func1(0x12ee7260)
/go/src/github.com/elastic/beats/filebeat/publisher/sync.go:45 +0xc8 fp=0x227cffc8 sp=0x227cff98
runtime.goexit()
/usr/local/go/src/runtime/asm_386.s:1612 +0x1 fp=0x227cffcc sp=0x227cffc8
created by github.com/elastic/beats/filebeat/publisher.(*syncLogPublisher).Start
/go/src/github.com/elastic/beats/filebeat/publisher/sync.go:50 +0x9e

goroutine 1 [chan receive]:
github.com/elastic/beats/filebeat/beater.(*Filebeat).Run(0x12ee8fa0, 0x12ed4180, 0x0, 0x0)
/go/src/github.com/elastic/beats/filebeat/beater/filebeat.go:200 +0xe46
github.com/elastic/beats/libbeat/beat.(*Beat).launch(0x12ed4180, 0x9b59b8, 0x0, 0x0)
/go/src/github.com/elastic/beats/libbeat/beat/beat.go:225 +0x900
github.com/elastic/beats/libbeat/beat.Run(0x959ac5, 0x8, 0x0, 0x0, 0x9b59b8, 0x0, 0x0)
/go/src/github.com/elastic/beats/libbeat/beat/beat.go:141 +0x57
main.main()
/go/src/github.com/elastic/beats/filebeat/main.go:22 +0x40

goroutine 35 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:116 +0x11e
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x1a
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x36

goroutine 19 [select]:
github.com/elastic/beats/vendor/golang.org/x/sys/windows/svc/debug.Run.func1(0x12ee2240, 0x12ee21c0, 0x12ee2200)
/go/src/github.com/elastic/beats/vendor/golang.org/x/sys/windows/svc/debug/service.go:32 +0x14c
created by github.com/elastic/beats/vendor/golang.org/x/sys/windows/svc/debug.Run
/go/src/github.com/elastic/beats/vendor/golang.org/x/sys/windows/svc/debug/service.go:38 +0xd2

goroutine 14 [chan receive]:
github.com/elastic/beats/libbeat/logp.logMetrics(0x12ed4200)
/go/src/github.com/elastic/beats/libbeat/logp/metrics.go:28 +0x2b4
created by github.com/elastic/beats/libbeat/logp.Init
/go/src/github.com/elastic/beats/libbeat/logp/logp.go:161 +0x381

goroutine 37 [select]:
github.com/elastic/beats/libbeat/publisher.(*messageWorker).run(0x12ee6960)
/go/src/github.com/elastic/beats/libbeat/publisher/worker.go:63 +0x1c3
created by github.com/elastic/beats/libbeat/publisher.(*messageWorker).init
/go/src/github.com/elastic/beats/libbeat/publisher/worker.go:57 +0xff

goroutine 38 [select]:
github.com/elastic/beats/libbeat/publisher.(*bulkWorker).run(0x12eef640)
/go/src/github.com/elastic/beats/libbeat/publisher/bulk.go:54 +0x284
created by github.com/elastic/beats/libbeat/publisher.newBulkWorker
/go/src/github.com/elastic/beats/libbeat/publisher/bulk.go:42 +0x1da

goroutine 39 [chan receive]:
github.com/elastic/beats/libbeat/service.HandleSignals.func1(0x12eefac0, 0x12ee8f94, 0x12ee8fb0)
/go/src/github.com/elastic/beats/libbeat/service/service.go:32 +0x31
created by github.com/elastic/beats/libbeat/service.HandleSignals
/go/src/github.com/elastic/beats/libbeat/service/service.go:35 +0x158

6

Interesting, now it seems to run out of memory when logging the event (for debugging). I tried reproducing, but on macOS things seem to be working fine. Perhaps the OS has troubles finding a continuous memory buffer that large.

Going back to the big picture, perhaps filebeat is not the right tool for this case. If it's a single file and event, maybe you want to just index it with a simple curl command instead?

7

Thanks @tudor. Feels same that Filebeat is not the right tool for this. I have tried similar thing using logstash using multiline codec with input plugin using combined configuration of max_lines and max_bytes. This seems to be working fine till the control goes to logstash output plugin where it is failing with the below response

 {:code=>400, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to derive x
content\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to derive xcontent\"},\"status\":400}", :request_body=>""}

I will raise a separate question for that.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.