OutOfMemoryError after PDTrueTypeFont Using fallback font warnings

Cordobel · November 26, 2021, 8:54pm

I noticed twice that Elasticsearch crashed after some warnings about PDTrueTypeFont Using fallback font. After these warnings, the garbage collector starts to alert about the overload. Then a fatal error - lambda $MaybeDieOnAnotherThread $2 - and finally a Java heap space OutOfMemoryError that interrupts Elasticsearch.
I'm not really sure if the warnings about PDTrueTypeFont Using fallback font are related to OutOfMemoryError. That's what I plan to get help to find out.
Forgive me if I wrote it in any strange way or if I seem rude or ignorant. English is not my native language.
Log file below.

[2021-11-08T15:19:10,970][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_dsapp006mv] [fbradesco_41_poc/TXlfrdo2QLirjKexGB9VTw] deleting index
[2021-11-08T15:19:11,732][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_dsapp006mv] [fbradesco_41_poc] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[2021-11-08T15:19:11,842][INFO ][o.e.c.s.IndexScopedSettings] [node_dsapp006mv] updating [index.highlight.max_analyzed_offset] from [1000000] to [60000000]
[2021-11-08T15:19:11,852][INFO ][o.e.c.s.IndexScopedSettings] [node_dsapp006mv] updating [index.highlight.max_analyzed_offset] from [1000000] to [60000000]
[2021-11-08T15:19:12,825][WARN ][o.a.p.p.f.PDTrueTypeFont ] [node_dsapp006mv] Using fallback font 'LiberationSans' for 'ArialMT'
[2021-11-08T15:19:12,826][WARN ][o.a.p.p.f.PDTrueTypeFont ] [node_dsapp006mv] Using fallback font 'LiberationSans' for 'TimesNewRomanPSMT'
[2021-11-08T15:19:12,831][WARN ][o.a.p.p.f.PDTrueTypeFont ] [node_dsapp006mv] Using fallback font 'LiberationSans' for 'Arial-BoldMT'
[2021-11-08T15:19:12,987][INFO ][o.e.c.m.MetaDataMappingService] [node_dsapp006mv] [fbradesco_41_poc/Ad2DU6XORry4Tflx4qY9pQ] update_mapping [_doc]
[2021-11-08T15:19:13,191][WARN ][o.a.p.p.f.PDTrueTypeFont ] [node_dsapp006mv] Using fallback font 'LiberationSans' for 'ArialMT'
[2021-11-08T15:19:13,192][WARN ][o.a.p.p.f.PDTrueTypeFont ] [node_dsapp006mv] Using fallback font 'LiberationSans' for 'TimesNewRomanPSMT'
[2021-11-08T15:19:13,195][WARN ][o.a.p.p.f.PDTrueTypeFont ] [node_dsapp006mv] Using fallback font 'LiberationSans' for 'Arial-BoldMT'
[2021-11-08T16:23:28,087][INFO ][o.a.p.p.f.PDCIDFontType2 ] [node_dsapp006mv] OpenType Layout tables used in font CIDFont+F1 are not implemented in PDFBox and will be ignored
[2021-11-08T16:23:28,119][INFO ][o.a.p.p.f.PDCIDFontType2 ] [node_dsapp006mv] OpenType Layout tables used in font CIDFont+F2 are not implemented in PDFBox and will be ignored
[2021-11-08T16:23:28,125][INFO ][o.a.p.p.f.PDCIDFontType2 ] [node_dsapp006mv] OpenType Layout tables used in font CIDFont+F3 are not implemented in PDFBox and will be ignored
[2021-11-09T10:07:27,030][WARN ][o.e.m.j.JvmGcMonitorService] [node_dsapp006mv] [gc][12415235] overhead, spent [552ms] collecting in the last [1s]
[2021-11-09T10:07:28,048][INFO ][o.e.m.j.JvmGcMonitorService] [node_dsapp006mv] [gc][12415236] overhead, spent [391ms] collecting in the last [1s]
[2021-11-09T10:07:30,048][INFO ][o.e.m.j.JvmGcMonitorService] [node_dsapp006mv] [gc][12415238] overhead, spent [365ms] collecting in the last [1s]
[2021-11-09T10:07:31,475][INFO ][o.e.m.j.JvmGcMonitorService] [node_dsapp006mv] [gc][12415239] overhead, spent [693ms] collecting in the last [1.4s]
[2021-11-09T10:07:33,476][INFO ][o.e.m.j.JvmGcMonitorService] [node_dsapp006mv] [gc][12415241] overhead, spent [357ms] collecting in the last [1s]
[2021-11-09T10:07:34,477][INFO ][o.e.m.j.JvmGcMonitorService] [node_dsapp006mv] [gc][12415242] overhead, spent [332ms] collecting in the last [1s]
[2021-11-09T10:07:35,706][INFO ][o.e.m.j.JvmGcMonitorService] [node_dsapp006mv] [gc][12415243] overhead, spent [390ms] collecting in the last [1.2s]
[2021-11-09T10:07:39,123][WARN ][o.e.m.j.JvmGcMonitorService] [node_dsapp006mv] [gc][12415245] overhead, spent [2.3s] collecting in the last [2.4s]
[2021-11-09T10:07:44,165][WARN ][o.e.m.j.JvmGcMonitorService] [node_dsapp006mv] [gc][12415249] overhead, spent [1.5s] collecting in the last [2s]
[2021-11-09T10:07:47,694][WARN ][o.e.m.j.JvmGcMonitorService] [node_dsapp006mv] [gc][12415250] overhead, spent [3.4s] collecting in the last [3.5s]
[2021-11-09T10:08:00,927][ERROR][o.e.ExceptionsHelper     ] [node_dsapp006mv] fatal error
	at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$2(ExceptionsHelper.java:310)
	at java.base/java.util.Optional.ifPresent(Optional.java:183)
	at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:300)
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:76)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:276)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:268)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1375)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:276)
	at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:918)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:174)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:835)
[2021-11-09T10:08:00,956][WARN ][o.e.h.AbstractHttpServerTransport] [node_dsapp006mv] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/172.16.0.10:9200, remoteAddress=/172.30.28.19:61869}
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:79) [transport-netty4-client-7.2.0.jar:7.2.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:276) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:268) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1375) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:276) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:918) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:174) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.35.Final.jar:4.1.35.Final]
	at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at io.netty.util.internal.PlatformDependent.allocateUninitializedArray(PlatformDependent.java:206) ~[?:?]
	at io.netty.buffer.PoolArena$HeapArena.newByteArray(PoolArena.java:679) ~[?:?]
	at io.netty.buffer.PoolArena$HeapArena.newChunk(PoolArena.java:689) ~[?:?]
	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245) ~[?:?]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:227) ~[?:?]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:147) ~[?:?]
	at io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:325) ~[?:?]
	at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:168) ~[?:?]
	at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:159) ~[?:?]
	at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:141) ~[?:?]
	at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147) ~[?:?]
	... 7 more
[2021-11-09T10:08:01,034][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node_dsapp006mv] fatal error in thread [Thread-564], exiting
java.lang.OutOfMemoryError: Java heap space
	at io.netty.util.internal.PlatformDependent.allocateUninitializedArray(PlatformDependent.java:206) ~[?:?]
	at io.netty.buffer.PoolArena$HeapArena.newByteArray(PoolArena.java:679) ~[?:?]
	at io.netty.buffer.PoolArena$HeapArena.newChunk(PoolArena.java:689) ~[?:?]
	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245) ~[?:?]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:227) ~[?:?]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:147) ~[?:?]
	at io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:325) ~[?:?]
	at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:168) ~[?:?]
	at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:159) ~[?:?]
	at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:141) ~[?:?]
	at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]

Christian_Dahlqvist · November 26, 2021, 8:59pm

Which version are you using? Do you have any custom or third party plugins installed?

Cordobel · November 26, 2021, 9:01pm

7.2

Ingest Attachment

Christian_Dahlqvist · November 26, 2021, 9:33pm

Elasticsearch 7.2 is very old. I would recommend upgrading to see if this is an issue that may have been resolved.

warkolm · November 28, 2021, 9:40pm

Agreed, 7.2 is EOL and no longer supported. Upgrading is the first best step in debugging this.

Cordobel · November 29, 2021, 2:32pm

Okay guys. Thank you so much for your time. The update will likely take some time. Last time we updated it required a lot of adaptations to our product and our team is too small to test all the consequences of updates on third party products. Also part of the team expressed concerns about the license changes. It is because of these points that we are stagnant in 7.2.

warkolm · November 29, 2021, 8:18pm

What are the concerns with the license?

Cordobel · December 10, 2021, 10:04am

The latest changes have left us in doubt about how free to use Elasticsearch is and how this use can or cannot be interpreted as simple consumption or reselling. But this, our legal support validates from time to time.

system · January 7, 2022, 10:05am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.