Hi everybody,
I have a big problem with rollup jobs.
When I try to execute a rollup job of 51 million recors, I don't get the expected result.
I have various terms and metrics to aggregate; down below you can check the rollup job configuration that I used.
{
"config": {
"id": "Rollup-big-index2",
"index_pattern": "filebeat-7.16.2-2022.11.14-prod",
"rollup_index": "rollup-big-index7000",
"cron": "0 48 * * * ?",
"groups": {
"date_histogram": {
"fixed_interval": "15m",
"field": "@timestamp",
"time_zone": "Europe/Rome"
},
"terms": {
"fields": [
"source.geo.country_name",
"destination.port",
"destination.domain",
"http.request.method",
"http.version",
"user_agent.os.name",
"http.backend_name",
"user.name",
"user_agent.name",
"url.easymarket_category",
"http.response.status_code"
]
}
},
"metrics": [
{
"field": "http.response.duration",
"metrics": [
"avg",
"max",
"min"
]
},
{
"field": "http.response.body.bytes",
"metrics": [
"avg",
"max",
"sum",
"min"
]
}
],
"timeout": "20s",
"page_size": 3000
}
Problem is: when I use the visualization option, the result between the original and the rollup index doesn't match. For example, the field destination.domain lost some domain value, and I noticed some empty spaces in the graphic.
Here an example:
Just something I would like to add: when the rollup job starts, the Elasticsearch CPU process spikes between 150 to 190% (I'm using debianVM powered by vmSphere by the way). I think those two things could be related.
If it can be helpful, I'd like to add the hot_threads api result
::: {elsrc-m02-staging}{H-uiw_vSQ5KV0UsZ-Qf2DA}{eohlLdyUR9ai6Z2R1iIILA}{172.16.37.88}{172.16.37.88:9300}{lmr}{ml.machine_memory=2046341120, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=1073741824, transform.node=false}
Hot threads at 2022-11-29T16:40:04.330Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {elsrc-m01-staging}{sx6sFIAQR_mGlkCZyS5a4w}{UAxQIE6yQdiIxi6u_L6ixg}{172.16.37.87}{172.16.37.87:9300}{lmr}{ml.machine_memory=2046341120, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=1073741824, transform.node=false}
Hot threads at 2022-11-29T16:40:04.339Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {elsrc-d02-staging}{5Kgu_uRPRh61JOAWB85CVA}{b2KPyo1vT3S3QIejTwxqgQ}{172.16.37.95}{172.16.37.95:9300}{cdfhilrstw}{ml.machine_memory=6197186560, ml.max_open_jobs=512, box_type=hot, xpack.installed=true, ml.max_jvm_size=1073741824, transform.node=true}
Hot threads at 2022-11-29T16:40:04.305Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {elsrc-d01-staging}{9KishCb5S-2vFI2mmObQbg}{qe9p-M6MTdmsHbI1MKsdoQ}{172.16.37.90}{172.16.37.90:9300}{cdfhilrstw}{ml.machine_memory=6197186560, ml.max_open_jobs=512, box_type=hot, xpack.installed=true, ml.max_jvm_size=1073741824, transform.node=true}
Hot threads at 2022-11-29T16:40:04.307Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {elsrc-c01-staging}{Yz_kg2Z3RtCAWbrhzDn14Q}{ULQPOfGYT5u3hL4E_mKCVw}{172.16.37.86}{172.16.37.86:9300}{ilr}{ml.machine_memory=2046341120, xpack.installed=true, transform.node=false, ml.max_open_jobs=512, ml.max_jvm_size=1073741824}
Hot threads at 2022-11-29T16:40:04.315Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=1.6%, other=98.4%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elsrc-c01-staging][transport_worker][T#2]'
unique snapshot
org.elasticsearch.xpack.security.audit.AuditUtil.extractRequestId(AuditUtil.java:77)
org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:184)
org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$5905/0x00000008018cfc88.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
app//org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)
org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102)
org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)
org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182)
org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124)
app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)
app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154)
app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
app//org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95)
app//org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73)
app//org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407)
app//org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:492)
app//org.elasticsearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:106)
app//org.elasticsearch.rest.action.document.RestBulkAction$$Lambda$6106/0x0000000801a9a2a0.accept(Unknown Source)
app//org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:109)
org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:90)
org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$6005/0x0000000801a75628.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:84)
org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator$$Lambda$6008/0x0000000801a75cc0.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:94)
org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:78)
org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:85)
org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$5974/0x0000000801a4a788.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
app//org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)
app//org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389)
org.elasticsearch.xpack.security.authc.AuthenticatorChain.writeAuthToContext(AuthenticatorChain.java:376)
org.elasticsearch.xpack.security.authc.AuthenticatorChain.finishAuthentication(AuthenticatorChain.java:352)
org.elasticsearch.xpack.security.authc.AuthenticatorChain.maybeLookupRunAsUser(AuthenticatorChain.java:205)
org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$doAuthenticate$1(AuthenticatorChain.java:128)
org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$5977/0x0000000801a4b040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141)
org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$3(AuthenticatorChain.java:175)
org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$5981/0x0000000801a4bd68.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$3(RealmsAuthenticator.java:219)
org.elasticsearch.xpack.security.authc.RealmsAuthenticator$$Lambda$5984/0x0000000801a6e478.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141)
org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$0(RealmsAuthenticator.java:164)
org.elasticsearch.xpack.security.authc.RealmsAuthenticator$$Lambda$5988/0x0000000801a6eb08.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:155)
org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$6002/0x0000000801a74f98.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:242)
org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:139)
org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$5994/0x0000000801a6fa30.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55)
org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:134)
org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105)
org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$2(RealmsAuthenticator.java:148)
org.elasticsearch.xpack.security.authc.RealmsAuthenticator$$Lambda$5983/0x0000000801a6e238.accept(Unknown Source)
org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117)
org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:233)
org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:84)
org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:171)
org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$5979/0x0000000801a4b4a0.accept(Unknown Source)
org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)
org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165)
org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$5979/0x0000000801a4b4a0.accept(Unknown Source)
org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)
org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165)
org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$5979/0x0000000801a4b4a0.accept(Unknown Source)
org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)
org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165)
org.elasticsearch.xpack.security.authc.AuthenticatorChain$$Lambda$5979/0x0000000801a4b4a0.accept(Unknown Source)
org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117)
org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:143)
org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:104)
org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149)
org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127)
org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:79)
app//org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:327)
app//org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:393)
app//org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:245)
app//org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:382)
app//org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:461)
app//org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:357)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:32)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:18)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:48)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
100.0% [cpu=1.1%, other=98.9%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elsrc-c01-staging][transport_worker][T#1]'
unique snapshot
java.base@17.0.1/java.lang.StringLatin1.hashCode(StringLatin1.java:196)
java.base@17.0.1/java.lang.String.hashCode(String.java:2339)
java.base@17.0.1/java.util.HashMap.hash(HashMap.java:338)
java.base@17.0.1/java.util.HashMap.put(HashMap.java:610)
java.base@17.0.1/java.util.HashSet.add(HashSet.java:221)
io.netty.handler.codec.DefaultHeaders.names(DefaultHeaders.java:287)
io.netty.handler.codec.HeadersUtils.namesAsString(HeadersUtils.java:107)
io.netty.handler.codec.http.DefaultHttpHeaders.names(DefaultHttpHeaders.java:349)
org.elasticsearch.http.netty4.Netty4HttpRequest$HttpHeadersMap.entrySet(Netty4HttpRequest.java:307)
app//org.elasticsearch.http.HttpClientStatsTracker$ClientStatsBuilder.getFirstValueForHeader(HttpClientStatsTracker.java:234)
app//org.elasticsearch.http.HttpClientStatsTracker$ClientStatsBuilder.update(HttpClientStatsTracker.java:221)
app//org.elasticsearch.http.HttpClientStatsTracker.updateClientStats(HttpClientStatsTracker.java:135)
app//org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:354)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:32)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:18)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:48)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
::: {elsrc-m03-staging}{F1WVhUAaRKiQ0SCwZBVbjQ}{XNCiFW_SR4amGApVxOMTkA}{172.16.37.89}{172.16.37.89:9300}{lmr}{ml.machine_memory=2046341120, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=1073741824, transform.node=false}
Hot threads at 2022-11-29T16:40:04.337Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
Please could you give me some help to find a solution? I have lost a lot of time trying to solve this problem.
Thank you for helping,
Nicolò