Hello, I am a Java developer. My online application encountered problems after the operation and maintenance personnel introduced elastic-apm-java. It is related to heap usage. I have recommended that the operation and maintenance personnel upgrade to the latest apm version【they haven't done it yet，At the moment I think we need to give them a strong statement】, but I still don't understand the cause of the problem. I hope to get some answers from you. Thank you!
java version:
<java.version>17</java.version>
APM Server version:
1.30.0
APM Agent language and version:
language: java
apm version:1.30.0
Fresh install or upgraded from other version?
Fresh install
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
k8s Pod restarts due to jvm memory overload，This phenomenon often occurs after the introduction of elastic apm。We previously thought it was related to scheduled tasks, but after moving out of the scheduled task module, this phenomenon still exists, but the frequency is not so high.
The following two pictures are large objects analyzed using jprofiler：
The following is a section of thread top list data provided by operation and maintenance personnel:
</>[1] Busy(99.9%) thread(228/0xe4) stack of java process(6) under user(root):
"http-nio-80-exec-23" #218 daemon prio=5 os_prio=0 cpu=836101504.82ms elapsed=1826109.15s tid=0x00007f60c2ded1c0 nid=0xe4 runnable [0x00007f5e90ee2000]
java.lang.Thread.State: RUNNABLE
at co.elastic.apm.agent.impl.transaction.Span.tryToCompressComposite(Span.java:419)
at co.elastic.apm.agent.impl.transaction.Span.tryToCompress(Span.java:350)
at co.elastic.apm.agent.impl.transaction.Span.afterEnd(Span.java:325)
at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:476)
at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:458)
at co.elastic.apm.agent.redisson.RedisConnectionInstrumentation$AdviceClass.afterSend(RedisConnectionInstrumentation.java:79)
at java.lang.invoke.LambdaForm$DMH/0x0000000800d3c000.invokeStatic(java.base@17.0.2/LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/0x0000000800efd000.linkToTargetMethod(java.base@17.0.2/LambdaForm$MH)
at org.redisson.client.RedisConnection.send(RedisConnection.java:206)
at org.redisson.command.RedisExecutor.sendCommand(RedisExecutor.java:544)
at org.redisson.command.RedisExecutor.lambda$execute$2(RedisExecutor.java:159)
at org.redisson.command.RedisExecutor$$Lambda$1916/0x0000000801d951c8.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@17.0.2/CompletableFuture.java:863)
at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(java.base@17.0.2/CompletableFuture.java:887)
at java.util.concurrent.CompletableFuture.whenComplete(java.base@17.0.2/CompletableFuture.java:2325)
at org.redisson.command.RedisExecutor.execute(RedisExecutor.java:147)
at org.redisson.command.CommandAsyncService.async(CommandAsyncService.java:529)
at org.redisson.command.CommandAsyncService.readAsync(CommandAsyncService.java:298)
at org.redisson.spring.data.connection.RedissonConnection.read(RedissonConnection.java:743)
at org.redisson.spring.data.connection.RedissonConnection.get(RedissonConnection.java:479)
at org.springframework.data.redis.connection.DefaultStringRedisConnection.get(DefaultStringRedisConnection.java:438)
at org.springframework.data.redis.core.DefaultValueOperations$1.inRedis(DefaultValueOperations.java:58)
at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:61)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:191)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:97)
at org.springframework.data.redis.core.DefaultValueOperations.get(DefaultValueOperations.java:54)
at com.xxx.util.RedisUtil.getValue(RedisUtil.java:106)
at com.xxx.service.impl.signin.SignInJigsawsServiceImpl.get(SignInJigsawsServiceImpl.java:29)
at com.xxx.service.impl.signin.SignInJigsawsServiceImpl.getForWx(SignInJigsawsServiceImpl.java:46)
at com.xxx.service.impl.signin.SignInIterServiceImpl.updateSignInStatusIfNecessary(SignInIterServiceImpl.java:546)
at com.xxx.service.impl.signin.SignInIterServiceImpl$$FastClassBySpringCGLIB$$46cf7474.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386)
at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703)
at com.xxx.service.impl.signin.SignInIterServiceImpl$$EnhancerBySpringCGLIB$$a84c3165.updateSignInStatusIfNecessary()
at com.xxx.yyy.hwb.service.impl.PartyCServiceImpl.genSignInStatus(PartyCServiceImpl.java:802)
at com.xxx.yyy.hwb.service.impl.PartyCServiceImpl.simplifySignInCenterIndex(PartyCServiceImpl.java:548)
at com.xxx.yyy.hwb.controller.PartyCController.simplifySignInCenterIndex(PartyCController.java:123)
at com.xxx.yyy.hwb.controller.PartyCController$$FastClassBySpringCGLIB$$5a2489b5.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707)
at com.xxx.yyy.hwb.controller.PartyCController$$EnhancerBySpringCGLIB$$d56d634.simplifySignInCenterIndex()
at jdk.internal.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.2/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@17.0.2/Method.java:568)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:529)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at com.lylog.filter.LogFilter.doFilter(LogFilter.java:85)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:765)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(java.base@17.0.2/Thread.java:833)
[2] Busy(99.9%) thread(321/0x141) stack of java process(6) under user(root):
"pool-5-thread-2" #312 prio=5 os_prio=0 cpu=18771144.43ms elapsed=1826108.63s tid=0x00007f5f201ef100 nid=0x141 runnable [0x00007f5e8b187000]
java.lang.Thread.State: RUNNABLE
at co.elastic.apm.agent.impl.transaction.Span.tryToCompressComposite(Span.java:419)
at co.elastic.apm.agent.impl.transaction.Span.tryToCompress(Span.java:350)
at co.elastic.apm.agent.impl.transaction.Span.afterEnd(Span.java:325)
at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:476)
at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:458)
at co.elastic.apm.agent.redisson.RedisConnectionInstrumentation$AdviceClass.afterSend(RedisConnectionInstrumentation.java:79)
at java.lang.invoke.LambdaForm$DMH/0x0000000800d3c000.invokeStatic(java.base@17.0.2/LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/0x0000000800efd000.linkToTargetMethod(java.base@17.0.2/LambdaForm$MH)
at org.redisson.client.RedisConnection.send(RedisConnection.java:206)
at org.redisson.command.RedisExecutor.sendCommand(RedisExecutor.java:544)
at org.redisson.command.RedisExecutor.lambda$execute$2(RedisExecutor.java:159)
at org.redisson.command.RedisExecutor$$Lambda$1916/0x0000000801d951c8.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@17.0.2/CompletableFuture.java:863)
at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(java.base@17.0.2/CompletableFuture.java:887)
at java.util.concurrent.CompletableFuture.whenComplete(java.base@17.0.2/CompletableFuture.java:2325)
at org.redisson.command.RedisExecutor.execute(RedisExecutor.java:147)
at org.redisson.command.CommandAsyncService.async(CommandAsyncService.java:529)
at org.redisson.command.CommandAsyncService.readAsync(CommandAsyncService.java:298)
at org.redisson.spring.data.connection.RedissonConnection.read(RedissonConnection.java:743)
at org.redisson.spring.data.connection.RedissonConnection.get(RedissonConnection.java:479)
at org.springframework.data.redis.connection.DefaultStringRedisConnection.get(DefaultStringRedisConnection.java:438)
at org.springframework.data.redis.core.DefaultValueOperations$1.inRedis(DefaultValueOperations.java:58)
at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:61)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:191)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:97)
at org.springframework.data.redis.core.DefaultValueOperations.get(DefaultValueOperations.java:54)
at com.xxx.service.TraceLinkService.recordHeaderIntercept(TraceLinkService.scala:156)
at com.xxx.event.RecieptSender.visitToUserEvent(RecieptSender.scala:159)
at jdk.internal.reflect.GeneratedMethodAccessor184.invoke(Unknown Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.2/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@17.0.2/Method.java:568)
at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:144)
- locked <0x000000071f8b5a30> (a com.google.common.eventbus.Subscriber$SynchronizedSubscriber)
at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.2/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.2/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17.0.2/Thread.java:833)
[3] Busy(7.8%) thread(38/0x26) stack of java process(6) under user(root):
"elastic-apm-server-reporter" #23 daemon prio=5 os_prio=0 cpu=68498116.77ms elapsed=1826132.75s tid=0x00007f60c11aeb60 nid=0x26 waiting on condition [0x00007f60527f7000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.2/Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@17.0.2/LockSupport.java:376)
at co.elastic.apm.agent.report.disruptor.ExponentionallyIncreasingSleepingWaitStrategy.applyWaitMethod(ExponentionallyIncreasingSleepingWaitStrategy.java:81)
at co.elastic.apm.agent.report.disruptor.ExponentionallyIncreasingSleepingWaitStrategy.waitFor(ExponentionallyIncreasingSleepingWaitStrategy.java:64)
at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at java.lang.Thread.run(java.base@17.0.2/Thread.java:833)
[4] Busy(3.9%) thread(236/0xec) stack of java process(6) under user(root):
"http-nio-80-exec-31" #226 daemon prio=5 os_prio=0 cpu=3537803.51ms elapsed=1826109.14s tid=0x00007f60c35e14e0 nid=0xec waiting on condition [0x00007f5e906dd000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.2/Native Method)
- parking to wait for <0x0000000789648d98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@17.0.2/LockSupport.java:341)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.2/AbstractQueuedSynchronizer.java:506)
at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.2/ForkJoinPool.java:3463)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.2/ForkJoinPool.java:3434)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.2/AbstractQueuedSynchronizer.java:1623)
at java.util.concurrent.LinkedBlockingQueue.take(java.base@17.0.2/LinkedBlockingQueue.java:435)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:141)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1114)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(java.base@17.0.2/Thread.java:833)
[5] Busy(3.9%) thread(380/0x17c) stack of java process(6) under user(root):
"current-time-millis" #363 daemon prio=5 os_prio=0 cpu=19112365.52ms elapsed=1826071.80s tid=0x00007f5fc80b4140 nid=0x17c runnable [0x00007f5e871fc000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.2/Native Method)
- parking to wait for <0x0000000789650340> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@17.0.2/LockSupport.java:252)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@17.0.2/AbstractQueuedSynchronizer.java:1672)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17.0.2/ScheduledThreadPoolExecutor.java:1182)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17.0.2/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.2/ThreadPoolExecutor.java:1062)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.2/ThreadPoolExecutor.java:1122)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.2/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17.0.2/Thread.java:833)
</>
dockerfile config:
FROM XXX/library/xxx-apm-base:debian-jdk17
VOLUME /tmp
ADD target/xxx-0.0.1-SNAPSHOT.jar xxx.jar
ENV apm_service_name=xxx
ENV apm_server_urls=server.monitor:8200
ENV apm_environment=prod
ENTRYPOINT ["/bin/sh","-c","java
${JAVA_OPTS}
-Djava.security.egd=file:/dev/./urandom
-javaagent:/ems/elastic-apm-agent/elastic-apm-agent-1.30.0.jar
-Delastic.apm.service_name=$apm_service_name
-Delastic.apm.server_urls=$apm_server_urls
-Delastic.apm.environment=$apm_environment
-Delastic.apm.secret_token=
-Delastic.apm.application_packages=org.example
-Drocketmq.client.logUseSlf4j=true
-Dlog4j2.formatMsgNoLookups=true
-jar xxx.jar
--spring.profiles.active=prod
--spring.config.use-legacy-processing=true"]
Provide logs and/or server output (if relevant):
