# Kafka can collect java stack log but elastic search cannot. How to fix? **URL:** \ **Category:** Elasticsearch\ **Created:** [May 5, 2023, 5:21am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612 "2023-05-05T05:21:51Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Alwyn\_Tiu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alwyn_tiu/32/120679_2.png) [@Alwyn\_Tiu](https://discuss.elastic.co/u/Alwyn_Tiu)\ **Post date:** [May 5, 2023, 5:21am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612/1 "2023-05-05T05:21:52Z")
Kafka can collect java stack log but Elasticsearch cannot. How to fix? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 5, 2023, 5:23am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612/2 "2023-05-05T05:23:50Z")
It's not clear what you are asking sorry. ---
**Author:** ![Alwyn\_Tiu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alwyn_tiu/32/120679_2.png) [@Alwyn\_Tiu](https://discuss.elastic.co/u/Alwyn_Tiu)\ **Post date:** [May 5, 2023, 5:51am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612/3 "2023-05-05T05:51:42Z")
The log below can be collected to kafka and also can transfer to logstash | offset: 6686403225 isValid: true crc: null keySize: -1 valueSize: 2976 CreateTime: 1683261945722 baseOffset: 6686403220 lastOffset: 6686403233 baseSequence: -1 lastSequence: -1 producerEpoch: -1 partitionLeaderEpoch: 2 batchSize: 5985 magic: 2 compressType: SNAPPY position: 19496109 sequence: -1 headerKeys: payload: {"agent":{"version":"7.14.0","id":"xxx","hostname":"filebeat-filebeat-l7j7n","ephemeral\_id":"xxx","name":"filebeat-filebeat-l7j7n","type":"filebeat"},"datacenter":"xxx","xxx":"xxx","xxx":"xxx","xxx":"xxx","@timestamp":"2023-05-05T04:45:43.633Z","time":"2023-05-05 12:45:43.633","logger":"main.java.com.gw.datacenter.order.service.OrderServiceImpl","uuid":"xxx","stackTrace":"java.lang.Exception: JDBOrderRecords status error8006\n\tat main.java.com.gw.common.system.parse.JDBOrderHandle.getJDBOrderRecords(JDBOrderHandle.java:148)\n\tat main.java.com.gw.datacenter.order.service.OrderServiceImpl.insertOrder4JDB(OrderServiceImpl.java:201)\n\tat main.java.com.gw.datacenter.order.service.OrderServiceImpl$$FastClassBySpringCGLIB$$5d1c0a15.invoke()\n\tat org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)\n\tat org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)\n\tat org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692)\n\tat main.java.com.gw.datacenter.order.service.OrderServiceImpl$$EnhancerBySpringCGLIB$$1ff87b87.insertOrder4JDB()\n\tat main.java.com.gw.common.system.timer.Order4JDBTimer.lambda$execute$0(Order4JDBTimer.java:98)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n","tags":["beats\_input\_codec\_plain\_applied"],"thread":"pool-6-thread-107","ecs":{"version":"1.10.0"},"stack":"main.java.com.gw.datacenter.order.service.OrderServiceImpl:227[insertOrder4JDB]","level":"ERROR","xxx":"uat","log.type":"non-tracing","message":"JDBOrderRecords status error8006","fields":{"tags":"xxx"},"container":{"runtime":"docker","id":"xxx","image":{"name":"xxx"}},"xxx":"xxx"} But from logstash to Elasticsearch the log doesn't show ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 5, 2023, 5:59am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612/4 "2023-05-05T05:59:01Z")
What is inserting data into Kafka? Is it in exactly this format there? What does your Logstash config look like? Are there any errors in the Logstash logs? Can you output the final event processed by Logstash to file so we can see what the fully processed event looks like? ---
**Author:** ![Alwyn\_Tiu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alwyn_tiu/32/120679_2.png) [@Alwyn\_Tiu](https://discuss.elastic.co/u/Alwyn_Tiu)\ **Post date:** [May 5, 2023, 6:29am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612/5 "2023-05-05T06:29:31Z")
Configuration in kafka see below input { kafka{ bootstrap\_servers =\> "kafka-ip:port" topics =\> ["xxxx","xxxx"] codec =\> json session\_timeout\_ms =\> "30000" max\_poll\_records =\> "200" max\_poll\_interval\_ms =\> "600000" fetch\_min\_bytes =\> "1" request\_timeout\_ms =\> "305000" auto\_offset\_reset =\> "latest" group\_id =\> "xxxxxxx" } } out{ if [fields][tags] in ["xxxx","xxxxx"] { if "_dateparsefailure" not in [tags] and "grokparsefailure" not in [tags] and "timestampfailure" not in [tags] { elasticsearch { hosts =\> ["es-ip:port","es-ip:port"] index =\> "uat%{[k8s.container]}%{+YYYY.MM.dd}" user =\> xxxx password =\> "xxxxx" ssl =\> true ssl\_certificate\_verification =\> true cacert =\> "/etc/logstash/ssl/ca.pem" } } else { file { path =\> "/var/log/logstash/error/uat_%{[tags]}\_error-%{+YYYY.MM.dd}.log" codec =\> rubydebug } } } } } ---
**Author:** ![Alwyn\_Tiu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alwyn_tiu/32/120679_2.png) [@Alwyn\_Tiu](https://discuss.elastic.co/u/Alwyn_Tiu)\ **Post date:** [May 5, 2023, 6:30am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612/6 "2023-05-05T06:30:26Z")
This is the error in logstash elasticsearch - Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:id=\>"2023-05-05T05:52:40.099Z%{fingerprint}", :index=\>"%{[fields][tags]}-2023.05", :routing=\>nil}, {"message"=\>""stack":"main.java.com.gw.datacenter.order.service.OrderServiceImpl:227[insertOrder4JDB]","time":"2023-05-05 12:45:38.560","xxx":"xxx","xxx":"xxx","fields":{"tags":"xxx"},"@timestamp":"2023-05-05T04:45:38.560Z","xxx":"uat","tags":["beats\_input\_codec\_plain\_applied"],"datacenter":"xxx","thread":"pool-6-thread-107","uuid":"xxx","message":"JDBOrderRecords status error8006","container":{"image":{"name":"img.xxx"},"id":"xxx","runtime":"docker"},"logger":"main.java.com.gw.datacenter.order.service.OrderServiceImpl","agent":{"hostname":"filebeat-filebeat-l7j7n","type":"filebeat","version":"7.14.0","name":"filebeat-filebeat-l7j7n","ephemeral\_id":"xxx","id":"xxx"},"k8s.node":"xxx","level":"ERROR","ecs":{"version":"1.10.0"},"xxx":"xxx","log.type":"non-tracing","stackTrace":"java.lang.Exception: JDBOrderRecords status error8006\n\tat main.java.com.gw.common.system.parse.JDBOrderHandle.getJDBOrderRecords(JDBOrderHandle.java:148)\n\tat main.java.com.gw.datacenter.order.service.OrderServiceImpl.insertOrder4JDB(OrderServiceImpl.java:201)\n\tat main.java.com.gw.datacenter.order.service.OrderServiceImpl$$FastClassBySpringCGLIB$$5d1c0a15.invoke()\n\tat org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)\n\tat org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)\n\tat org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692)\n\tat main.java.com.gw.datacenter.order.service.OrderServiceImpl$$EnhancerBySpringCGLIB$$1ff87b87.insertOrder4JDB()\n\tat main.java.com.gw.common.system.timer.Order4JDBTimer.lambda$execute$0(Order4JDBTimer.java:98)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n"", "@timestamp"=\>2023-05-05T05:52:40.099Z, "path"=\>"/usr/share/logstash/bin/bb.txt", "@version"=\>"1", "host"=\>"xxx"}], :response=\>{"index"=\>{"\_index"=\>"%{[fields][tags]}-2023.05", "\_type"=\>"\_doc", "\_id"=\>"2023-05-05T05:52:40.099Z%{fingerprint}", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"object mapping for [host] tried to parse field [host] as object, but found a concrete value"}}}} ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 5, 2023, 6:34am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612/7 "2023-05-05T06:34:16Z")
> [@Alwyn\_Tiu](#): > > "reason"=\>"object mapping for [host] tried to parse field [host] as object, but found a concrete value" You have a mapping conflict. In the index you are indexing this into the `host` field is defined as an object, but in the event shown it is a string: > [@Alwyn\_Tiu](#): > > "host"=\>"xxx" You will need to change the structure of the event as you can not have multiple mappings for the same field. ---
**Author:** ![Alwyn\_Tiu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alwyn_tiu/32/120679_2.png) [@Alwyn\_Tiu](https://discuss.elastic.co/u/Alwyn_Tiu)\ **Post date:** [May 5, 2023, 7:29am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612/8 "2023-05-05T07:29:39Z")
All the "xxx" is actually us who edited the actual text before posting here to avoid leaking confidential info. Any other possibilities as to why the log cannot go into Elasticsearch? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 5, 2023, 7:30am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612/9 "2023-05-05T07:30:40Z")
The error message is very clear. The data (`host` field should be an object, not a string) does not match what is already in the index and therefore need to be changed. Try changing `"host": "xxx"` to `"host": {"name":"xxx"}` or something similar. You may want to check the mappings to see how the `host` field is structured in other documents. ---
**Author:** ![Alwyn\_Tiu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alwyn_tiu/32/120679_2.png) [@Alwyn\_Tiu](https://discuss.elastic.co/u/Alwyn_Tiu)\ **Post date:** [May 5, 2023, 10:08am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612/10 "2023-05-05T10:08:44Z")
Already tried `"host": {"name":"xxx"}` but still doesn't show in Elasticsearch. We tried configuring from logstash to output the log to Elasticsearch and local machine. The log can be created normally in logstash but in Elasticsearch cannot. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 5, 2023, 10:10am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612/11 "2023-05-05T10:10:39Z")
Please show what you canged and what the result was. Did the error message change? ---
**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\ **Post date:** [May 5, 2023, 11:02am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612/14 "2023-05-05T11:02:30Z")
Hi @Alwyn_Tiu, Can you share those logs as `code` or text snippets rather than images as you did previously? ---
**Author:** ![Alwyn\_Tiu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alwyn_tiu/32/120679_2.png) [@Alwyn\_Tiu](https://discuss.elastic.co/u/Alwyn_Tiu)\ **Post date:** [May 9, 2023, 3:40am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612/17 "2023-05-09T03:40:19Z")
The problem is solved. We found out that there is a "ignore\_above: 1500" in the configuration. After we adjusted to 3500 the log can now be shown in Elasticsearch. We'd like to ask if ignore\_above is a default restriction for the length of the log? Or is there any other to restrictions like block the log above the length being set and then record an error message? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 6, 2023, 3:41am UTC](https://discuss.elastic.co/t/kafka-can-collect-java-stack-log-but-elastic-search-cannot-how-to-fix/332612/18 "2023-06-06T03:41:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.