# Elasticsearch crashes on filebeat pipeline

**URL:** <https://discuss.elastic.co/t/elasticsearch-crashes-on-filebeat-pipeline/137112>\
**Category:** Elasticsearch\
**Created:** [June 23, 2018, 9:46am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-on-filebeat-pipeline/137112 "2018-06-23T09:46:50Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ben5556](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben5556/32/101629_2.png) [@ben5556](https://discuss.elastic.co/u/ben5556)\
**Post date:** [June 23, 2018, 9:46am UTC](https://discuss.elastic.co/t/elasticsearch-crashes-on-filebeat-pipeline/137112/1 "2018-06-23T09:46:50Z")

</div>

Would like some input Elasticsearch on why our cluster is crashing. Below is an excerpt of the full stack trace of the error as the message body has a limit and I can't paste the full stack trace (How to upload the full log file ?)

> [2018-06-20T03:52:58,913][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ip-172-31-7-59.ap-southeast-2.compute.internal] fatal error in thread [elasticsearch[ip-172-31-7-59.ap-southeast-2.compute.internal][masterService#updateTask][T#1]], exiting  
> java.lang.StackOverflowError: null  
> at java.util.HashMap.hash(Unknown Source) ~[?:1.8.0\_171]  
> at java.util.LinkedHashMap.get(Unknown Source) ~[?:1.8.0\_171]  
> at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseNested(ObjectMapper.java:214) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:168) ~[elasticsearch-6.3.0.jar:6.3.0]  
> .......  
> .......  
> at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:278) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:199) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.index.mapper.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:131) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:112) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:92) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:736) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:264) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:267) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:197) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:132) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-6.3.0.jar:6.3.0]  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:1.8.0\_171]  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:1.8.0\_171]  
> at java.lang.Thread.run(Unknown Source) [?:1.8.0\_171]

This fatal error occurs when ingesting logs to Elasticsearch which goes through a pipeline parser. This crash only occurs when parsing a specific app logs, for other apps it works fine. Below is the parser pipeline which does some simple parsing. Are you able to point why this issue is occurring ?

"standard\_mule\_log\_parser": {  
"description": "Pipeline that parses standard format logs and splits log message into fields",  
"processors": [  
{  
"grok": {  
"field": "message",  
"patterns": [  
"%{TIMESTAMP\_ISO8601:log-timestamp} %{GRABBETWEENBRACKETS:threadname} %{LOGLVL:loglevel} %{UNTILNEXTSPACECHAR:classname} %{MULTILINE:message}"  
],  
"pattern\_definitions": {  
"GRABBETWEENBRACKETS": """[(._)]""",  
"LOGLVL": "(INFO|ERROR|WARN|DEBUG|TRACE)",  
"UNTILNEXTSPACECHAR": """(\s_[^\s]+)""",  
"MULTILINE": "(?m).\*"  
}  
}  
},  
{  
"gsub": {  
"field": "message",  
"pattern": "-",  
"replacement": "",  
"ignore\_missing": true  
}  
},  
{  
"kv": {  
"field": "message",  
"field\_split": ", ",  
"value\_split": ": ",  
"ignore\_failure": true  
}  
},  
{  
"trim": {  
"field": "message",  
"ignore\_missing": true  
}  
},  
{  
"trim": {  
"field": "classname",  
"ignore\_missing": true  
}  
},  
{  
"rename": {  
"field": " System",  
"target\_field": "System",  
"ignore\_missing": true  
}  
},  
{  
"grok": {  
"field": "source",  
"patterns": [  
"%{APPNAME:app\_name}"  
],  
"pattern\_definitions": {  
"APPNAME": "(test|test1)"  
}  
}  
},  
{  
"convert": {  
"field": "Elapsed",  
"type": "integer",  
"ignore\_missing": true  
}  
},  
{  
"date": {  
"field": "log-timestamp",  
"formats": [  
"yyyy-MM-dd HH:mm:ss,SSS"  
],  
"timezone": "Pacific/Auckland"  
}  
}  
]  
}

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [June 29, 2018, 1:16pm UTC](https://discuss.elastic.co/t/elasticsearch-crashes-on-filebeat-pipeline/137112/2 "2018-06-29T13:16:14Z")

</div>

Could you post a complete stacktrace by any chance?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 27, 2018, 1:16pm UTC](https://discuss.elastic.co/t/elasticsearch-crashes-on-filebeat-pipeline/137112/3 "2018-07-27T13:16:25Z")

</div>

Closed on July 27, 2018, 1:16pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
