# Filebeat loads record with error whereas same record works from STDIN

**URL:** <https://discuss.elastic.co/t/filebeat-loads-record-with-error-whereas-same-record-works-from-stdin/67986>\
**Category:** Elasticsearch\
**Created:** [December 5, 2016, 5:39am UTC](https://discuss.elastic.co/t/filebeat-loads-record-with-error-whereas-same-record-works-from-stdin/67986 "2016-12-05T05:39:51Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Matt\_Simcox](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Matt\_Simcox](https://discuss.elastic.co/u/Matt_Simcox)\
**Post date:** [December 5, 2016, 5:39am UTC](https://discuss.elastic.co/t/filebeat-loads-record-with-error-whereas-same-record-works-from-stdin/67986/1 "2016-12-05T05:39:51Z")

</div>

Just getting started with ES and I've hit a snag.

I have logstash being fed data by filebeat on server1.  
Logstash then sends data to Elasticsearch on server2.

If I set logstash to read the following line from stdin it load successfully:

2016-12-02\_17:08:01.541 [transaction-2] INFO web.engine.TransactionHandler - Transaction Completed : SOCKETID=52290,TXNREFERENCE=1000000000000503,CLIENTID=10000000,RESPONSECODE=00,RESPONSETEXT=APPROVED,DURATION=78,TRANSACTIONTYPE=PURCHASE,INTERFACE=CREDITCARD

If I set filebeat to read this record in from a file it throws the following error in ES:

[webpay\_tran\_track-2016.12.02/nWK4cPgJSTC2zV18vLJrtA]]], type [["log", "PURCHASE"]]  
org.elasticsearch.indices.InvalidTypeNameException: mapping type name [["log", "PURCHASE"]] should not include ',' in it

```
    at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:296) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:277) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:323) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:241) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.cluster.service.ClusterService.runTasksForExecutor(ClusterService.java:555) [elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:896) [elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:451)[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:238) [elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201) [elasticsearch-5.0.1.jar:5.0.1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

```

[2016-12-05T16:31:02,540][DEBUG][o.e.a.b.TransportShardBulkAction] [zzmGnbh] [webpay\_tran\_track-2016.12.02][2] failed to execute bulk item (index) index {[webpay\_tran\_track-2016.12.02][["log", "PURCHASE"]][AVjNdsOuDmAdf7iQ47rE], source[{"cl  
ientid":"10000000","offset":1061,"resptext":"APPROVED","input\_type":"log","txnref":"1000000000000503","source":"C:\webpay\logs\tran\_track\_engine\_temp.log","socketval":"52290","message":"2016-12-02\_17:08:01.541 [transaction-2] INFO webpa  
y.engine.TransactionHandler - Transaction Completed : SOCKETID=52290,TXNREFERENCE=1000000000000503,CLIENTID=10000000,RESPONSECODE=00,RESPONSETEXT=APPROVED,DURATION=78,TRANSACTIONTYPE=PURCHASE,INTERFACE=CREDITCARD","type"  
:["log","PURCHASE"],"respcode":"00","interface":"CREDITCARD","tags":["beats\_input\_codec\_plain\_applied"],"duration":"78","@timestamp":"2016-12-02T06:08:01.541Z","@version":"1","beat":{"hostname":"sydwpayapp01","name":"sydwpayapp01","version"  
:"5.0.1"},"host":"sydwpayapp01","time":"2016-12-02\_17:08:01.541"}]}

the filter for this is:

filter {  
grok {  
patterns\_dir =\> ["./patterns"]  
match =\> { "message" =\> "%{TRANTRACK\_DATE:time} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} - %{NOTSPACE} Completed :\s\w\*=(%{WORD:socketval})?,\w\*=(%{WORD:txnref})?,\w\*=(%{WORD:clientid})?,\w\*=(%{WORD:respcode})?,\w\*=(%{MULTI\_WORD:resptext})?,\w\*=(%{WORD:duration})?,\w\*=(%{TRAN\_WITH\_SUBTYPE:type})?,\w\*=(%{WORD:interface})?" }  
overwrite =\> ["message"]   
}  
if ([message] =~ "Transaction Start") {  
drop {}  
}  
date {  
match =\> ["time","yyyy-MM-dd\_HH:mm:ss.SSS"]  
}  
}

I have tested this on a single server and it worked fine.

Que?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 5, 2016, 9:33am UTC](https://discuss.elastic.co/t/filebeat-loads-record-with-error-whereas-same-record-works-from-stdin/67986/2 "2016-12-05T09:33:14Z")

</div>

Hey,

the problem is not the message that does not get processed, but setting the type of the document, which is set to `[["log", "PURCHASE"]` - and triggering an error. Did you configure anything for the `document_type` setting in your filebeat configuration?

--Alex

---

<div class="post-metadata">

**Author:** ![Matt\_Simcox](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Matt\_Simcox](https://discuss.elastic.co/u/Matt_Simcox)\
**Post date:** [December 5, 2016, 10:39pm UTC](https://discuss.elastic.co/t/filebeat-loads-record-with-error-whereas-same-record-works-from-stdin/67986/3 "2016-12-05T22:39:06Z")

</div>

My filebeat config is

filebeat.prospectors:

- input\_type: log

output.logstash:

# The Logstash hosts

hosts: ["localhost:5043"]

---

<div class="post-metadata">

**Author:** ![Matt\_Simcox](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Matt\_Simcox](https://discuss.elastic.co/u/Matt_Simcox)\
**Post date:** [December 5, 2016, 11:57pm UTC](https://discuss.elastic.co/t/filebeat-loads-record-with-error-whereas-same-record-works-from-stdin/67986/4 "2016-12-05T23:57:20Z")

</div>

Think I've sorted this now. In my logstash elasticsearch output plugin I added:

document\_type =\> "%{[@metadata][type]}"

Found this at: [https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html](https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html)

In retrospect seems an obvious place to look (RTFM)

Seems to load now

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 2, 2017, 11:58pm UTC](https://discuss.elastic.co/t/filebeat-loads-record-with-error-whereas-same-record-works-from-stdin/67986/5 "2017-01-02T23:58:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.