# FIleBeat not pushing few lines to Logstash

**URL:** <https://discuss.elastic.co/t/filebeat-not-pushing-few-lines-to-logstash/289011>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 11, 2021, 6:02pm UTC](https://discuss.elastic.co/t/filebeat-not-pushing-few-lines-to-logstash/289011 "2021-11-11T18:02:15Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![nandatibco](https://avatars.discourse-cdn.com/v4/letter/n/ed8c4c/32.png) [@nandatibco](https://discuss.elastic.co/u/nandatibco)\
**Post date:** [November 11, 2021, 6:02pm UTC](https://discuss.elastic.co/t/filebeat-not-pushing-few-lines-to-logstash/289011/1 "2021-11-11T18:02:15Z")

</div>

File beat is not pushing all lines in a file to Logstash, and it is not consistent, it skips 3 lines in between. Im new to File Beat, please suggest if I missed any configuration.  
For example, my log file looks like this:  
Line1  
Line2  
Line3  
Line4  
Some time I see only line1 is pushed, next time I see only line3 is pushed, any time it is not pushing all lines to Logstash.  
Randomly pushes only 1 line always.

---

<div class="post-metadata">

**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\
**Post date:** [November 11, 2021, 6:12pm UTC](https://discuss.elastic.co/t/filebeat-not-pushing-few-lines-to-logstash/289011/2 "2021-11-11T18:12:12Z")

</div>

Is your single message over multiple lines? Or is every line its own message?

If its the first case, you need to add multiline configuration to your log input in filebeat.yml.

Do you have 2-3 example lines?

---

<div class="post-metadata">

**Author:** ![nandatibco](https://avatars.discourse-cdn.com/v4/letter/n/ed8c4c/32.png) [@nandatibco](https://discuss.elastic.co/u/nandatibco)\
**Post date:** [November 12, 2021, 4:17am UTC](https://discuss.elastic.co/t/filebeat-not-pushing-few-lines-to-logstash/289011/3 "2021-11-12T04:17:39Z")

</div>

Hi Marius,  
Every line its own message. At the end of every line there is a carriage return(empty line). File contains json segments as below:  
Example:  
{ "payloadID":"22810.ABC.XYZ.16366895388792810", "correlationID":"ABC.XYZ.16366895388792810", "country":"AB", "serviceName":"XYZ", "operationName":"getcustomerDetails", "type":"audit", "servicePayload":"\<?xml version=\"1.0\" encoding=\"UTF-8\"?\>\<ns0:getcustomerDetails xmlns:ns0="[http://www.abc.com/ErpSchema](http://www.abc.com/ErpSchema)"\>ns0:customerid40008\</ns0:customerid\>\</ns0:getcustomerDetails\>", "timestamp":"2021-11-12T04:58:58.879+01:00"}

{ "payloadID":"22810.ABC.XYZ.16366895388792810", "correlationID":"ABC.XYZ.16366895388792810", "country":"AB", "serviceName":"XYZ", "operationName":"getcustomerDetails", "type":"audit", "servicePayload":"\<?xml version=\"1.0\" encoding=\"UTF-8\"?\>\<v1:customerDetailsRequest xmlns:soapenv="[http://schemas.xmlsoap.org/soap/envelope/](http://schemas.xmlsoap.org/soap/envelope/)" xmlns:v1="[http://www.abc.com/myshop/mediation/normalizedMoABl/v1](http://www.abc.com/myshop/mediation/normalizedMoABl/v1)" xmlns:SOAP-ENV="[http://schemas.xmlsoap.org/soap/envelope/](http://schemas.xmlsoap.org/soap/envelope/)"\>\<v1:webshop banner="GY" country="GY" division="?" webshopURL="?"\>v1:webshopNameXYZ\</v1:webshopName\>\</v1:webshop\>v1:ERPCustomerIDv1:accountNumber40008\</v1:accountNumber\>v1:project?\</v1:project\>v1:language?\</v1:language\>\</v1:ERPCustomerID\>v1:checkString?\</v1:checkString\>\</v1:customerDetailsRequest\>", "timestamp":"2021-11-12T04:58:58.879+01:00"}

{ "payloadID":"22810.ABC.XYZ.16366895388792810", "correlationID":"ABC.XYZ.16366895388792810", "country":"AB", "serviceName":"XYZ", "operationName":"getcustomerDetails", "type":"audit", "servicePayload":"\<?xml version=\"1.0\" encoding=\"UTF-8\"?\>\<getcustomerDetailsResponse xmlns="[http://www.abc.com/ErpSchema](http://www.abc.com/ErpSchema)" xmlns:SOAP-ENV="[http://schemas.xmlsoap.org/soap/envelope/](http://schemas.xmlsoap.org/soap/envelope/)" xmlns:soap="[http://schemas.xmlsoap.org/soap/envelope/](http://schemas.xmlsoap.org/soap/envelope/)"\>40008aaa bbb cccaaa bbb ccc72aaa bbb cccaaa bbb cccRIEDSTR. 4284082ABCDEFGHABfalse94366 abcd efgn ancd efghABfalsetruefalseOptional[https://www.abcd-easypanel.AB](https://www.abcd-easypanel.AB)RESResiABntialAaaa bbb ccc)truetruefalse", "timestamp":"2021-11-12T04:58:59.254+01:00"}

{ "payloadID":"22810.ABC.XYZ.16366895388792810", "correlationID":"ABC.XYZ.16366895388792810", "country":"AB", "serviceName":"XYZ", "operationName":"getcustomerDetails", "type":"audit", "servicePayload":"\<?xml version=\"1.0\" encoding=\"UTF-8\"?\>\<ns0:customerDetails xmlns:ns0="[http://www.abc.com/myshop/mediation/normalizedMoABl/v1](http://www.abc.com/myshop/mediation/normalizedMoABl/v1)"\>ns0:ERPCustomerIDns0:accountNumber40008\</ns0:accountNumber\>ns0:languageAB\</ns0:language\>\</ns0:ERPCustomerID\>ns0:companyNameaaa bbb ccc\</ns0:companyName\>ns0:managingBranch72\</ns0:managingBranch\>ns0:billingABtailsns0:billingAddressns0:companyNameaaa bbb ccc\</ns0:companyName\>ns0:address1RIEDSTR. 42\</ns0:address1\>ns0:address2aaa bbb ccc\</ns0:address2\>ns0:postalCoAB84082\</ns0:postalCoAB\>ns0:cityABCDEFGH\</ns0:city\>ns0:countryAB\</ns0:country\>\</ns0:billingAddress\>\</ns0:billingABtails\>ns0:ABliveryABtailsns0:ABliveryAddressns0:addressID0\</ns0:addressID\>ns0:companyNameaaa bbb ccc\</ns0:companyName\>ns0:address1RIEDSTR. 42\</ns0:address1\>ns0:address2aaa bbb ccc\</ns0:address2\>ns0:address3/ns0:postalCoAB84082\</ns0:postalCoAB\>ns0:cityABCDEFGH\</ns0:city\>ns0:countryAB\</ns0:country\>ns0:provinceCoAB/ns0:ABfaultABliveryAddressIDtrue\</ns0:ABfaultABliveryAddressID\>\</ns0:ABliveryAddress\>\</ns0:ABliveryABtails\>ns0:activeCustomerfalse\</ns0:activeCustomer\>ns0:customerInfons0:custPurchaseOrdNoOptional\</ns0:custPurchaseOrdNo\>ns0:salesRepID94\_366\</ns0:salesRepID\>ns0:canReturnOrABrns0:accessRighttrue\</ns0:accessRight\>\</ns0:canReturnOrABr\>ns0:isExpressABliveryfalse\</ns0:isExpressABlivery\>ns0:isCustomerRFQtrue\</ns0:isCustomerRFQ\>ns0:customerGroupRES\</ns0:customerGroup\>ns0:customerGroupA\</ns0:customerGroup\>ns0:customerGroupNL72\</ns0:customerGroup\>ns0:accountMarketRES\</ns0:accountMarket\>ns0:personalizedSegmentCategoriesA\</ns0:personalizedSegmentCategories\>\</ns0:customerInfo\>ns0:easypanelurl[https://www.abc-abc.AB](https://www.abc-abc.AB)\</ns0:easypanelurl\>\</ns0:customerDetails\>", "timestamp":"2021-11-12T04:58:59.254+01:00"}

---

<div class="post-metadata">

**Author:** ![nandatibco](https://avatars.discourse-cdn.com/v4/letter/n/ed8c4c/32.png) [@nandatibco](https://discuss.elastic.co/u/nandatibco)\
**Post date:** [November 20, 2021, 3:30pm UTC](https://discuss.elastic.co/t/filebeat-not-pushing-few-lines-to-logstash/289011/4 "2021-11-20T15:30:05Z")

</div>

Identified the cause for this issue, the _"payloadID":"22810.ABC.XYZ.16366895388792810",_ is common in all the lines, Filebeat was treating them as duplicate entries and skipping.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 18, 2021, 5:30pm UTC](https://discuss.elastic.co/t/filebeat-not-pushing-few-lines-to-logstash/289011/5 "2021-12-18T17:30:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.