FIleBeat not pushing few lines to Logstash

Elastic Stack Beats
1

File beat is not pushing all lines in a file to Logstash, and it is not consistent, it skips 3 lines in between. Im new to File Beat, please suggest if I missed any configuration.
For example, my log file looks like this:
Line1
Line2
Line3
Line4
Some time I see only line1 is pushed, next time I see only line3 is pushed, any time it is not pushing all lines to Logstash.
Randomly pushes only 1 line always.

2

Is your single message over multiple lines? Or is every line its own message?

If its the first case, you need to add multiline configuration to your log input in filebeat.yml.

Do you have 2-3 example lines?

3

Hi Marius,
Every line its own message. At the end of every line there is a carriage return(empty line). File contains json segments as below:
Example:
{ "payloadID":"22810.ABC.XYZ.16366895388792810", "correlationID":"ABC.XYZ.16366895388792810", "country":"AB", "serviceName":"XYZ", "operationName":"getcustomerDetails", "type":"audit", "servicePayload":"<?xml version=\"1.0\" encoding=\"UTF-8\"?><ns0:getcustomerDetails xmlns:ns0="http://www.abc.com/ErpSchema">ns0:customerid40008</ns0:customerid></ns0:getcustomerDetails>", "timestamp":"2021-11-12T04:58:58.879+01:00"}

{ "payloadID":"22810.ABC.XYZ.16366895388792810", "correlationID":"ABC.XYZ.16366895388792810", "country":"AB", "serviceName":"XYZ", "operationName":"getcustomerDetails", "type":"audit", "servicePayload":"<?xml version=\"1.0\" encoding=\"UTF-8\"?><v1:customerDetailsRequest xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v1="http://www.abc.com/myshop/mediation/normalizedMoABl/v1" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><v1:webshop banner="GY" country="GY" division="?" webshopURL="?">v1:webshopNameXYZ</v1:webshopName></v1:webshop>v1:ERPCustomerIDv1:accountNumber40008</v1:accountNumber>v1:project?</v1:project>v1:language?</v1:language></v1:ERPCustomerID>v1:checkString?</v1:checkString></v1:customerDetailsRequest>", "timestamp":"2021-11-12T04:58:58.879+01:00"}

{ "payloadID":"22810.ABC.XYZ.16366895388792810", "correlationID":"ABC.XYZ.16366895388792810", "country":"AB", "serviceName":"XYZ", "operationName":"getcustomerDetails", "type":"audit", "servicePayload":"<?xml version=\"1.0\" encoding=\"UTF-8\"?><getcustomerDetailsResponse xmlns="http://www.abc.com/ErpSchema" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">40008aaa bbb cccaaa bbb ccc72aaa bbb cccaaa bbb cccRIEDSTR. 4284082ABCDEFGHABfalse94366 abcd efgn ancd efghABfalsetruefalseOptionalhttps://www.abcd-easypanel.ABRESResiABntialAaaa bbb ccc)truetruefalse", "timestamp":"2021-11-12T04:58:59.254+01:00"}

{ "payloadID":"22810.ABC.XYZ.16366895388792810", "correlationID":"ABC.XYZ.16366895388792810", "country":"AB", "serviceName":"XYZ", "operationName":"getcustomerDetails", "type":"audit", "servicePayload":"<?xml version=\"1.0\" encoding=\"UTF-8\"?><ns0:customerDetails xmlns:ns0="http://www.abc.com/myshop/mediation/normalizedMoABl/v1">ns0:ERPCustomerIDns0:accountNumber40008</ns0:accountNumber>ns0:languageAB</ns0:language></ns0:ERPCustomerID>ns0:companyNameaaa bbb ccc</ns0:companyName>ns0:managingBranch72</ns0:managingBranch>ns0:billingABtailsns0:billingAddressns0:companyNameaaa bbb ccc</ns0:companyName>ns0:address1RIEDSTR. 42</ns0:address1>ns0:address2aaa bbb ccc</ns0:address2>ns0:postalCoAB84082</ns0:postalCoAB>ns0:cityABCDEFGH</ns0:city>ns0:countryAB</ns0:country></ns0:billingAddress></ns0:billingABtails>ns0:ABliveryABtailsns0:ABliveryAddressns0:addressID0</ns0:addressID>ns0:companyNameaaa bbb ccc</ns0:companyName>ns0:address1RIEDSTR. 42</ns0:address1>ns0:address2aaa bbb ccc</ns0:address2>ns0:address3/ns0:postalCoAB84082</ns0:postalCoAB>ns0:cityABCDEFGH</ns0:city>ns0:countryAB</ns0:country>ns0:provinceCoAB/ns0:ABfaultABliveryAddressIDtrue</ns0:ABfaultABliveryAddressID></ns0:ABliveryAddress></ns0:ABliveryABtails>ns0:activeCustomerfalse</ns0:activeCustomer>ns0:customerInfons0:custPurchaseOrdNoOptional</ns0:custPurchaseOrdNo>ns0:salesRepID94_366</ns0:salesRepID>ns0:canReturnOrABrns0:accessRighttrue</ns0:accessRight></ns0:canReturnOrABr>ns0:isExpressABliveryfalse</ns0:isExpressABlivery>ns0:isCustomerRFQtrue</ns0:isCustomerRFQ>ns0:customerGroupRES</ns0:customerGroup>ns0:customerGroupA</ns0:customerGroup>ns0:customerGroupNL72</ns0:customerGroup>ns0:accountMarketRES</ns0:accountMarket>ns0:personalizedSegmentCategoriesA</ns0:personalizedSegmentCategories></ns0:customerInfo>ns0:easypanelurlhttps://www.abc-abc.AB</ns0:easypanelurl></ns0:customerDetails>", "timestamp":"2021-11-12T04:58:59.254+01:00"}

4

Identified the cause for this issue, the "payloadID":"22810.ABC.XYZ.16366895388792810", is common in all the lines, Filebeat was treating them as duplicate entries and skipping.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.