Multiline Filebeat suppresses lines

Elastic Stack Beats
1

Hey guys,

I have a problem in the filbeat multline that is not concatenating correctly with the defined pattern. When I look at the result using it through the output.console I see that it suppresses part of the line that should be concatenated. Here's the XML, my confguraton and the log result.

My XML

<?xml version="1.0" encoding="utf-8"?><proc schema="procNFe_v4.00.xsd" NSU="818802256" endIP="177.73.245.101" Caminho="\\XXXXXXXXXX\99999999.xml" Tipo="NFe" 
...
...
...
e1IdgsBMuj9EGPfqJ28T6ltpWKRptbdzFG1CzvsNsR2l+wMhKjMYG8hF9ReuQtJ+Oh2YGPQBWvyA
AV0Z0Kz2hSg41ZJb04QEER0gIyAD13J8b63ep0JEYlp3iZfrtNJ5ylYLem4kOlCe2LeHcPXyXQVp
zzDhaQhjHH1PjBqlSV5KZK0KrLGPLQcvLdGHudfavbx0HDvahbiTDttZiopLZ6mJvRibUTYHv5qI
KWYo5vqxEmHGpdNydg==</X509Certificate></X509Data></KeyInfo></Signature></NFe><protNFe versao="4.00" xmlns="http://www.portalfiscal.inf.br/nfe"><infProt><tpAmb>1</tpAmb><verAplic>SVRSnfce201908091113</verAplic><chNFe>32200100304588000177650020000543199790045832</chNFe><dhRecbto>2020-01-06T08:53:01-03:00</dhRecbto><nProt>332200006701120</nProt><digVal>Y8E1tdQCYI7SO88jzN/1uNAJ8kQ=</digVal><cStat>100</cStat><xMotivo>Autorizado o uso da NF-e</xMotivo></infProt></protNFe></nfeProc></proc>

My Config

- type: log
  enabled: true
  close_eof: false
  paths:
    - /opt/nfe/06/81881/XXXXX.xml

  multiline.type: pattern
  multiline.pattern: '^<\?xml *'
  multiline.negate: true
  multiline.match: after
  multiline.flush_pattern: '<\/proc>'
  max_bytes: 10024

Log Result:

2021-11-18T11:33:45.032-0300    INFO    [input.harvester]       log/harvester.go:309    Harvester started for file.     {"input_id": "d502529c-8a47-4ba9-b853-ba1101a81a3b", "source": "/opt/nfe/06/81881/000000818802256.xml", "state_id": "native::1373438-2049", "finished": false, "os_id": "1373438-2049", "harvester_id": "a84dee6c-09af-4e90-bfe6-6f03f2327b81"}
{
  "@timestamp": "2021-11-18T14:33:45.032Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.15.2"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/opt/nfe/06/81881/000000818802256.xml"
    },
    "flags": [
      "truncated",
      "multiline"
    ]
  },
  "message": "<?xml version=\"1.0\" encoding=\"utf-8\"?><proc schema=\"procNFe_v4.00.xsd\" NSU=\"818802256\" endIP=\"177.73.245.101\" Caminho=\"\\\\s3a11120p3\\ARQUIVOS\\NFCE_XML\\Producao\\NFe\\2020\\01\\06\\81881\\000000818802256.xml\" Tipo=\"NFe\" xmlns=\"http://www.portalfiscal.inf.br/nfe\"><nfeProc xmlns=\"http://www.portalfiscal.inf.br/nfe\" versao=\"4.00\"><NFe xmlns=\"http://www.portalfiscal.inf.br/nfe\"><infNFe Id=\"NFe32200100304588000177650020000543199790045832\" versao=\"4.00\"><ide><cUF>32</cUF><cNF>79004583</cNF><natOp>Venda de mercadoria, adquirida ou recebida de terceiros, su</natOp><mod>65</mod><serie>2</serie><nNF>54319</nNF><dhEmi>2020-01-06T08:49:11-02:00</dhEmi><tpNF>1</tpNF><idDest>1</idDest><cMunFG>3202801</cMunFG><tpImp>4</tpImp><tpEmis>9</tpEmis><cDV>2</cDV><tpAmb>1</tpAmb><finNFe>1</finNFe><indFinal>1</indFinal><indPres>1</indPres><procEmi>0</procEmi><verProc>7.5.4</verProc><dhCont>2020-01-06T08:54:11-02:00</dhCont><xJust>Sem comunicacao com o webservice</xJust></ide><emit><CNPJ>00304588000177</CNPJ><xNome>MERKATO DISTRIBUIDORA LTDA ME</xNome><xFant>MERKATO DISTRIBUIDORA LTDA</xFant><enderEmit><xLgr>AV ITAPEMIRIM</xLgr><nro>2728</nro><xBairro>PRAIA DE ITAOCA</xBairro><cMun>3202801</cMun><xMun>ITAPEMIRIM</xMun><UF>ES</UF><CEP>29338000</CEP><cPais>1058</cPais><xPais>BRASIL</xPais><fone>2835292328</fone></enderEmit><IE>081712677</IE><CRT>3</CRT></emit><det nItem=\"1\"><prod><cProd>4921</cProd><cEAN>SEM GTIN</cEAN><xProd>LIMAO KG</xProd><NCM>96161000</NCM><CFOP>5102</CFOP><uCom>KG</uCom><qCom>0.9550</qCom><vUnCom>3.98</vUnCom><vProd>3.80</vProd><cEANTrib>SEM GTIN</cEANTrib><uTrib>KG</uTrib><qTrib>0.9550</qTrib><vUnTrib>3.98</vUnTrib><indTot>1</indTot></prod><imposto><vTotTrib>0.99</vTotTrib><ICMS><ICMS40><orig>0</orig><CST>40</CST></ICMS40></ICMS><PIS><PISNT><CST>06</CST></PISNT></PIS><COFINS><COFINSNT><CST>06</CST></COFINSNT></COFINS></imposto></det><det nItem=\"2\"><prod><cProd>2925</cProd><cEAN>SEM GTIN</cEAN><xProd>BATATA KG</xProd><NCM>07019000</NCM><CFOP>5102</CFOP><uCom>KG</uCom><qCom>0.7150</qCom><vUnCom>4.90</vUnCom><vProd>3.50</vProd><cEANTrib>SEM GTIN</cEANTrib><uTrib>KG</uTrib><qTrib>0.7150</qTrib><vUnTrib>4.90</vUnTrib><indTot>1</indTot></prod><imposto><vTotTrib>0.75</vTotTrib><ICMS><ICMS40><orig>0</orig><CST>40</CST></ICMS40></ICMS><PIS><PISNT><CST>06</CST></PISNT></PIS><COFINS><COFINSNT><CST>06</CST></COFINSNT></COFINS></imposto></det><det nItem=\"3\"><prod><cProd>250</cProd><cEAN>7896036090244</cEAN><xProd>OLEO DE SOJA LIZA 900ML</xProd><NCM>15079011</NCM><CEST>1706500</CEST><CFOP>5405</CFOP><uCom>UN</uCom><qCom>3</qCom><vUnCom>4.75</vUnCom><vProd>14.25</vProd><cEANTrib>7896036090244</cEANTrib><uTrib>UN</uTrib><qTrib>3</qTrib><vUnTrib>4.75</vUnTrib><indTot>1</indTot></prod><imposto><vTotTrib>1.60</vTotTrib><ICMS><ICMS60><orig>0</orig><CST>60</CST></ICMS60></ICMS><PIS><PISNT><CST>06</CST></PISNT></PIS><COFINS><COFINSNT><CST>06</CST></COFINSNT></COFINS></imposto></det><det nItem=\"4\"><prod><cProd>3235</cProd><cEAN>7896029310014</cEAN><xProd>SAL REFINADO GLOBO 1KG</xProd><NCM>25010020</NCM><CFOP>5102</CFOP><uCom>UN</uCom><qCom>1</qCom><vUnCom>1.70</vUnCom><vProd>1.70</vProd><cEANTrib>7896029310014</cEANTrib><uTrib>UN</uTrib><qTrib>1</qTrib><vUnTrib>1.70</vUnTrib><indTot>1</indTot></prod><imposto><vTotTrib>0.19</vTotTrib><ICMS><ICMS20><orig>0</orig><CST>20</CST><modBC>3</modBC><pRedBC>58.8200</pRedBC><vBC>0.70</vBC><pICMS>17.00</pICMS><vICMS>0.12</vICMS></ICMS20></ICMS><PIS><PISAliq><CST>01</CST><vBC>1.70</vBC><pPIS>1.65</pPIS><vPIS>0.03</vPIS></PISAliq></PIS><COFINS><COFINSAliq><CST>01</CST><vBC>1.70</vBC><pCOFINS>7.60</pCOFINS><vCOFINS>0.13</vCOFINS></COFINSAliq></COFINS></imposto></det><det nItem=\"5\"><prod><cProd>1862</cProd><cEAN>7896110061221</cEAN><xProd>PAPEL TOALHA SNOB 2 ROLOS DE 60 TOALHAS</xProd><NCM>48189090</NCM><CEST>2004700</CEST><CFOP>5102</CFOP><uCom>UN</uCom><qCom>1</qCom><vUnCom>4.20</vUnCom><vProd>4.20</vProd><cEANTrib>7896110061221</cEANTrib><uTrib>UN</uTrib><qTrib>1</qTrib><vUnTrib>4.20</vUnTrib><indTot>1</indTot></prod><imposto><vTotTrib>1.12</vTotTrib><ICMS><ICMS00><orig>0</orig><CST>00</CST><modBC>3</modBC><vBC>4.20</vBC><pICMS>17.00</pICMS><vICMS>0.71</vICMS></ICMS00></ICMS><PIS><PISAliq><CST>01</CST><vBC>4.20</vBC><pPIS>1.65</pPIS><vPIS>0.07</vPIS></PISAliq></PIS><COFINS><COFINSAliq><CST>01</CST><vBC>4.20</vBC><pCOFINS>7.60</pCOFINS><vCOFINS>0.32</vCOFINS></COFINSAliq></COFINS></imposto></det><det nItem=\"6\"><prod><cProd>2935</cProd><cEAN>7896339810464</cEAN><xProd>PAPEL HIGIENICO CARINH0 PLUS 60M</xProd><NCM>48181000</NCM><CEST>2004200</CEST><CFOP>5102</CFOP><uCom>UN</uCom><qCom>1</qCom><vUnCom>4.20</vUnCom><vProd>4.20</vProd><cEANTrib>7896339810464</cEANTrib><uTrib>UN</uTrib><qTrib>1</qTrib><vUnTrib>4.20</vUnTrib><indTot>1</indTot></prod><imposto><vTotTrib>1.05</vTotTrib><ICMS><ICMS00><orig>0</orig><CST>00</CST><modBC>3</modBC><vBC>4.20</vBC><pICMS>17.00</pICMS><vICMS>0.71</vICMS></ICMS00></ICMS><PIS><PISNT><CST>06</CST></PISNT></PIS><COFINS><COFINSNT><CST>06</CST></COFINSNT></COFINS></imposto></det><total><ICMSTot><vBC>9.10</vBC><vICMS>1.54</vICMS><vICMSDeson>0.00</vICMSDeson><vFCP>0.00</vFCP><vBCST>0.00</vBCST><vST>0.00</vST><vFCPST>0.00</vFCPST><vFCPSTRet>0.00</vFCPSTRet><vProd>31.65</vProd><vFrete>0.00</vFrete><vSeg>0.00</vSeg><vDesc>0.00</vDesc><vII>0.00</vII><vIPI>0.00</vIPI><vIPIDevol>0.00</vIPIDevol><vPIS>0.10</vPIS><vCOFINS>0.45</vCOFINS><vOutro>0.00</vOutro><vNF>31.65</vNF><vTotTrib>5.70</vTotTrib></ICMSTot></total><transp><modFrete>9</modFrete></transp><pag><detPag><tPag>04</tPag><vPag>31.65</vPag><card><tpIntegra>1</tpIntegra><CNPJ>01425787000104</CNPJ><tBand>99</tBand><cAut>500022-250363896</cAut></card></detPag></pag><infAdic><infCpl>MD-5: ;Trib aprox R$: 1,91 Fed e 3,79 Est;Fonte: IBPT/empresometro.com.br ES      5A16F8;Operador: 8 Vendedor: -;Obrigado - Volte sempre!</infCpl></infAdic><infRespTec><CNPJ>01212344000127</CNPJ><xContato>CLAYTON GRAF</xContato><email>contato@intelidata.inf.br</email><fone>4732511750</fone></infRespTec></infNFe><infNFeSupl><qrCode><![CDATA[http://app.sefaz.es.gov.br/ConsultaNFCe/qrcode.aspx?p=32200100304588000177650020000543199790045832|2|1|06|31.65|5938453174645143594937534f38386a7a4e2f31754e414a386b513d|1|856E1F6642920D84D1166588C48C7D580AE79E24]]></qrCode><urlChave>www.sefaz.es.gov.br/nfce/consulta</urlChave></infNFeSupl><Signature xmlns=\"http://www.w3.org/2000/09/xmldsig#\"><SignedInfo><CanonicalizationMethod Algorithm=\"http://www.w3.org/TR/2001/REC-xml-c14n-20010315\" /><SignatureMethod Algorithm=\"http://www.w3.org/2000/09/xmldsig#rsa-sha1\" /><Reference URI=\"#NFe32200100304588000177650020000543199790045832\"><Transforms><Transform Algorithm=\"http://www.w3.org/2000/09/xmldsig#enveloped-signature\" /><Transform Algorithm=\"http://www.w3.org/TR/2001/REC-xml-c14n-20010315\" /></Transforms><DigestMethod Algorithm=\"http://www.w3.org/2000/09/xmldsig#sha1\" /><DigestValue>Y8E1tdQCYI7SO88jzN/1uNAJ8kQ=</DigestValue></Reference></SignedInfo><SignatureValue>fPaD2TMNRcFXz3UBnv8/HePE3r4bEkQXdnoPruGpyz6tJVg4G2kBf3dxbBorNMV4vVGFWTzo8Qvm\n2DR4xg6gO877EZNRmwpf6xULhXDh0M+5DLlq/FdLGV1io0bknlY19mS/rJ3K5GCap/WRetLo5NEB\n4SJyIFuKzcovJVR1udI2QkyaJJnRsi82pA4A3QYKAljF6iJy43tE+7pB3PDCsMWAyJ6Wk+tuDGa+\n5wriNbb7WMspTUHz1LaQTjnplGgVox0GvZptNO5Os/znLzYapuNr4fP3D4kI94QYUlz8LPxj70l3\nK0zvJaUVgTb1ridSGAjLXdSRdt9FO/xowgUH5A==</SignatureValue><KeyInfo><X509Data><X509Certificate>MIIHYjCCBUqgAwIBAgIRALtuXsKzwCA3t6b9rPJ+wqowDQYJKoZIhvcNAQELBQAwgYkxCzAJBgNV\nBAYTAkJSMRMwEQYDVQQKDApJQ1AtQnJhc2lsMTYwNAYDVQQLDC1TZWNyZXRhcmlhIGRhIFJlY2Vp\ndGEgRmVkZXJhbCBkbyBCcmFzaWwgLSBSRkIxLTArBgNVBAMMJEF1dG9yaWRhZGUgQ2VydGlmaWNh\nZG9yYSBTRVJQUk9SRkJ2NTAeFw0xOTExMDYxMTIxMDFaFw0yMDExMDUxMTIxMDFaMIH2MQswCQYD\nVQQGEwJCUjELMAkGA1UECAwCRVMxEzARBgNVBAcMCklUQVBFTUlSSU0xEzARBgNVBAoMCklDUC1C\ncmFzaWwxFzAVBgNVBAsMDjM0MDI4MzE2MDAwMTAzMTYwNAYDVQQLDC1TZWNyZXRhcmlhIGRhIFJl\nY2VpdGEgRmVkZXJhbCBkbyBCcmFzaWwgLSBSRkIxEzARBgNVBAsMCkFSQ09SUkVJT1MxFjAUBgNV\nBAsMDVJGQiBlLUNOUEogQTExMjAwBgNVBAMMKU1FUktBVE8gRElTVFJJQlVJRE9SQSBMVERBOjAw\nMzA0NTg4MDAwMTc3MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAkBHDTJY3CGa+7iAO\nrMA0vQwg7DIc/nIJ6SD9udYGnke3z0y/tlgY9uAMPZTJpyiWqqwAmxoarojlN20l+jfgm6tERw6t\n7UcNJPIMxvSHgrI3z67+eQxFlfHf6w+f6o9HVnWQMYHk82WHID/R9cTt1Bnyd94c0SKNwKjGUkc3\noTfs7a9QrcA5chYWmhjmSFE5V/dySHvey2wX3UxHfE3gAf4BcO2wTj/xgnKb5nK7vWM1hrEz6h70\nUJ89yititRabfnFBAfQ2wgR4TvsRbQd2QFztfRTEPX3bqzbMvewc4NA6ZnQBpwPOjz6PHOd7V+8L\nNvP7KFX3swZcFDckS26/YQIDAQABo4ICVDCCAlAwHwYDVR0jBBgwFoAUFIAtnX6aRcDxWz8Z1UCw\nby9l4OkwWwYDVR0gBFQwUjBQBgZgTAECAQowRjBEBggrBgEFBQcCARY4aHR0cDovL3JlcG9zaXRv\ncmlvLnNlcnByby5nb3YuYnIvZG9jcy9kcGNhY3NlcnByb3JmYi5wZGYwgYgGA1UdHwSBgDB+MDyg\nOqA4hjZodHRwOi8vcmVwb3NpdG9yaW8uc2VycHJvLmdvdi5ici9sY3IvYWNzZXJwcm9yZmJ2NS5j\ncmwwPqA8oDqGOGh0dHA6Ly9jZXJ0aWZpY2Fkb3MyLnNlcnByby5nb3YuYnIvbGNyL2Fjc2VycHJv\ncmZidjUuY3JsMFYGCCsGAQUFBwEBBEowSDBGBggrBgEFBQcwAoY6aHR0cDovL3JlcG9zaXRvcmlv\nLnNlcnByby5nb3YuYnIvY2FkZWlhcy9hY3NlcnByb3JmYnY1LnA3YjCBvQYDVR0RBIG1MIGyoDgG\nBWBMAQMEoC8ELTA1MTIxOTc1MDAxMjI0ODU2ODgxOTAyMDM3OTU5NjAwMDAwMDAwMDAwMDAwMKAt\nBgVgTAEDAqAkBCJST1NJTEFJTkUgRVVTVEFRVUlBIERBIFNJTFZBIEdPTUVToBkGBWBMAQMDoBAE\nDjAwMzA0NTg4MDAwMTc3oBcGBWBMAQMHoA4EDDAwMDAwMDAwMDAwMIETZWRnYXJAdG90YWxjb250\nLm5ldDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMA0GCSqG\nSIb3DQEBCwUAA4ICAQCD0IXJl5ahvmfiSSrLHgA6CribQk210X3B9bNAwn2119VKYJ+ej757cgOd\nhnlKN2D6RmGeebOxqw92s3EENZZzka84sZQ98FmUGYhKctMlQc/qbJ4Q1ZaoPjQOobrCG7tqgyis\nMtfyHjAXTRNKROLD0rR+LE9Z8WNclX9lMXYfoyTZ2XdBQxk6GjxUmdE/zIg5pEkDP3Pv0UvbhNVy\nNraFKJp9g6GuyeTAghti6pqr7n81VRZNsYcSsEh5d9TmBkvLdT/S4QlNu/oUKt7kxETrER6k8/1o\nnwdgNmHvvzWxjEUDVQIb8WxDNA6eEC5m8WydEuASkGu32nw5UPKHIE6/loq4hxrzpqJEkGz2dzJ/\nJyUlaS0dOCPFEFypXYTJ44cFvL1WpHpvG02HKd+2ziw532Dginj7OrJeqbh55Ay1oypOY6EO1f86\ne1IdgsBMuj9EGPfqJ28T6ltpWKRptbdzFG1CzvsNsR2l+wMhKjMYG8hF9ReuQtJ+Oh2YGPQBWvyA\nAV0Z0Kz2hSg41ZJb04QEER0gIyAD13J8b63ep0JEYlp3iZfrtNJ5ylYLem4kOlCe2LeHcPXyXQVp\nzzDhaQhjHH1PjBqlSV5KZK0KrLGPLQcvLdGHudfavbx0HDvahbiTDttZiopLZ6mJvRibUTYHv5qI\nKWYo5vqxEmHGpdNydg==</X509Certificate></X509Data></KeyInfo></Signature></NFe><protNFe versao=\"4.00\" xmlns=**\"http://www.portalfiscal.inf.br/nfe\"><infProt><tpAmb>1",**
  "input": {
    "type": "log"
  },
  "ecs": {
    "version": "1.11.0"
  },
  "host": {
    "name": "S3A11214H1"
  },
  "agent": {
    "hostname": "S3A11214H1",
    "ephemeral_id": "b2755732-a573-4967-b3be-b1c4e0b49672",
    "id": "132370bb-b6a5-48ba-97c4-156412ae21ff",
    "name": "S3A11214H1",
    "type": "filebeat",
    "version": "7.15.2"
  }
}

Note that at the end of the message field it suppresses the rest of the XML.

If anyone has any solution, I appreciate it!

Att,

Rodrigo Tornis

2

Could you please share debug logs?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.