Value too large to output

Elastic Stack Beats
1

Hi,

FileBeat is outputting to logstash text like: Value too large to output (2939 bytes)! First 255 chars are: ...

  1. Yes the logline is actually 2939 bytes
  2. Yes I do want it sent to logstash

I have looked at the max_bytes option in FileBeat but it does not have an affect.

Is there a way to configure FileBeat to either allow 10000 bytes or to break it up in packets?

thanks

Phill

2

do you have a sample log plus filebeat debug log? I don't see why a ~2k event should be too large.

3

here is an example.
Input into Filebeat

2 2016-12-05 00:01:10.103 default-workqueue-1: Host Response
Inbound Message
----------------------------
ID: 3975975
Response-Code: 200
Encoding: UTF-8
Content-Type: text/xml; charset=utf-8
Headers: {Cache-Control=[private, max-age=0], Connection=[close], Content-Language=[en-ZA], Content-Length=[1870], Content-Type=[text/xml; charset=utf-8], Date=[Sun, 04 Dec 2016 22:01:09 GMT], Server=[Microsoft-IIS/7.5], X-AspNet-Version=[4.0.30319], X-Powered-By=[ASP.NET]}
Payload: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">&#xd;
  <soap:Body>&#xd;
    <r0:creditVendResp xmlns:b0="http://www.nrs.eskom.co.za/xmlvend/base/2.1/schema" xmlns:r0="http://www.nrs.eskom.co.za/xmlvend/revenue/2.1/schema">&#xd;
      <b0:clientID xsi:type="b0:EANDeviceID" ean="**************"/>&#xd;
      <b0:serverID xsi:type="b0:EANDeviceID" ean="**************"/>&#xd;
      <b0:terminalID xsi:type="b0:GenericDeviceID" id="1"/>&#xd;
      <b0:reqMsgID dateTime="20161205001309" uniqueNumber="59602"/>&#xd;
      <b0:respDateTime>2016-12-05T00:01:09.7574997+02:00</b0:respDateTime>&#xd;
      <b0:dispHeader>CREDIT VEND - TAX INVOICE</b0:dispHeader>&#xd;
      <b0:operatorMsg>060000 Warning: This meter is not configured for FBE.</b0:operatorMsg>&#xd;
      <b0:custMsg>Meter not registered for Free Basic Electricity. Please apply at your local office.</b0:custMsg>&#xd;
      <b0:clientStatus>&#xd;
        <b0:availCredit value="15345748.71" symbol="R"/>&#xd;
      </b0:clientStatus>&#xd;
      <b0:utility name="Eskom Online" address="Megawatt Park, Contact Centre tel 086-003-****" taxRef="4740101508"/>&#xd;
      <r0:creditVendReceipt receiptNo="**************">&#xd;
        <r0:transactions>&#xd;
          <r0:tx xsi:type="r0:CreditVendTx">&#xd;
            <r0:amt value="50" symbol="R"/>&#xd;
            <r0:creditTokenIssue xsi:type="b0:SaleCredTokenIssue">&#xd;
              <b0:desc>Normal Sale</b0:desc>&#xd;
              <b0:meterDetail msno="***********" sgc="100600" krn="1" ti="07">&#xd;
                <b0:meterType at="07" tt="02"/>&#xd;
              </b0:meterDetail>&#xd;
              <b0:token xsi:type="b0:STS1Token">&#xd;
                <b0:stsCipher>70476449248744181379</b0:stsCipher>&#xd;
              </b0:token>&#xd;
              <b0:units value="48" siUnit="kWh"/>&#xd;
              <b0:resource xsi:type="b0:Electricity"/>&#xd;
            </r0:creditTokenIssue>&#xd;
            <r0:tariff>&#xd;
              <b0:name>SG=100600 TI=07</b0:name>&#xd;
              <b0:desc>48 kWh @ 104.26 c/kWh</b0:desc>&#xd;
            </r0:tariff>&#xd;
          </r0:tx>&#xd;
          <r0:tenderAmt value="50" symbol="R"/>&#xd;
        </r0:transactions>&#xd;
      </r0:creditVendReceipt>&#xd;
    </r0:creditVendResp>&#xd;
  </soap:Body>&#xd;
</soap:Envelope>

--------------------------------------

In Logstash the first Grok is:

^[0-9]?\s%{PESTIMESTAMP:logtimestamp}\s%{THREAD_NAME8:threadName}:\s%{GREEDYDATA:restOfTheMessage}

Then the Warning below happens on the next Grok

Payload:\s%{XML_PAYLOAD:restOfTheMessage}[-]{28}$

where

MILLISECOND [0-9]{3}
PESTIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY}\s%{HOUR}:%{MINUTE}:%{SECOND}\.%{MILLISECOND}
THREAD_NAME8 default-workqueue-[0-9]{1,4}
XML_PAYLOAD <(.*[\n|\r])*

Filebeat has some log lines that might indicated why this is happening:

2017-01-19T13:54:48+02:00 ERR Failed to publish events caused by: read tcp [::1]:65251->[::1]:5044: i/o timeout
2017-01-19T13:54:48+02:00 INFO Error publishing events (retrying): read tcp [::1]:65251->[::1]:5044: i/o timeout
2017-01-19T13:54:48+02:00 DBG  close connection
2017-01-19T13:54:48+02:00 DBG  send fail

But I can see why the 2nd grok fails with the error and not the 1st.
There does not seem to be an issue with the Groks because they are fine for all other log lines and don't fails with these examples when tested in http://grokconstructor.appspot.com/

Result

Timeout executing grok 'Payload:\s%{XML_PAYLOAD:restOfTheMessage}[-]{28}$' against field 'message' with value 'Value too large to output (2939 bytes)! First 255 chars are: 2 2016-12-05 00:01:10.103 default-workqueue-1: Host Response
Inbound Message

ID: 3975975
Response-Code: 200
Encoding: UTF-8
Content-Type: text/xml; charset=utf-8
Headers: {Cache-Control=[private, max-age=0], Connection=[close],'!

4

The error message is i/o timeout. The default i/o timeout is 30 seconds. If writing stalls within these 30 seconds (because sending/receiving OS/network making no progress on TCP stream), the connection might timeout. In this case filebeat normally tries to reconnect and sends the events again (potentially with smaller window size in the hope of working around the problem by decreasing amount of data to be send/processed). Have you tried to increase the timeout.

5

I think the "Value too large to output" is coming from logstash and is applied at logging time, so it's not the source of the issue. The source of the issue I think it's the "Timeout executing grok". See if the comments from this ticket are helping, if not, I recommend asking in the Logstash forum.

6

hmmm, I am using LogStash 5.1 with grok 3.3.0

and with a timeout of 180 seconds

and still no luck

I will continue on a Logstash thread

thanks

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.