Filebeat stops sending json logs midway. Version: 6.0.0-rc1

Elastic Stack Beats
1

I have some test logs in json format. The number of entries are 900+ in the file I'm testing. After a fresh installation, Filebeat starts sending the logs but after some time (a few minute) starts giving this err:

ERR Failed to publish events: temporary bulk send failure

Enabling debug logs, shows this :

DBG Bulk item insert failed (i=13, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}

The final count of documents in the index created in elasticsearch is 631, less than the 900+ in log file (Untitled-2.json in the logs attached). It has sometimes stopped as early as 100s.

The debug logs and the filebeat configuration are attached in this gist:

https://gist.github.com/rohit-smpx/79669da791f8ec76f93043ae9fb505d4

2

Thanks for reporting. I wonder if the error happens when processing event from the syslog module or due to the json prospector. Either way, it might be a bug.

Can you test with only the syslog and only the json prospector being enabled? If one or the other fails can you try to reduce the log files, until we find a few events causing the issue?

3

Also check the Elasticsearch logs. The full exception including a stack-trace should be available in the logs.

4

I updated the gist to include the elasticsearch logs. I didn't see any ERR in there. I did check with the json prospector disabled, and there were no errors. I'll reduce the events and try again.

5

Disabling syslog fixed it for now. I'll add some debug logs tomorrow.

6

This is fixed. In the filebeat conf I had set up the index name as

index: "filebeat.%{[table]}.%{+yyyy.MM}"

Where 'table' was a field in the json logs. But the syslog logs didn't have those, so I think the error was because of that. I changed it to this:

 indices:
    - index: "filebeat.%{[table]}.%{+yyyy.MM}"
      when.regexp:
        table: ".*"
  
  index: "filebeat.syslog.%{+yyyy.MM.dd}"

Now, the default index is syslog and if the log has a table field it will go to it's respective index.

7

Oh, seems like the index was not correctly set due to this. You can also use 'defaults' like this:

index: 'filebeat.%{[table]:syslog}.%{+yyyy.MM}'
8

Thanks, will do :slight_smile:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.