How to handle the log partialy updated

Elastic Stack Logstash
1

Hi there,

WebSphere Application Server has ffdc exception log, and it looks like below, when the ffdc event happened, it search the log first, if it find same exception, it updates count only, otherwise append the log to the end.

Now, my problem is:
Everytime, there is a new line appended, Filebeat resends all lines, to logstash, and in pie chart, I sum the the counts by exception, and the sum become extremely high eventually.

My question: is Logstash/Filebeat capable to process this kind of logs?

Index Count Time of first Occurrence Time of last Occurrence Exception SourceId ProbeId
------+------+---------------------------+---------------------------+---------------------------
0 44 9/7/17 9:17:59:696 EDT 9/7/17 9:18:04:130 EDT com.ibm.CSIv2Security.CSIv2RequirementsNotSatisfied com.ibm.ISecurityLocalObjectCSIv2UtilityImpl.CSIv2EffectivePerformPolicy.getInstance 2106 /devZone/logs/wsqa/wasJVM/ffdc/server1_591f46d8_17.09.07_09.17.59.7334786773500846563884.txt
1 11 9/7/17 9:17:59:827 EDT 9/7/17 9:18:04:132 EDT java.lang.reflect.InvocationTargetException com.ibm.ejs.oa.LocationService.runAsSystem 465 /devZone/logs/wsqa/wasJVM/ffdc/server1_591f46d8_17.09.07_09.17.59.8308500032144821282524.txt
2 195 9/7/17 9:23:33:886 EDT 9/7/17 14:43:35:167 EDT javax.transaction.RollbackException com.ibm.tx.jta.impl.TransactionImpl.prepareResources 1505 /devZone/logs/wsqa/wasJVM/ffdc/server1_17f46a58_17.09.07_09.23.33.8914122574254552181999.txt
------+------+---------------------------+---------------------------+---------------------------

Thanks and Regards,

Fei

2

Everytime, there is a new line appended, Filebeat resends all lines, to logstash

Exactly how are lines appended? Are they literally appended or is the old and new data written to a new file that's renamed to replace the old file? That's very atypical for log files but would match the behavior you describe.

3

Thanks, Magnus

Good question! I don't have the details how the file updated.

But if you have something I can test, I would like to figure it out.

Regards,

Fei

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.