How to deal with duplicate fields in a message

Elastic Stack Logstash
1

Hi Guys, for swap transactions there are usually 2 legs. Unfortunately the application spews everything out and doesn't give the legs any distinct name so I end up with the value for both legs of the transaction duplicated. here's an example message:

[ requestId=R20170704l, requestType=Credit Check Request, timeStamp=2017-07-04T12:50:59.892Z, deals { Deal [ dealId=R20170704l, counterPartyId=12345, memberBank=YOBANK, legs { Leg [ legId=R20170704l.1.1, product=FX_FORWARD, symbol=USDHUF, baseCcy=USD, termCcy=HUF, dealCcy=HUF, dealtAmount=4500000000, contraCcy=USD, contraAmount=16549289.30, valueDate=20170705, spotDate=20170706 ]Leg [ legId=R20170704l.1.2, product=FX_SPOT, symbol=USDHUF, baseCcy=USD, termCcy=HUF, dealCcy=HUF, dealtAmount=4500000000, contraCcy=USD, contraAmount=16549289.30, valueDate=20170706, spotDate=20170706 ] } ] } ]]

So I end up with

How do I filter this out so I can name both legs separately and allow their corresponding values to match up?

2

Are you using a kv filter for parsing this? If yes, wouldn't allow_duplicate_values => false help?

Apart from that I don't think there's any standard filter to help you here, but a simple ruby filter that rewrites a field if it's an array with two identical values.

3

Thanks Magnus that really helped