Hi Community, Thanks for your great work.
I am very new this ELK stack and gone through many tutorials sofar. I wanted to setup log monitoring for my trading application where my application receive trades and slices into multiple child orders to different exchanges. I wanted to setup log monitoring on these below example log patterns. so exploring the best way to setup the index in ES. Normally Trading application have new order then amends orders. I have few queries on these setup
my application logs patterns looks like below ( 4 different types)
LOG LINE TYPE 1: Recieved NewTrade TradeId=1 Price=99.9 Quantity=100 Stock=AAP.NY Client=ABC ...(several other fields)
LOG LINE TYPE 2: Sending New Child Order1: TradeId=2 Price=99.9 Quantity=50 Exch=NY ....(several other fields)
LOG LINE TYPE 2: Sending New Child Order2: TradeId=3 Price=99.9 Quantity=50 Exch=LN ....(several other fields)
LOG LINE TYPE 3: Recieved AmendTrade TradeId=1 PendingPrice=99.8 PendingQuantity=150
LOG LINE TYPE 4: Sending Child Order1 amend: TradeId=2 PendingPrice=99.8 PendingQuantity=150 Exch=NY
LOG LINE TYPE 4: Sending Child Order2 amend: TradeId=3 PendingPrice=99.8 PendingQuantity=150 Exch=LN
Which option is good?
option1: setting up different indexes on each log pattern above and visulalize? Will this give correct index optimization in ES?
option2: streamline all these patterns into single unique pattern so that only one index will be created? there are few fields which were missing/extra for few log patterns. in this case will it be best performance in ES?
Please suggest me your ideas also to keep me on right direction. Thanks a lot advance.