I implemented pipeline with JDBC input plugin, my table in mysql follow the EAV (Entity-Attribute-Value) structure, therefore I can't get all attributes that related to one user (Entity) in one row with mysql query. I need to get the attributes in separated rows and combine them in Application layer (in my case logstash filter stage).
My question is How can I combine the sepated rows (events) in one row (event) in logstash filter stage?
For example I getting this events as a query result:
[user_id] [last_sync_time] [name] [level] [property_name] [property_value] [hero_name] [hero_level] [hero_skill] 1423352 1631916107 ❤RAGA3❤ 247 user_yellow_artifact 65 hero_angel 78 15 1423352 1631916107 ❤RAGA3❤ 247 user_purple_artifact 15 hero_angel 78 15 1423352 1631916107 ❤RAGA3❤ 247 user_food 10638173 hero_victory 96 15 1423352 1631916107 ❤RAGA3❤ 247 user_energy 574822 hero_victory 96 15
I want to combine these events with this rules:
- All events with same user_id must be combine as one event
- Aggregating repeatitive fields such as last_sync_time, name, level as one field
- Adding each unique property_value such as user_yellow_artifact, user_purple_artifact and ... to final combined event
The final result after combining should be like this:
[user_id] [last_sync_time] [name] [level] [user_yellow_artifact] [user_purple_artifact] [user_food] [user_energy] [hero_angel_level] [hero_angel_skill] [hero_victory_level] [hero_victory_skill] 1423352 1631916107 ❤RAGA3❤ 247 65 15 10638173 574822 78 15 96 15
I think this can performing with logstash aggregate plugin, but this plugin designed for log combining, how can I use it in my case?
Thanks for any help.