I'm new to logstash / beats and I'm wondering how best to go about transforming / merging two JSON documents. Here's an example of what I'm trying to do:
Document 1:
{
"id": 1234,
"things": [
{
"id": "1234",
"lat": 45.12344,
"long": 0.333232
},
{
"id": "1234",
"lat": 45.3322,
"long": 0.456543
}
]
}
Document 2:
{
"id": 1234,
"newField": 4321
}
What I want to go into elasticsearch:
{
"id": 1234,
"newField": 4321,
"things": [
{
"lat": 45.12344,
"long": 0.333232
},
{
"lat": 45.3322,
"long": 0.456543
}
]
}
So ... I have two documents at source (a REST API) that I want to merge based on the "id" field matching. I want to end up with a single document that contains elements of both source documents. In addition I am eliminating duplicate (and redundant) "id" fields in the first document.
Question:
Architecturally, how is the best way to go about this? Should I pre-process the documents before they hit logstash (maybe with a custom beat?), or can I handle this case via the aggregate and json plugins directly in logstash? Should I build a custom (specialized) plugin for logstash?