From an application, we generate well-formed json responses into a text file (read: structured output from Telegraf for InfluxDB, formatted in json), e.g.
{"tags":{"tag_1":"tag_1_value","tag_2":"tag_2_value"},"fields":{"field_1":31,"field_2":"two words"},"name":"test_meas","timestamp":1458229140000}
...parsed with jq for readability here:
{
"tags": {
"tag_1": "tag_1_value",
"tag_2": "tag_2_value"
},
"fields": {
"field_1": 31,
"field_2": "two words"
},
"name": "test_meas",
"timestamp": 1458229140000
}
Filebeat picks up these text files and passes them to Logstash for enrichment and ultimately output into InfluxDB.
My goal is to have many different metrics use the same Logstash code, without having to manually code Logstash for each different metric being output.
The inbound json documents is already defined with both the structure and the data that will go into InfluxDB.
So far, I have used a json output plugin to parse the field "message" into a field "payload". (I did this so I don't have to deal with all of the extraneous data that comes with the document, e.g. host, agent, log, ecs, @version, etc. and to future proof it from other new fields.) So, as I see it, use_event_fields_for_data_points is not appropriate here since I am only using a subset of the fields in the document.
But this is where I get stuck. As I understand it, I need InfluxDB output plugin's "data_points" to have ALL of the tags and fields. And "send_as_tags" needs to have an to enumerate list of the tags. Lastly, I need to coerce values that should be integers for consistency.
QUESTIONS
- Is there any elegant way to translate the payload's field and tag nodes into the hash that the data_points expects?
- Is there any elegant way to translate the payload's tag node to generate the array that send_as_tags expects?
Still to be resolved: how to enumerate a set of "coerce_values" keys for those fields that end in "i" to be integers (bonus credit)