Is there a way to split an event into multiple messages, and send them to elasticsearh?
The problem is the event cannot be splitted in a automatic way using plugins, because the data isn't exactly structured and may vary. The only option is to substring it with 'if' clauses and add new fields as is needed, so the splitted messages will be different between each other.
If there is no way of doing this inside logstash, i guess i will have to pre-process the logs before sending them to logstash.
As you can see, the messages are just a bunch of bytes separated by semicolons. The thing is that sometimes a group of these bytes represents a single field in elasticsearch, so the spliting needs to be scripted.
In this example i would compare the first byte ('03') to a specific value, if that checks then i would proceed to split the message in the following manner:
Note: the start and end of each msg may vary, and i can determine those positions using 'if' clauses (the values checked aren't important, they are in each submessage and i only need to look for the 'first byte' of each one).
After splitting the event, i need to assign new fields to each submessage in a scripted way (by using 'if' clauses, processing the bytes inside). All the messages will have the same fields, but the value of each one may be obtained in different ways.
At this point i will have, theoretically, a list of different messages ready to be sent to elasticsearch.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.