Formatting output of report for Logstash / Elasticsearch

Hey there. I have been tasked with writing a report to obtain data re: cpanel account sites (think disk and bandwidth utilization, account packages, etc.). This started me down the path of the Elastic Stack, which I have zero experience with. Last evening I set up a VM and have it all configured to play around with, but have been having issues with formatting the output of my report in a manner in which it can easily be ingested into Elasticsearch directly.

My question is this: does it make sense to format my report in a manner that is delimited, rather than JSON, so that I can simply set a grok pattern in Logstash, and ingest it that way? To me that seems to be the easiest means of approaching this. Otherwise, I would have to programmatically generate the JSON output, or create a complex mapping just to get the original output to ingest - both much more difficult tasks, unless I am missing something.

Not sure if this is entirely clear, and am happy to answer questions as needed. I am really just looking for comments on whether or not I am way off the path here. Thanks!

Hi @createchange You don't have to programmatically generate JSON yourself and let Logstash or Elasticsearch do it. Raw data can be processed using Logstash grok or dissect filter plugins. If your data in key-value format, then kv filter can be used.
Processing of data can be done in Elasticsearch as well, using ingestion pipelines. Dissect, grok and kv ingestion processors are available in Elasticsearch.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.