Question about sending zipped data over TCP to logstash

Elastic Stack Logstash
1

Hi Folks,

I've been reading through the logstash documentation but am still a noob so sorry if this is a naive question. Here is what I want to solve :

We have about 1000 remote clients. Each client will send zipped (lzip) text data over TCP to a server/daemon at frequent intervals. The daemon needs to accept the data, unzip it, parse it, then write the parsed data to an SQL server. The zipped data will vary in size and the rate of sending data could be high.

After looking around it seems logstash may be the the way to do this. I will likely need to write some plugins to get exactly what i need - input , codec and output plugins to do the accept, parse and write-to-SQL portions.

My question is - does it seem I'm on the right track ? Also, if anyone has done something similar and has any pointers or advice I would be grateful.

Thanks
arn

2

LS has mixed support for compressed files. The TCP input won't do it so you'd have to figure out something to dump it to a file, then unzip it and then process it.

I'm not entirely sure that LS is good for the middle parts, but maybe others can comment too.