Sending log payload to ES

[Souciance_Eqdam_Rash (Souciance Eqdam Rashti)]

Hello,

We have an application that generates logs which contain data payload. The data payload itself can be json, xml, some random CSV or any other format.

Is it possible to send the logs with the included payload to ES via normal tcp or http and then search view the data (including the data in the payload) in Kibana?

Do we need to configure any filters in logstash?

We are not interested in generating an output but simply to feed to ES data of varying format and be able to search it.

Thanks,

Best
Souciance

[warkolm (Mark Walkom)]

You can do it but the results may not be the best due to the varied formats.
Using Logstash would be better.

[Souciance_Eqdam_Rash (Souciance Eqdam Rashti)]

But even with logstash, how would you go about doing that? Basically we have logs that have the structure:

blabalbla xml-payload

or
blablalba json-payload
or
blablalba<random CSV or other similar format payload

If we were to send this via TCP, would logstash "automagically" parse this?

[warkolm (Mark Walkom)]

No it wouldn't, you'd need to figure out a way to run a conditional on it.