ELK web APIs

Elastic Stack Logstash
1

Hello,

I'm new to ELK Stack.

We would like to feed logs that are generated from our middleware application (Biztalk). I'm considering couple of options:

  1. My app generated an XML logs and whats the best way to feed them to ELK.
    Beats (read xml) -> Logstash (transform) -> Elk stack is the correct approach?

  2. My app can generated JSON if needed so can I leverage ELK webapi (if any) and feed ELK. This way I can avoid Logstash.

Please recommend.

2

JSON logs (one line per log entry, i.e. not pretty printed JSON) would be the easiest option and is much preferable to XML. Whether you want to have Beats post directly to Elasticsearch is up to you and your needs.

3

FYI we’ve renamed ELK to the Elastic Stack, otherwise Beats and APM feel left out! :wink:

4

Thanks Magnus,

If I choose to feed ElasticSearch directly (avoid Beats , logstash) what would be my options, Web Apis?

Also, what would be the best practice in general?

5

If I choose to feed Elasticsearch directly (avoid Beats , logstash) what would be my options, Web Apis?

Yes, Elasticsearch has an extensive REST API.

Also, what would be the best practice in general?

Don't make your logging framework (Log4j or whatever) send stuff over the network. If you have issues with the network or Elasticsearch you either have to

  • drop log messages,
  • buffer them in memory, or
  • block the application,

none of which are very appealing options. Just write the logs to disk (preferably in JSON format) and have Filebeat pick them up and ship somewhere.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.