Skipping Integrations Server in favor of Logstash

Elastic Stack Logstash
1

Hello everyone,

I am new to the Elastic Stack and have learned a lot about it in the past weeks. I am currently looking into the Elastic Cloud as it got me exited after researching the Elastic Stack.

One thing I noticed is that Logstash is not incorporated into the Elastic Cloud, I only seem to find the Integrations Server when looking at the pricing calculator: ...

After doing some reading it became clear that the idea is that you still host your own Logstash instances. My question is the following: Is it possible to skip the Integrations Server completely, and to send logs (API's, syslog) into Logstash for transformation, and then connect it to Elasticsearch in the cloud? Would it still be possible to make us of Beats/Agents? Would skipping the Integrations Server be a bad idea?

Looking forward to your insights, as i'm kinda stuck on this. Thanks!

2

Welcome to the community.

You can send data to ES from:

  • any app directly to ES by using REST APIs
  • FB directly to ES via ingest pipelines
  • LS directly to ES, maybe best option if you have complex transformation
  • FB to LS then to ES, usually in case of net. segments, firewalls...
  1. Yes, you can send logs to LS, depend on inputs. Check input LS filters which support files, syslog, beats,elastic_agent,....You will need cloud_id, cloud_auth or api_key
input {
  syslog {
    port => 1514
  }
}
output {
  elasticsearch {
    cloud_id => "..."
    cloud_auth => "user:pass"
 }
}
  1. Yes it's possible to use beats and agents independently of LS. Only matter is to keep the same data structure of index.
  2. It's hard to say, depend on a case.
  • FB is data shipper with small events transformation
  • EA is almost-all-beats-in-one

Of course, others from The Jedi Order will suggest own opinions.

1 Like
3

Thank you for your reply, this helps a lot :slight_smile:

I'm planning to pull logdata from about 5 or 6 different systems (EDR, Firewall, APM), so I assume Logstash will be the best choice for my use case then.

4

Are you planning to use the APM tools from Elastic? If so, the integrations server is a requirement.

5

The organisation currently uses another tool for this, I intend to gather it's logs and feed it into Elastic.

I am interested in making use of Elastic APM tools to replace our current solution in the future, but that goes beyond me and is a management decision which might never happen.

6

You have Elastic APM agents, which is not the same as beats. Check which agents are supported for your case.

1 Like