Is Logstash 5.5 compatible with Elasticsearch 6.2?

Elastic Stack Logstash
1

Hi,

Is it possible to send logs to Elasticsearch 6.2 from Logstash 5.5? When I look at matrix table it says, Elasticsearch 6.2 is compatible only with Logstash 5.6 - 6.2. Does that mean that Logstash 5.5 will not be able to send logs to Elasticsearch 6.2? Please shed some light on this.

Thanks

2

It depends.

If you can't use Logstash 6.x for some reason, I'd recommend updating your Logstash to the latest-available 5.6.x, which as of the time of this writing is v5.6.8; doing so will also get you Logstash-5.x-compatible improvements that we've made to the Elasticsearch Input, Output, and Filter plugins in the months since 6.0 was released.

Those plugins are tested extensively in a variety of combinations, including by running on Logstash 5.6.x while connecting to Elasticsearch 6.2 (example: output plugin test matrix).

Logstash 5.5.3 was the last release in the 5.5 series, and since it was released in August of 2017 before Elasticsearch 6.0 was released, it doesn't include any improvements we've made to the plugins; since the 5.6 series is fully backward compatible with 5.5, there's no reason not to upgrade.

In order to ensure our community and customers have a good experience, we put a lot of effort into testing our products together, and we release the whole Elastic Stack in lock-step, ensuring that by simply using matching versions, folk can use a combination that is known to have undergone extensive testing. The more the versions diverge, the less likely it is that anyone has really drilled into that specific combination and the more likely that the two are incompatible in some way.

In the Elastic Stack, version 5.6 was something of a "forward compatibility release"; it had fewer features than we normally pack into a minor version update, but we focused very heavily on ensuring that it bridged the gap to make it easier for users to upgrade (and that includes a gradual migration, where one product is migrated before the others, which would mirror your configuration).

2 Likes
3

Thank you. I will upgrade logstash to 6.2. I have another question- This is main Logstash and this Logstash would receive logs from multiple Logstash(Logstash 5.5, Logstash 2.4). If I upgrade my main Logstash to 6.2, would it be possible for it to receive logs from Logstash 2.4 and Logstash 5.5?

4

Again, keep in mind that the wider variety of versions we are using, the more likely we are to run into funky edge cases.

What are you using the other Logstashes for? Do they include complex transformation pipelines, or are you simply trying to get logs off of edge nodes? If the latter, the super-lightweight Filebeat is meant to be a really good way to get logs off of a box and shipped to Logstash.

5

Some App servers have Java 1.7 and that's the reason I installed Logstash 2.4(since Logstash 5.x or Logstash 6.x can't be installed on Java 1.7). And other App servers have Logstash 5.5(that was the latest when I installed Logstash). In the first case I can't upgrade Logstash to higher versions and in the second case I can upgrade but it would still be a pain since there are lot of Logstash that I need to upgrade.

Yes, logs undergo complex transformations(grok) which filebeat can't support.

So what would be your suggestion in my scenario?

6

A typical architecture these days is something like the following:

  • Filebeat installed on many edge machines to simply ship logs to one or more dedicated Logstash hosts (depending on volume); Beats are ultra lightweight, which minimises how much they interfere with the performance of your edge machines that are actually running the applications
  • Logstash on dedicated hardware (or virtualised hardware) receives those logs into one or more pipelines, and does all of the heavy lifting to transform the messages from their origin format into something structured and fully-contextualised.

This separates the concerns so that your application hardware is no longer directly tied down by the weight of processing its own logs.

On a separate note, Java 1.7 reached end-of-life in April of 2015, and is no longer receiving security updates from Oracle; you may be better off running a later JDK in 1.7-mode, which ensures language-level compatibility while benefitting from improvements to the JVM itself.

7

Got it. Thanks for the suggestion.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.