LS environment variable handling doesn't appear to behave as documented

Elastic Stack Logstash
1

Hi,

LS Version: 5.1.2 (RPM)

From the docs (https://www.elastic.co/guide/en/logstash/current/environment-variables.html):

You can add environment variable references in any plugin option type : string, number, boolean, array, or hash.

The way I read this, I should be able to use an environment variable to create any of the above types. Am I interpreting this correctly? Every single example I have seen is used for a string type and no matter what I try, I can't get LS to treat an environment variable as an array, particularly an empty array.

Or am I reading this wrong, and actually have to do something like this?

key => ["%{ARRAY_VAR_1}","${ARRAY_VAR_2}"]

If so, then how to I represent an empty array? I need to be able to create an empty array for the particular input that I am using. For the beats input I am trying to specify an empty SSL CA list through environment variables.

Thanks!
Nick

2

Actually, you can inject an environment variable into a logstash array. But you can't define an array in an environment variable that you inject in logstash.
Sorry.

Here's a sample :
https://www.elastic.co/guide/en/logstash/current/environment-variables.html#_setting_the_value_of_a_tag

3

Thanks Fabian,

I had a feeling this might be the case. This will probably effect anyone who has similar requirements to me, which are:

  • Logstash configuration is stored in a GIT repo that is deployed to the box via an orchestration tool (e.g Puppet vcs repo resource). I imagine this is a common use case for non-trivial logstash configs, or where the logstash configuration team is separate from the network operations team.
  • Want to control input plugin parameters via environment variables. As basic flow control is unavailable to input plugins, the only apparent way to control the input plugins (externally) is through environment variables.

I can potentially work around this by using .gitignore against my beats plugin config files, but this means that I'll have to separate my logstash configuration between two unrelated GIT repos (logstash & puppet), which will complicate things.

I see a couple of ways this could be fixed in logstash.

  1. Allow flow control constructs in input plugins (such as if-else).
  2. Allow one to define any type of logstash type through environment variables. This seems like it wouldn't be very difficult to implement, a simple string replacement would suffice (i.e if environment variables included in the config didn't have to be wrapped in quotes).

Do you know if either of these "features" are on the logstash dev pipeline?

Regards,
Nick

4

To my known, no such feature is in the roadmap.
But feel free to open a new issue for that :

But maybe you could simply use array injection like that :
key => ["${ARRAY_VAR_1}","${ARRAY_VAR_2}"]

5

One last thing : env var injection is not enabled by default if you use Logstash 2.x
You have to set a command line flag to enable it.

6

Thanks Fabian..

As far as this bit goes

key => ["${ARRAY_VAR_1}","${ARRAY_VAR_2}"]

It works great until I need to specify an empty array (or just nothing), which is what I want to do on the beats input for the list of SSL CA file paths. It's currently the only way to configure the beats input to use SSL without the use of client-side certs. I've filed a separate pull-request to have this issue fixed (https://github.com/logstash-plugins/logstash-input-beats/pull/182)

Also, regarding the flag to enable environment variables; as far as I know, I haven't done anything to logstash to enable environment variables, but they are working just fine if I put them in /etc/sysconfig/logstash (RedHat based OS) .

FYI, this is what I'm trying to do, and it all works great except for the CA list:

input {
  beats {
    include_codec_tag           => false
    port                        => "${LS_BEATS_PORT:5044}"
    ssl                         => "${LS_BEATS_SSL:true}"
    ssl_certificate             => "${LS_BEATS_CRT:/etc/logstash/cert.crt}"
    ssl_certificate_authorities => "${LS_BEATS_CA:[]}" <- I wish this would work
    ssl_key                     => "${LS_BEATS_KEY:/etc/logstash/key.pkcs8}"
    ssl_verify_mode             => "${LS_BEATS_VERIFY:none}"
  }
}

Cheers,
Nick

7

If it works out-of-the-box, you probably use Logstash 5.x.

To define an empty array, have you try this config :
ssl_certificate_authorities => "${LS_BEATS_CA:}"

And if you want to define a single value, you could set into LS_BEATS_CA, a path without brackets. It should work.

8

Thanks for the hint Fabien, (sorry I've been misspelling your name!)

I have tried that and this is what happens:

[logstash.inputs.beats    ] Error when creating a connection {:exception=>" (No such file or directory)"}

And I believe the reason for this is because "${LS_BEATS_CA:}" evaluates to an empty string, not nothing. The logstash beats logstash input takes this empty string and tries to set up a CA list based on it, making the SSL library upset.

Cheers,
Nick

9

No problem for my name , your're not the first and probably not the last to mispell it :slight_smile:

Concerning your problem, your analysis is right.
I hoped that beats input ignores empty strings, but it doesn't.

To tell you everything, I'm the guy who implemented environment variable injection. And when I did it, I knew that someday, someone would ask array injection... and this is the day :slight_smile:

I invite you to open an issue on logstash github to ask both full array injection and empty array injection.
I have ideas to implement both.
I will try to do that.

10

For no ca file, you can eventually test :

  • /dev/null
  • reference an empty file
11

Thanks for that, much appreciated.

I'll create an issue in github and reference this.

12

Hi Fabien,

I have tried '/dev/null' and an empty file. What happens is the beats input passes these paths to the SSL library. When a client then attempts to communicate to the beats input, the SSL library then forces the use of client side certs, and then fails when no certificate is provided (by the client). Having a look at https://github.com/logstash-plugins/logstash-input-beats/blob/master/src/main/java/org/logstash/netty/SslSimpleBuilder.java, it appears by setting anything in the CA list, requireClientAuth() will return true, then because verify mode defaults to FORCE_PEER, SSL connections fail, even if the user has specified NONE as the verify mode in the beats configuration.

I've put in a pull-request (https://github.com/logstash-plugins/logstash-input-beats/pull/182) to have the beats input fixed. Still waiting on a response for that.

I have also created this in an attempt to have the Java SSL code fixed up. Apparent bug in logstash-input-beats in SslSimpleBuilder.java. If someone from the appropriate team is happy, I'll create another issue for that.

Cheers,
Nick

13

Indeed, your PR makes sense. It looks like a bug fix.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.