Logstash connecting to both QA and PROD?

Elastic Stack Logstash
1

When Logstash bootstraps in my container, the logstash output is indicating it is registering BOTH my qa and prod endpoints. I have a series of if-else checks to connect to one or the other. But it doesn't look like logstash is honoring this and is registering both qa and prod endpoints. Can anyone suggest what is going on? This is the essence of my logastash config (Running in Docker) script:

output {
    if [Environment == "qa"] {
        elasticsearch {
           hosts => ["https://my-qa-endpoint:443/"]
        }
     } else if [Environment == "prod"] {
        elasticsearch {
           hosts => ["https://my-prod-endpoint:443/"]
        }   
      }
}

Docker Inspect shows the environment correctly being set:

        "Env": [
            "Environment=qa",
             "LANG=en_US.UTF-8",
              ...
         ]

Logs:

[2019-03-26T17:41:29,638][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://elastic:xxxxxx@my-qa-endpoint:443/, :path=>"/"}
[2019-03-26T17:41:29,913][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@my-qa-endpoint:443/"}
[2019-03-26T17:41:29,966][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-03-26T17:41:30,061][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://elastic:xxxxxx@my-prod-endpoint:443/]}}
[2019-03-26T17:41:30,062][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://elastic:xxxxxx@my-prod-endpoint:443/, :path=>"/"} [2019-03-26T17:41:50,109][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@my-prod-endpoint:443/", :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@my-prod-endpoint::443/][Manticore::ConnectTimeout] connect timed out"}

2

Not sure what you are trying to do there.

No matter. The elasticsearch output establishes a connection to ES on startup. It does not wait for an event that needs to be routed to ES.

3

We have two different endpoints - one to QA and one to PROD. Since we're using a docker container and the container is immutable - the only variables we have to play with is the "Environment" using which we either send traffic to QA elastic endpoint when the container is spun up in our QA environment or the PROD elastic endpoint when the container is spun up in our PROD environment.

Does this mean the the if-else conditions are pointless? How does logstash know to connect to QA or PROD endpoints at startup? is it just scanning the logstash.conf from top to bottom and plucking both elasticsearch endpoints and registering them?

4

In essence, yes.

Because it connects at startup, you cannot use a sprintf reference to a field on an event. In other words, this does not work...

mutate { add_field => { "[@metadata][environment]" => "${environment}" } }
if [@metadata][environment] == "QA" {
    mutate { add_field => { "[@metadata][instance]" => "https://qa-es.example.com:9200" } }
    else if ...
}
output { elasticsearch { hosts => "%{[@metadata][instance]}" } }

It's going to try to connect to both.

5

Ouch. This puts me in a bind. Here's another question then - lets say it DOES connect to both on startup (thats correct I hope - it will try to connect to both?) and the PROD connection fails in the QA environment due to security group rules (and vice versa ) - will the event traffic go to the connection that is successful in each environment?

6

No, you will have to route events conditionally. Otherwise (as I understand it) in the QA environment the failed connection to production will result in events backing up. This will eventually fill the in-memory queue and the input will stop reading new events. So you will need something like

filter {
    mutate { add_field => { "[@metadata][environment]" => "${environment}" } }
}
output {
     if [@metadata][environment] == QA {
         elasticsearch {
             hosts => ["https://my-qa-endpoint:443/"]
         }
 } else  {
        elasticsearch {
             hosts => ["https://my-prod-endpoint:443/"]
        }   
  }

You cannot reference ${environment} in a conditional in the filter section, that is why I do the add_field.

7

Interesting. Just to sum this up - the events will route conditionally BUT at startup Logstash will try to connect to both QA and PROD in QA environment, just that during Runtime, no events will be sent to PROD?

Also a nuance - are these equivalent?
if [@metadata][environment] == QA {
and
if [@metadata][environment] == "QA" {

8

Your summary is correct.

Whoops. I would expect a syntax error at startup for

if [@metadata][environment] == QA {

The double quotes are required.

9

So I just created a new image with the suggested. IT looks like when elastic tries to post an event, its only trying to post to the PROD (nothing is going to QA even though in the beginning logstash is indicating its connecting to QA successfully but PROD connections keep failing). In the logs I clearly see an Error posting to the PROD endpoint and QA doesn't show this event as well.

I'm wondering - if I CONSTRUCT the endpoints as part of the mutate and use those in the elasticsearch section of the conf - would that solve this? (Thinking of ways to fool bootstrapping to think there's only one endpoint)

10

That suggests to me that the wrong value is in [@metadata][environment].

You cannot construct the endpoint using a mutate, because the connections to endpoints are done at startup, before any events have passed through the pipeline to be mutated. That was what I was trying to say earlier when I mentioned sprintf references.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.