Programmatically update Logstash configuration file

[Matt_Jones (Matthew Jones)]

Hi,

I would like to create multiple ingestion pipelines in Elastic Cloud, each of which containing a pipeline configuration (inputs, filters, outputs).

Different pipelines are being created to support different customers (we’re an MSP), therefore each pipeline will be slightly different, in that different syslog and beats ports will be specified in the respective inputs. Likewise, outputs will be different in terms of index names, each of which will contain a unique customer Id (taken from a CRM system).

So far, I’ve used Python to search for strings matching the pattern of certain placeholders (ports and customer Id), update them with new values, and then create the pipeline in Elastic Cloud using the API. In addition, I’m also reading and updating logstash.yaml with new pipeline IDs.

Ideally, the pipeline configuration (inputs, filters, outputs) would be in a format recognised by Python (and other languages) such as JSON or YAML (like other Logstash configuration files), so it could be easily parsed and the key values updated, rather than searching for and replacing strings which seems clunky in what appears to be a proprietary configuration format?

What I would like to know is, is there a better way of achieving this, as in deploying custom pipelines in Elastic Cloud that contain differences based on the customer the pipeline relates to? Am I going about this the wrong way? Can the pipeline configuration be parsed using Python or some other method?

Any advice is much appreciated.

Thanks,

Matt

[rugenl (Len Rugen)]

If it's basically a template with different values for each customer, something like Ansible might work. I use it on our own logstash systems, not sure how the pipeline configs are done in the cloud.

[Matt_Jones (Matthew Jones)]

Thanks for the reply!

Yes, that's what I'm trying to figure out.

For example, this is part of what is in the configuration/pipeline definition template:

input {
  syslog {
    port => <syslog_port>
    type => "syslog"
  }
}

Then, for customer #1, I would like to generate the following:

input {
  syslog {
    port => 12345
    type => "syslog"
  }
}

Then for customer #2, I would like to generate:

input {
  syslog {
    port => 23456
    type => "syslog"
  }
}

And so on for subsequent customers (e.g. +1 for each port number)

In my code (Python), I'm currently doing the following:

logstash_template = "template.config"
port_placeholder = "<syslog_port>"

with open(logstash_template, "rt") as logstash_config:
        data = logstash_config.read()
        data = data.replace(port_placeholder, '{0}'.format(port_new))
        data = data.replace(org_placeholder, '{0}'.format(org_new.lower()))
        return data

If the configuration/pipeline definition was say in JSON format, I could use python to treat it as a dictionary and do something like:

data["input"]["syslog"]["port"] = 34567

Where this gets messy is that for each pipeline definition that I generate, I would like to iterate port numbers by +1 of the port number in the most recent pipeline.

Using the Elastic API to retrieve a pipeline configuration (https://./api/logstash/pipeline/), it returns a JSON formatted response as below (shown in part):

{
    "id": "Customer1",
    "description": "Customer1 pipeline",
    "username": "abcd",
    "pipeline": "input {\n  syslog {\n    port => <syslog_port>...."
}

Whilst I could easily parse the JSON response using Python, the "pipeline" key value is a string that I then need to mangle using string slices or something else which is nasty! Again, if it was nested JSON, I could parse and update the "port" key value no problem.

As I said at the start, I might be going about this the wrong way and hoping that someone out there has faced into a similar challenge and knows a better way or can confirm that the nasty way is the only way!

Thanks,

Matt

[rugenl (Len Rugen)]

My pipelines are different enough that I can't just +1 the port and reuse the templates. A lot of ELK is first doing the messy, then revise it a few times till it's elegant