Creation of multiple logstash pipelines using API

Elastic Stack Logstash
1

I have a use case to create multiple Logstash pipelines inside a running Logstash container, Is it possible to use Logstash APIs as I do not have Elasticsearch and Kibana host. Just a Logstash running inside a cluster.

I am able to process a GET response, but PUT, POST is giving me 404 not found error.

2

Can you share what you are trying to do? I don't think Logstash has any API endpoint to create pipelines.

3

yes @leandrojmp, so I have a Logstash container running on my EKS via helm chart.
My use case is if I want to create a new pipeline in that Logstash I want to call an API to create it.
As of now, I am updating the pipeline details and upgrading the helm chart and re-deploying Logstash.

Is there any possibility that I can do it via an API endpoint?

4

That is what I asked, you said you got an 404 error, but what endpoint you tried?

Logstash does not have any endpoint to update its pipeline, the only exposed endpoint is the monitoring endpoint.

5

I tried a GET request which worked, showing the current pipelines.

bash-4.2$ curl -XGET http://localhost:9600/_node/stats/pipelines?pretty
{
  "host" : "test-logstash-0",
  "version" : "7.16.2",
  "http_address" : "0.0.0.0:9600",
  "id" : "73c80a9d-5d75-45bf-a4d4-2a42362d8139",
  "name" : "test-logstash-0",
  "ephemeral_id" : "ccb8eeb9-ff3d-48f5-a3c9-ae9351ba3c8d",
  "status" : "green",
  "snapshot" : false,
  "pipeline" : {
    "workers" : 1,
    "batch_size" : 125,
    "batch_delay" : 50
  },
  "pipelines" : {
    "main" : {
      "events" : {
        "filtered" : 1203,
        "duration_in_millis" : 761,
        "queue_push_duration_in_millis" : 0,
        "out" : 1203,
        "in" : 1203
      },....

But when I tried creation a POST method, It gave me error:

curl -XPUT -H 'Content-Type: application/json' 'http://localhost:9600/_node/pipelines/my-pipeline' -d '
{
  "description": "My new pipeline",
  "processors": [
    {
      "rename": {
        "field": "foo",
        "target_field": "bar"
      }
    }
  ]
}'
"
ERROR:

{"path":"/_node/pipelines/my-pipeline","status":404,"error":{"message":"Not Found"}}

6

Yeah, this endpoint does not exist.

As I said, there is no API endpoint to update logstah pipelines, you need to edit the files directly in the filesystem.

Also, the pipeline you tried to apply is not a Logstash pipeline, it is an Elasticsearch Ingest pipleine, which is a different thing.

1 Like
7

okay, thank you for the clarification.
Is there any other way I could create new pipelines without updating the file?

8

The only other way is the Centralized Pipeline Management, which needs Elasticsearch and Kibana with a paid license.

So, without it is not possible, only editing the files.

9

ok, Thank you for your help!

10

The centralized management might be used in the trial 30days mode. Latter just revert to the basic license.

If you want to rename a field name, do it directly in .conf with mutate-rename.
Or you can use runtime fields in ES.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.