2 Inputs and 1 Output. Can we pass events from 1 input to other input

Elastic Stack Logstash
1

Hi All

i Have created 2 conf files
Conf1 : No problem in this conf file

    { input
         file....
    }

    {
    output
             elasticsearch 
            { 
             index => "test1"
                      }
    }

In the Second Conf file conf2:
I have 2 inputs and 1 output

{ input
elasticsearch { 
       index => "test1"
    }

 http_poller
{

Here i am passing the value from the output of config file1 which is basically the input now : conf1 as %{ variable}

}
}

   {
        output
           elasticsearch { 
    index => "test2"
        }
        {

I am not able to pass the value by reading the value from index test1 to the input of http_poller.
Please let me know if this is possible?

2

Did any one tried such scenarios?

3

I can't quite follow what you're trying to do here. Could you please re-explain it?

4

Hi

I have 2 sources
Source# 1 : Contains list of Incident#. This source contains on the incident numbers
Source#2 : Contains details on those Incident# like who created, when created, severity etc.This can be achieved on by passing the incident number

I have created a conf file : conf1

{ input
         file....
    }

    {
    output
             elasticsearch 
            { 
             index => "test1"
                      }
    }

With this conf file i am getting list of all the incidents.

Since now i want the details on those incidents, i want to pass this incident numbers one by one to other conf file

So here i am keeping 2 input.
Input 1 which is already streamed from Source 1 and having incident#

The value from input1( test1 index) as "incident number" will be passed to second input block on http_poller to get the details on incident in test 2 index

    { input
    elasticsearch { 
           index => "test1"
        }

     http_poller
    {

    Here i am passing the value from the output of config file1 which is basically the input now : conf1 as %{ variable}

    }
    }

       {
            output
               elasticsearch { 
        index => "test2"
            }
            {

This is bit complex, but its not working for me.
In log file, i am getting clear text as %{incidentnumber}
Instead i am expecting value from test1 index

5

Are these pipelines in different Logstash instances? They would need to be, if you're planning on having pipeline2 ingest the results of pipeline1. You cannot currently have one pipeline feed to another within the same Logstash instance.

6

Assume that you have 100 incident in conf1/test1 index. we can stop running this pipeline once it is loaded

In conf2. I am using API with http_poller input plugin to get details on each incident by passing incident number from test1 index one by one.

So not necessarily to run both together

7

That's really just 2 logstash instances with some sort of broker in between, then.

SOURCE1 <- LS1 -> OUTPUT1 <- LS2 -> OUTPUT2

It seems that your "broker" here is Elasticsearch.

Why are you using the http_poller to read from Elasticsearch instead of the elasticsearch input plugin?

8

I have file input as Source1 which just lists the number of incidents.
After posting this to ES ( test 1index)

I have an API which just gives the details of incident based on incident number. That is the reason i use
http_poller to run the API which gives the incident details.THis API works on the parameter of incident number. Now which is available in index: test 1.

I use http_poller to read from application as SOAP API to get detailed incident details

I hope now it is clear

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.