Type string in a multiple input Http_Poller for different Grok pattern

Elastic Stack Logstash
1

Hi All,

I am trying to using multiple Url's for the http_poller and need to use a different Grok patterns for each of the Url's
I am aware that i can make use of type string in the Input plugins and make Grok patterns using 'If' condition , if it matches the type.
I tried to built with the below sample file ,Provide me help on building it .

input {
    http_poller {
    urls => {
    url1 => "https://hedeededededje"
    type[url1] => "JVM_Memory"
    }
    }
    http_poller {
    urls => {
    url2 => "https://wwdwjjwhewkjek"
    type[url2] => "GC"
    }
    }
    request_timeout => 60
    user => "tttttt"
    password => "yyyyyy"
    request_timeout => 60
    schedule => { cron => "*/5 * * * *"}
    	codec => "json"
    	metadata_target => "http_metadata"

    }
    }
    filter {
    if type[url1] == "JVM_Memory"{
    split {
    terminator => "\n"
    field => "message"
    remove_field => "tags"
    }
    }
    if type[url2] == "GC" {
    split {
    terminator => "\n"
    field => "message"
    remove_field => "tags"
    }
    }
    }
2

I think you have two options.

  1. Use a single http_poller input, and use the request metadata from each event (e.g., the name of the url used to generate the event) later in processing to determine which grok patterns to apply
  2. Use separate http_poller inputs for your separate data sources, tagging events created by each with a type, using the type of each event later in processing to determine which grok patterns to apply

Single Input

If all of your URLs share credentials, pass back data that can be handled by the same codec, and should run on the same schedule, you may be best suited to share a single input.

The name of the URL is added to the metadata of the resulting event, and you can use this in your pipeline:

input {
  http_poller {
    urls => {
      "JVM_Memory" => "https://hedeededededje"
      "GC" => "https://wwdwjjwhewkjek"
    }
    user => "tttttt"
    password => "yyyyyy"
    request_timeout => 60
    schedule => { every => "5m"}
    metadata_target => "http_metadata"
  }
}
filter {
  if [http_metadata][name] == "JVM_Memory" {
    # ...
  } else if [http_metadata][name] == "GC" {
    # ...
  }
}

Multiple Inputs

If the results cannot be processed by the same codec, or if you want to run them at different schedules or require separate credentials, you can use multiple inputs, tagging each with the type, and then use the event's type later in the pipeline:

input {
  http_poller {
    type => "JVM_Memory"
    urls => { "JVM Memory" => "https://hedeededededje" }
    user => "tttttt"
    password => "yyyyyy"
    request_timeout => 60
    schedule => { every => "5m"}
    metadata_target => "http_metadata"
  }
  http_poller {
    type => "GC"
    urls => { "GC" => "https://wwdwjjwhewkjek" }
    user => "tttttt"
    password => "yyyyyy"
    request_timeout => 60
    schedule => { every => "5m"}
    metadata_target => "http_metadata"
  }
}
filter {
  if [type] == "JVM_Memory" {
    # ...
  } else if [type] == "GC" {
    # ...
  }
}
1 Like
3

Can you please let me know what i need add or replace with '[name]' provided above

Regards,
Kamaleshwar

4

In the http_poller input, you can specify where the metadata is placed (in my example I placed it in a key http_metadata). The input is also configured to hit one or more named URLs with a urls directive (e.g., "JVM_Memory" => "https://hedeededededje" and "GC" => "https://wwdwjjwhewkjek")

Once the http_poller input has been configured as such, each event that it generates will include several useful subkeys, including [http_metadata][name] with the name of the URL (in this case JVM_Memory or GC).

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.