Parsing web hook data in logstash

Elastic Stack Logstash
1

Hello,

I am bringing the data thru webhook into logstash but can i please know how the parse that data and send it to separate index. I have provided the http://logstashIp/_hooks/gitlab in the gitlab webhook section which is sending the data to logstash and when i start the log stash with below comment i could see the data.

bin/logstash -e "input { http { port => 9200 } } output { stdout { codec => rubydebug} }"

Can i please know how should i configure the logstash to parse the data and send it to elastic search.

Thanks

2

Use the filter between input and output block to parse the data and in output section use the Elasticsearch output to send the data to an index in ES. i would say use a logstash config file to write your configuration and run your logstash with below command.

bin/logstash -f logstash.conf(your config file)

3

What do you currently get from Logstash (i.e. what does your stdout output produce)? What would you like to see in Elasticsearch?

4

Thank you for the reply. I see the below JSON data in logstash when I start the logstash in rubydebug .. I am trying to extract specific field from JSON data like timestamp, email, URL and send this extracted data to an index called "webhook_git".Can you please help with input.conf (i think i have to mention json codec) , filter.conf, output.conf .

  "head_commit" => {
        "timestamp" => "2018-05-21T22:59:27+02:00",
         "distinct" => true,
           "author" => {
             "name" => "root",
            "email" => "root@apache.sas.usa.com"
        },
              "url" => "https://gitlab.slack.corp/servicecode/apaceh/commit/1977a0354343554346dfb25e58d3cc6e9ea99c7c0dd2",
          "removed" => [],
            "added" => [
            [0] "apps/apache/usa01/deployment-apachwebpage/apache_index_html/default/1"
5

JSON is a string representation of structured data. What you see here is structured data. There's nothing to extract. What you have can be sent straight to ES. If you don't want the fields to be nested under head_commit you can use a mutate filter's rename option to move the fields to the top level, i.e. rename [head_commit][timestamp] to timestamp and so on.

6

Generally , logtype is used in output.conf to send specific data to elastic search index -but how to configure JSON data in output.conf. Is the below configuration is correct ?

if [fields][logtype] == "json" {
  elasticsearch {
    hosts => ["https://15.1.200.31:9200","https://15.1.199.32:9200"]
    manage_template => false
    user => 'logstash_internal'
    password => 'xxxxxxxxxxxx'
    index => "webhook_json-%{+YYYY.MM}"

In filter.conf, can i please know what is the use of source , target ?

filter {
  json {
    source => "message"
    target => "parsed"

Thanks

7

but how to configure JSON data in output.conf. Is the below configuration is correct ?

There's nothing obviously wrong with your configuration snippet, but whether they work depends on what your events look like.

In filter.conf, can i please know what is the use of source , target ?

The json filter documentation describes what these options do, including examples. I don't know what else to say.

8

Thank you for the reply.. What i am trying to understand is how to write the inputs.conf , filter.conf for the JSON data coming from webhook. Like from below example, how can we write the configuration for JSON data from webhook and how to refer the JSON data in filter.conf ?

input {
  file {
    path => ["/var/log/syslog"]
    type => "syslog"
  }
}
9

I think it'll be much easier for everyone if you give a concrete example. What does your stdout output produce? What's the expected outcome?

See https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html for more on how to reference fields in an event from filters.

10

Below is the sample stdout of JSON data from webhook. It is directly coming from github webhook to logstash

   "after" => "3e3b658073459085164e78751b658e5593482391",

"head_commit" => {
"committer" => {
"email" => "root@apache.com",
"name" => "root"
},
"author" => {
"email" => "root@apache.com",
"name" => "root"
},
"removed" => [],
"timestamp" => "2018-03-21T22:39:46+02:00",
"message" => "apache configs",
"distinct" => true,
"url" => "https://github.com/apacheteam/apacheWeb/commit/3e3b658073459085164e78751b658e5593482391",
"id" => "3e3b658073459085164e78751b658e5593482391",
"tree_id" => "146cb13616443736a30b35505c8a0859373ffaca",
"added" => [],
"modified" => [
[0] "apps/apache/web/apps/deployment-apache"
]
},

My question is how to write inputs , filters , outputs for the above JSON data that is coming from webhook. Generally, i use logtype field to refer data in filter.conf and output.conf. Since this data is directly coming from webhook , i don't know how to refer the data.

11

My question is how to write inputs , filters , outputs for the above JSON data that is coming from webhook.

You already have the input and outputs and filters work exactly the same as if you had obtained a message with this structure from a file input.

Generally, i use logtype field to refer data in filter.conf and output.conf.

You can add such a field via your http input if you want. It works exactly the same as a file input.

Since this data is directly coming from webhook , i don't know how to refer the data.

Well, I can't give a specific answer unless you ask a specific question.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.