# Parsing web hook data in logstash **URL:** \ **Category:** Logstash\ **Created:** [August 20, 2018, 6:53pm UTC](https://discuss.elastic.co/t/parsing-web-hook-data-in-logstash/145224 "2018-08-20T18:53:36Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![iamlearner123](https://avatars.discourse-cdn.com/v4/letter/i/bcef8e/32.png) [@iamlearner123](https://discuss.elastic.co/u/iamlearner123)\ **Post date:** [August 20, 2018, 6:53pm UTC](https://discuss.elastic.co/t/parsing-web-hook-data-in-logstash/145224/1 "2018-08-20T18:53:36Z")
Hello, I am bringing the data thru webhook into logstash but can i please know how the parse that data and send it to separate index. I have provided the _**[http://logstashIp/\_hooks/gitlab](http://logstashIp/_hooks/gitlab)**_ in the gitlab webhook section which is sending the data to logstash and when i start the log stash with below comment i could see the data. _ **bin/logstash -e "input { http { port =\> 9200 } } output { stdout { codec =\> rubydebug} }"** _ Can i please know how should i configure the logstash to parse the data and send it to elastic search. Thanks ---
**Author:** ![chandra0651](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandra0651/32/30484_2.png) [@chandra0651](https://discuss.elastic.co/u/chandra0651)\ **Post date:** [August 20, 2018, 7:02pm UTC](https://discuss.elastic.co/t/parsing-web-hook-data-in-logstash/145224/2 "2018-08-20T19:02:08Z")
Use the filter between input and output block to parse the data and in output section use the Elasticsearch output to send the data to an index in ES. i would say use a logstash config file to write your configuration and run your logstash with below command. `bin/logstash -f logstash.conf(your config file)` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 20, 2018, 7:11pm UTC](https://discuss.elastic.co/t/parsing-web-hook-data-in-logstash/145224/3 "2018-08-20T19:11:34Z")
What do you currently get from Logstash (i.e. what does your stdout output produce)? What would you like to see in Elasticsearch? ---
**Author:** ![iamlearner123](https://avatars.discourse-cdn.com/v4/letter/i/bcef8e/32.png) [@iamlearner123](https://discuss.elastic.co/u/iamlearner123)\ **Post date:** [August 21, 2018, 9:26pm UTC](https://discuss.elastic.co/t/parsing-web-hook-data-in-logstash/145224/4 "2018-08-21T21:26:08Z")
Thank you for the reply. I see the below JSON data in logstash when I start the logstash in rubydebug .. I am trying to extract specific field from JSON data like timestamp, email, URL and send this extracted data to an index called "webhook\_git".Can you please help with input.conf (i think i have to mention json codec) , filter.conf, output.conf . ``` "head_commit" => { "timestamp" => "2018-05-21T22:59:27+02:00", "distinct" => true, "author" => { "name" => "root", "email" => "root@apache.sas.usa.com" }, "url" => "https://gitlab.slack.corp/servicecode/apaceh/commit/1977a0354343554346dfb25e58d3cc6e9ea99c7c0dd2", "removed" => [], "added" => [ [0] "apps/apache/usa01/deployment-apachwebpage/apache_index_html/default/1" ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 22, 2018, 5:39am UTC](https://discuss.elastic.co/t/parsing-web-hook-data-in-logstash/145224/5 "2018-08-22T05:39:13Z")
JSON is a string representation of structured data. What you see here is structured data. There's nothing to extract. What you have can be sent straight to ES. If you don't want the fields to be nested under `head_commit` you can use a mutate filter's rename option to move the fields to the top level, i.e. rename `[head_commit][timestamp]` to `timestamp` and so on. ---
**Author:** ![iamlearner123](https://avatars.discourse-cdn.com/v4/letter/i/bcef8e/32.png) [@iamlearner123](https://discuss.elastic.co/u/iamlearner123)\ **Post date:** [August 22, 2018, 6:42pm UTC](https://discuss.elastic.co/t/parsing-web-hook-data-in-logstash/145224/6 "2018-08-22T18:42:55Z")
Generally , logtype is used in output.conf to send specific data to elastic search index -but how to configure JSON data in output.conf. Is the below configuration is correct ? ``` if [fields][logtype] == "json" { elasticsearch { hosts => ["https://15.1.200.31:9200","https://15.1.199.32:9200"] manage_template => false user => 'logstash_internal' password => 'xxxxxxxxxxxx' index => "webhook_json-%{+YYYY.MM}" ``` In filter.conf, can i please know what is the use of source , target ? ``` filter { json { source => "message" target => "parsed" ``` Thanks ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 23, 2018, 6:04am UTC](https://discuss.elastic.co/t/parsing-web-hook-data-in-logstash/145224/7 "2018-08-23T06:04:13Z")
> but how to configure JSON data in output.conf. Is the below configuration is correct ? There's nothing obviously wrong with your configuration snippet, but whether they work depends on what your events look like. > In filter.conf, can i please know what is the use of source , target ? The json filter documentation describes what these options do, including examples. I don't know what else to say. ---
**Author:** ![iamlearner123](https://avatars.discourse-cdn.com/v4/letter/i/bcef8e/32.png) [@iamlearner123](https://discuss.elastic.co/u/iamlearner123)\ **Post date:** [August 27, 2018, 5:43pm UTC](https://discuss.elastic.co/t/parsing-web-hook-data-in-logstash/145224/8 "2018-08-27T17:43:34Z")
Thank you for the reply.. What i am trying to understand is how to write the inputs.conf , filter.conf for the JSON data coming from webhook. Like from below example, how can we write the configuration for JSON data from webhook and how to refer the JSON data in filter.conf ? ``` input { file { path => ["/var/log/syslog"] type => "syslog" } } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 27, 2018, 6:13pm UTC](https://discuss.elastic.co/t/parsing-web-hook-data-in-logstash/145224/9 "2018-08-27T18:13:35Z")
I think it'll be much easier for everyone if you give a concrete example. What does your stdout output produce? What's the expected outcome? See [https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html) for more on how to reference fields in an event from filters. ---
**Author:** ![iamlearner123](https://avatars.discourse-cdn.com/v4/letter/i/bcef8e/32.png) [@iamlearner123](https://discuss.elastic.co/u/iamlearner123)\ **Post date:** [August 27, 2018, 6:34pm UTC](https://discuss.elastic.co/t/parsing-web-hook-data-in-logstash/145224/10 "2018-08-27T18:34:34Z")
Below is the sample stdout of JSON data from webhook. It is directly coming from github webhook to logstash ``` "after" => "3e3b658073459085164e78751b658e5593482391", ``` "head\_commit" =\> { "committer" =\> { "email" =\> "root@apache.com", "name" =\> "root" }, "author" =\> { "email" =\> "root@apache.com", "name" =\> "root" }, "removed" =\> [], "timestamp" =\> "2018-03-21T22:39:46+02:00", "message" =\> "apache configs", "distinct" =\> true, "url" =\> "[https://github.com/apacheteam/apacheWeb/commit/3e3b658073459085164e78751b658e5593482391](https://github.com/apacheteam/apacheWeb/commit/3e3b658073459085164e78751b658e5593482391)", "id" =\> "3e3b658073459085164e78751b658e5593482391", "tree\_id" =\> "146cb13616443736a30b35505c8a0859373ffaca", "added" =\> [], "modified" =\> [ [0] "apps/apache/web/apps/deployment-apache" ] }, My question is how to write inputs , filters , outputs for the above JSON data that is coming from webhook. Generally, i use logtype field to refer data in filter.conf and output.conf. Since this data is directly coming from webhook , i don't know how to refer the data. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 27, 2018, 8:00pm UTC](https://discuss.elastic.co/t/parsing-web-hook-data-in-logstash/145224/11 "2018-08-27T20:00:08Z")
> My question is how to write inputs , filters , outputs for the above JSON data that is coming from webhook. You already have the input and outputs and filters work exactly the same as if you had obtained a message with this structure from a file input. > Generally, i use logtype field to refer data in filter.conf and output.conf. You can add such a field via your http input if you want. It works exactly the same as a file input. > Since this data is directly coming from webhook , i don't know how to refer the data. Well, I can't give a specific answer unless you ask a specific question. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 24, 2018, 8:00pm UTC](https://discuss.elastic.co/t/parsing-web-hook-data-in-logstash/145224/12 "2018-09-24T20:00:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.