# Logstash Http poller plugin

**URL:** <https://discuss.elastic.co/t/logstash-http-poller-plugin/256154>\
**Category:** Logstash\
**Created:** [November 20, 2020, 4:13pm UTC](https://discuss.elastic.co/t/logstash-http-poller-plugin/256154 "2020-11-20T16:13:51Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![jerin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerin/32/74357_2.png) [@jerin](https://discuss.elastic.co/u/jerin)\
**Post date:** [November 20, 2020, 4:13pm UTC](https://discuss.elastic.co/t/logstash-http-poller-plugin/256154/1 "2020-11-20T16:13:51Z")

</div>

Hello ,  
i am using logstash poller plugin to query JIRA API . it works okay .. i am trying to get the ticket status updated in last 5 mins .. the problem is when there is no update , it polls the API and retrives the fileds such as \_doc , \_version and \_score ..etc ..hence storage is getting full .. how do i remove these fields and can i remove these fields ? i used remove\_field but its not working as these fields are not from JIRA ..when there is no update , it should not return anything

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 20, 2020, 5:03pm UTC](https://discuss.elastic.co/t/logstash-http-poller-plugin/256154/2 "2020-11-20T17:03:29Z")

</div>

If the API responds then the input will create an event, even if it contains no useful data. You can use [conditionals](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html) to test whether the event contains information you want to retain, and [drop](https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html) the event if it does not.

---

<div class="post-metadata">

**Author:** ![jerin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerin/32/74357_2.png) [@jerin](https://discuss.elastic.co/u/jerin)\
**Post date:** [November 23, 2020, 4:09pm UTC](https://discuss.elastic.co/t/logstash-http-poller-plugin/256154/3 "2020-11-23T16:09:06Z")

</div>

Thanks for the response Badger .it works .

---

<div class="post-metadata">

**Author:** ![jerin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerin/32/74357_2.png) [@jerin](https://discuss.elastic.co/u/jerin)\
**Post date:** [November 24, 2020, 10:13am UTC](https://discuss.elastic.co/t/logstash-http-poller-plugin/256154/4 "2020-11-24T10:13:19Z")

</div>

Hello Badger ,  
I am facing an another issue .. While I am dropping the events that I dont want , some events are getting duplicated .. like when it polls the API , 3 or 4 times it sends the same ticket to ES .. Then I started to use fingerprint to de-duplicate which works but it deletes the previous events ..

Below is the code  
input {  
http\_poller {  
urls =\> {  
JIRA =\> {  
method =\> get  
user =\> "test"  
password =\> "test"  
url =\> "jiraurl"  
headers =\> {  
Accept =\> "application/json"  
}  
}}

```
  request_timeout => 20

  schedule => { every =>"1m"}

  codec => "json"

```

}}

filter {  
mutate {  
convert =\> { "total" =\> "integer" }  
}  
if [total] \> 0  
{

```
    fingerprint {

    add_field => {

            "reporter" => "%{[issues][0][fields][reporter][displayName]}"

            "assignee" => "%{[issues][0][fields][assignee][displayName]}"

            "status" => "%{[issues][0][fields][status][name]}"

            "ticketId" => "%{[issues][0][key]}"

            }

  source => ["reporter", "assignee", "status", "ticketId"]
  concatenate_sources => true
  target => "[@metadata][fingerprint]"
  method => "SHA256"                                                                          
  key => "test"
  base64encode => true                                                                            
       }

   mutate {

```

lowercase =\> ["%{application}"]  
}  
ruby {

```
    code => '

        event.to_hash.each { |k, v|

            if v == "" or v.to_s.start_with?("%{[issues]") or v.to_s.start_with?("%{application}")

                 event.remove(k)

            end

        }

    '

}

```

} else {

```
            drop {}

            }}

```

output

{

```
 elasticsearch {

 hosts => ["elasticsearch:9200"]

 index => "test-%{+YYYY.MM.dd}"

            document_id => "%{[@metadata][fingerprint]}"

```

}  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 24, 2020, 2:59pm UTC](https://discuss.elastic.co/t/logstash-http-poller-plugin/256154/5 "2020-11-24T14:59:18Z")

</div>

The add\_field option is part of event "decoration", which happens after the filter successfully executes (if it successfully executes). So the fields you are using for the fingerprint will not exist at the time they are used. Split that into a separate mutate filter before the fingerprint.

---

<div class="post-metadata">

**Author:** ![jerin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerin/32/74357_2.png) [@jerin](https://discuss.elastic.co/u/jerin)\
**Post date:** [November 24, 2020, 3:22pm UTC](https://discuss.elastic.co/t/logstash-http-poller-plugin/256154/6 "2020-11-24T15:22:55Z")

</div>

Hello Badger ,  
I have tried this option before .  
This approach does the below

1)Lets say I have ticket id called test-1234 status: open . Info gets updated in ES  
2) when I have updated the ticket test-1234 , status : InProgress .. instead of updating the ticket status to InProgress it puts an another entry

so I have two entries in ES  
test-1234 open  
test-1234 Inprogress  
I only need status to be updated instead of putting another entry ..

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 24, 2020, 4:32pm UTC](https://discuss.elastic.co/t/logstash-http-poller-plugin/256154/7 "2020-11-24T16:32:30Z")

</div>

If the update has all of the fields for the ticket then just use the ticket id as the document id.

If it does not then you can use conditionals to determine whether you are processing a complete ticket or an update and then set the options on the elasticsearch output as needed. I think that would look like

```
filter {
    if some condition {
        mutate { add_field => { "[@metadata][action]" => "index" "[@metadata][upsert]" = false } }
    } else {
        mutate { add_field => { "[@metadata][action]" => "update" "[@metadata][upsert]" = true } }
    }
}
output {
    elasticsearch {
        action => "%{[@metadata][action]}"
        doc_as_upsert=> "%{[@metadata][upsert]}"
        ...
    }
}
```

---

<div class="post-metadata">

**Author:** ![jerin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerin/32/74357_2.png) [@jerin](https://discuss.elastic.co/u/jerin)\
**Post date:** [November 25, 2020, 6:42am UTC](https://discuss.elastic.co/t/logstash-http-poller-plugin/256154/8 "2020-11-25T06:42:47Z")

</div>

Thanks Badger . i only updated the ticketid in the source fields instead of all .. its giving the results what I expected .. have to Monitor though for sometime

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 23, 2020, 6:43am UTC](https://discuss.elastic.co/t/logstash-http-poller-plugin/256154/9 "2020-12-23T06:43:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.