# Snow API to avoid duplicates

**URL:** <https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074>\
**Category:** Logstash\
**Created:** [December 26, 2019, 12:23pm UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074 "2019-12-26T12:23:11Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\
**Post date:** [December 26, 2019, 12:23pm UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/1 "2019-12-26T12:23:11Z")

</div>

Hi Team,  
im working with a Snow API on my ELK setup (7.2.0),  
and below is the configuration of my logsatsh config file.

> ```
> input {
> http_poller {
> urls => {
> snowinc => {
> url => "https://service-now.com/api/now/table/incident&sysparm_display_value=true&sysparm_limit=10&sysparm_exclude_reference_link=true"
> user => "logstash"
> password => "Pass123"
> headers => {Accept => "application/json"}
> }
> }
> request_timeout => 60
> metadata_target => "http_poller_metadata"
> schedule => { cron => "* * * * * UTC"}
> codec => "json"
> }
> }
> output {
> elasticsearch {
> hosts => ["10.116.15.27:9200","10.116.15.28:9200","10.116.15.29:9200"]
> index => "incidentsnow"
> action=>update
> document_id => "%{number}"
> doc_as_upsert =>true
> }
> stdout { codec => rubydebug }
> }
> 
> ```

I initially did not have the

> ```
> action=>update
> document_id => "%{number}"
> doc_as_upsert =>true
> 
> ```

part and i received all the data with duplicates.

After i did make a doc ID, Im only receiving one inc data where as the link actually has 10.  
How do i get all and only unique values of Incidents.

Also, How does an update work. If a ticket gets updated, will the above config help me update the ticket in my index instead of duplicating it / adding a new row?

Here's a the first result set of the link, 10 result sets come with different incident details like below:

> ```
> <response>
> <result>
> <parent/>
> <made_sla>true</made_sla>
> <caused_by/>
> <upon_reject>Cancel all future Tasks</upon_reject>
> <sys_updated_on>2019-12-23 05:00:00</sys_updated_on>
> <child_incidents>0</child_incidents>
> <hold_reason/>
> <approval_history/>
> <number>INC001566</number>
> <resolved_by>Ana Ken</resolved_by>
> <sys_updated_by>system</sys_updated_by>
> <opened_by/>
> <user_input/>
> <sys_created_on>2019-12-18 03:25:37</sys_created_on>
> <sys_domain>global</sys_domain>
> <state>Closed</state>
> <sys_created_by>sa_Service</sys_created_by>
> <calendar_stc>85,372</calendar_stc>
> <closed_at>2019-12-23 05:00:00</closed_at>
> <cmdb_ci/>
> <impact>3 - Low</impact>
> <active>false</active>
> <business_service/>
> <priority>5 - Planning</priority>
> <sys_domain_path>/</sys_domain_path>
> <business_impact/>
> <time_worked/>
> <expected_start/>
> <opened_at>2019-12-18 03:25:37</opened_at>
> <business_duration>8 Hours</business_duration>
> <resolved_at>2019-12-19 03:08:29</resolved_at>
> <reopened_time/>
> <subcategory>Datacenter</subcategory>
> <short_description>
> [Critical Zenoss] STR675GH is DOWN!
> </short_description>
> <assignment_group>L1.5</assignment_group>
> <description>
> Timestamp: 2019/12/18 11:25:32.000000 Device: STR675GH IP address: 10.116.45.116 Component: Severity: Critical (5) Event class: /Status/Ping Device class: /Server/Microsoft/Windows/strwindows Device/Element priority: Normal (3) Summary: STR675GH is DOWN! Message: STR675GH is DOWN!
> </description>
> <cause/>
> <sys_class_name>Incident</sys_class_name>
> <closed_by>Ana Ken</closed_by>
> <parent_incident/>
> <sys_id>037879a91b714c10fe92437cdc4bcb31</sys_id>
> <contact_type>Monitoring</contact_type>
> <incident_state>Closed</incident_state>
> <urgency>3 - Low</urgency>
> <problem_id/>
> <reassignment_count>1</reassignment_count>
> <assigned_to>Beth Bezo</assigned_to>
> <severity>3 - Low</severity>
> <sla_due>UNKNOWN</sla_due>
> <escalation>Normal</escalation>
> <upon_approval>Proceed to Next Task</upon_approval>
> <location/>
> <category>Network</category>
> </result>
> <result>
> <parent/>
> <made_sla>true</made_sla>
> <caused_by/>
> <upon_reject>Cancel all future Tasks</upon_reject>
> <sys_updated_on>2019-12-13 00:43:58</sys_updated_on>
> <child_incidents>0</child_incidents>
> <hold_reason/>
> <approval_history/>
> <number>INC0010206</number>
> <resolved_by>Abraham Lincoln</resolved_by>
> <sys_updated_by>anak</sys_updated_by>
> <opened_by>Angel Ken</opened_by>
> <user_input/>
> <sys_created_on>2019-12-11 01:53:10</sys_created_on>
> <sys_domain>global</sys_domain>
> <state>Canceled</state>
> <sys_created_by>angke</sys_created_by>
> <calendar_stc>81,153</calendar_stc>
> <closed_at>2019-12-13 00:43:58</closed_at>
> <cmdb_ci/>
> <impact>3 - Low</impact>
> <active>false</active>
> <business_service/>
> <priority>5 - Planning</priority>
> <sys_domain_path>/</sys_domain_path>
> <business_impact/>
> <time_worked/>
> <expected_start/>
> <opened_at>2019-12-11 01:51:21</opened_at>
> <business_duration>8 Hours</business_duration>
> <resolved_at>2019-12-12 00:23:54</resolved_at>
> <reopened_time/>
> <subcategory>DNS</subcategory>
> <short_description>test</short_description>
> <assignment_group>Service Desk</assignment_group>
> <description>test</description>
> <cause/>
> <sys_class_name>Incident</sys_class_name>
> <closed_by>Ana Ken</closed_by>
> <parent_incident/>
> <sys_id>0ecf9d931ba54410fe92437cdc4bcb50</sys_id>
> <contact_type>Phone</contact_type>
> <incident_state>Canceled</incident_state>
> <urgency>3 - Low</urgency>
> <problem_id/>
> <reassignment_count>0</reassignment_count>
> <assigned_to>Beth Bezo</assigned_to>
> <severity>3 - Low</severity>
> <sla_due>UNKNOWN</sla_due>
> <escalation>Normal</escalation>
> <upon_approval>Proceed to Next Task</upon_approval>
> <location/>
> <category>directory_services</category>
> </result>
> 
> ```

Please do help me with these clarifications.

Thank you!

---

<div class="post-metadata">

**Author:** ![andres-perez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andres-perez/32/136461_2.png) [@andres-perez](https://discuss.elastic.co/u/andres-perez)\
**Post date:** [December 27, 2019, 4:53pm UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/3 "2019-12-27T16:53:11Z")

</div>

Hi, you can check previous issues with servicenow in the forum:

In this thread, there is a JSON response with a `records` field that contains an array of entries. So json input codec and split filter are used to get the information of each entry.

> [@ServiceNow Incident Table data in ELK](https://discuss.elastic.co/t/servicenow-incident-table-data-in-elk/130216/13):
>
> Thank you very much @magnusbaeck for you guidance.....i'll go through docs. Thanks Gautham

Maybe you can work on something similar for your case, although your response seems XML instead of JSON.

On the other hand, your configuration of

```
action=>update
document_id => "%{number}"
doc_as_upsert =>true

```

will update the existing document if a new one is indexed with the same `number`

- New fields (not present in the existing document) will be added to the same document;
- Existing fields will have their values updated with the latest ones.

---

<div class="post-metadata">

**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\
**Post date:** [December 30, 2019, 5:48am UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/5 "2019-12-30T05:48:36Z")

</div>

Thank you @andres-perez, for helping me out.

I am trying the split filter as mentioned.

> ```
> filter {
> split {
> field => "[response][0][result]"
> }
> }
> 
> ```

based on my data. Is this right? Coz im still not able to see all my data.  
Also tried

> field =\> "[response]"

and

> field =\> "[result]"

Where am i going wrong?

---

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [December 30, 2019, 8:30am UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/6 "2019-12-30T08:30:51Z")

</div>

Hi @katara

In which field you are getting your incident details?  
How the data is getting alligned in you Discover tab?

Getting all data under single field?  
If yes then apply split filter.

In my case i'm getting the data in "result" field, so i'm splitting it.

Thanks  
Gauti

---

<div class="post-metadata">

**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\
**Post date:** [December 30, 2019, 9:29am UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/7 "2019-12-30T09:29:49Z")

</div>

Hi @Gauti,  
My structure of XML looks like this: (this is a browser , so i get an XML output. will be JSON if i curl)

```auto
<response>
<result>...</result>
<result>...</result>
<result>...</result>
<result>...</result>
<result>...</result>
<result>...</result>
<result>...</result>
<result>...</result>
<result>...</result>
<result>...</result>
</response>

```

under each result set, the Incident data is stored.

```auto
<result>
<made_sla>true</made_sla>
<upon_reject>Cancel all future Tasks</upon_reject>
<number>INC001566</number>
<resolved_by>Ana Ken</resolved_by>
<sys_updated_by>system</sys_updated_by>
</result>

```

In my discover page, all my inc details fall into one result.  
I want splits applied for each result, so I can have all Incident records as separate rows in my ES.  
Help me configure the same.  
I tried everything that I've mentioned earlier.  
How should i apply split here?

---

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [December 30, 2019, 11:01am UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/8 "2019-12-30T11:01:43Z")

</div>

Try this filter

```
filter
       {
         split
                 {
                 field => "result"
                 }

```

Thanks  
Gauti

---

<div class="post-metadata">

**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\
**Post date:** [December 30, 2019, 11:19am UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/9 "2019-12-30T11:19:57Z")

</div>

@Gauti,  
I tried the above and my data still isnt split.  
Only the first row of my XML result has been added.

The data just is shown as XML, this output above is from chrome.  
When i curl it, it will give me a JSON output. Would like to avoid that confusion.

---

<div class="post-metadata">

**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\
**Post date:** [December 30, 2019, 12:08pm UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/10 "2019-12-30T12:08:47Z")

</div>

@Gauti, attaching a JSON response for reference:

> ```
> {"result":[
> {
> "made_sla":"true",
> "upon_reject":"Cancel all future Tasks",
> "sys_updated_on":"2019-12-23 05:00:00",
> "number":"INC0010275",
> "category":"Network"} ,
> {
> "made_sla":"true",
> "upon_reject":"Cancel all future Tasks",
> "sys_updated_on":"2019-12-24 07:00:00",
> "number":"INC0010567",
> "category":"DB"}]}
> 
> ```

attached 2 sets and I need each set to occur as a separate row.  
So far with the filters tried above,  
Im only getting one.  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/d/5da752495d02f024ba4297e4e308d35be1fc6bc8.png)

also, im getting the data split as different columns in Kibana, except only the first row (only one inc ticket detail) is getting indexed in ES.  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/6/f61d051dff376a3b2c8d1846228c346ec6892515.png)

@andres-perez, i believe this is where i differ from the question in the link you've provided me.

---

<div class="post-metadata">

**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\
**Post date:** [December 30, 2019, 5:22pm UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/11 "2019-12-30T17:22:32Z")

</div>

Team, if anyone has anything I could try, it would be very useful to me! 🙂 apologies, if this is taking a bit long.

Thanks in advance.  
Katara

---

<div class="post-metadata">

**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\
**Post date:** [December 31, 2019, 5:53am UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/12 "2019-12-31T05:53:20Z")

</div>

update:  
I tried adding

> json { source =\> "message" }

in the filters,  
yet no results.  
I need to figure out a way to split each result set.  
When i do

> split{ field =\> "result"}

all my data inside result gets split up as a separate column.  
There is no succeeding name for all my result sets to split each result set. How do I fix this? Only the first result set is loading in the ES.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 31, 2019, 6:22am UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/13 "2019-12-31T06:22:21Z")

</div>

In the Elasticsearch output you use the ‘number’ field as document id but it does not look like you have this at the document root level. Instead it is nested under ‘result’ which causes the same document to be repeatedly updated. Try changing ‘number’ to ‘[result][number]’ and see if this makes a difference.

---

<div class="post-metadata">

**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\
**Post date:** [December 31, 2019, 6:38am UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/14 "2019-12-31T06:38:03Z")

</div>

@Christian_Dahlqvist,  
Wow! This totally worked,  
Thank you so much!! 🙂

---

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [December 31, 2019, 7:08am UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/15 "2019-12-31T07:08:59Z")

</div>

@katara great that things had worked out.  
You can post your working config here, so that it'll be helpful for others who face the same issue.

Thanks  
Gauti

---

<div class="post-metadata">

**Author:** ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)\
**Post date:** [December 31, 2019, 9:31am UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/16 "2019-12-31T09:31:28Z")

</div>

Hi guys,  
FYI, this is my working configuration.

> ```
> input {
> http_poller {
> urls => {
> snowinc => {
> url => "https://service-now.com"
> user => "your_user"
> password => "yourpassword"
> headers => {Accept => "application/json"}
> }
> }
> request_timeout => 60
> metadata_target => "http_poller_metadata"
> schedule => { cron => "* * * * * UTC"}
> codec => "json"
> }
> }
> filter
> {
> json {source => "result" }
> split{ field => ["result"] }
> }
> output {
> elasticsearch {
> hosts => ["yourelastuicIP"]
> index => "incidentsnow"
> action=>update
> document_id => "%{[result][number]}"
> doc_as_upsert =>true
> }
> stdout { codec => rubydebug }
> }
> 
> ```

You'll be able to find the sample JSON input in the replies for your reference.  
Thanks to everyone who helped me out here! 🙂

Hope this helps 🙂

Regards,  
Katara

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 28, 2020, 9:31am UTC](https://discuss.elastic.co/t/snow-api-to-avoid-duplicates/213074/17 "2020-01-28T09:31:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.