ServiceNow Incident Table data in ELK


(Gautham) #1

Hey Guys,

I have been trying to get servicenow incident data into ELK through http_poller method using logstash, i'm getting the data but all the incident records are sitting under a single field.

Because of this i'm not able to create any visualization, as all the data are in single field i'm not able to break the data.
FYI...

All the incident data is falling under this record field.

Am i doing it rite, or do we need to use some other method to GET servicenow data into ELK.Please advice.

Thanks
Gautham


(Magnus Bäck) #2

Make sure you have codec => "json" in your http_poller configuration. See the example in the plugin's documentation.


(Gautham) #3

Thanks for the response @magnusbaeck I do have codec => "json"
Here is my config file.

input {
http_poller {
urls => {
url => "https://dev.service-now.com/api/208950/test_response"
}
request_timeout => 60
proxy => { host => "10..." port => "" scheme => "http"}
user => "*****"
password => "
*"
schedule => { cron => "
* * * "}
metadata_target => "http_poller_metadata"
codec => "json"
}
}
output {
elasticsearch {
hosts => ["10.
.
.
:9200"]
index => "servicetest"
}
stdout {
codec => rubydebug
}
}

Thanks
Gautham


(Magnus Bäck) #4

That's odd. What does the HTTP response actually look like, i.e. what do you get if you fetch the URL with e.g. curl?


(Gautham) #5

@magnusbaeck Here is the output

curl "https://dev53030.service-now.com/api/now/table/incident?sysparm_display_value=false&sysparm_fields=number%2Cstate%2Cpriority&sysparm_limit=1" --request GET --user '*****':'*******'

{"result":[{"number":"INC0000001","state":"7","priority":"1"}]}

Thanks
Gautham


(Magnus Bäck) #6

Okay, but what about the records field that's included in the screenshot you posted earlier?


(Gautham) #7

That actually has a lot of data to be copied as i ran for whole Incident table.

Now for quick reference have ran for only one incident with limited fields as i posted the result above. still all the data are sitting under single field.

Thanks
Gautham


(Magnus Bäck) #8

I still want to see the raw contents of the records field.


(Gautham) #9

Sorry for the delay @magnusbaeck , here is the details you asked for.

records {
"impact": "1",
"assigned_to": "46b87022a9fe198101a78787e40d7547",
"closed_at": "2017-11-26 23:10:06",
"subcategory": "",
"work_start": "",
"sys_mod_count": "21",
"upon_reject": "",
"time_worked": "",
"additional_assignee_list": "",
"close_code": "Closed/Resolved by Caller",
"caller_id": "5137153cc611227c000bbd1bd8cd2005",
"upon_approval": "",
"sla_due": "",
"calendar_stc": "7937181",
"active": "false",
"calendar_duration": "1970-04-02 20:46:21",
"urgency": "1",
"work_notes": "",
"watch_list": "",
"approval": "",
"category": "network",
"resolved_by": "6816f79cc0a8016401c5a33be04be441",
"short_description": "Can't read email",
"severity": "1",
"escalation": "0",
"parent_incident": "",
"sys_domain_path": "/",
"company": "",
"sys_tags": "",
"closed_by": "9ee1b13dc6112271007f9d0efdb69cd0",
"opened_by": "681ccaf9c0a8016400b98a06818d57c7",
"problem_id": "9d3a266ac6112287004e37fb2ceb0133",
"correlation_display": "",
"cmdb_ci": "b0c4030ac0a800090152e7a4564ca36c",
"delivery_task": "",
"comments": "",
"sys_domain": "global",
"correlation_id": "",
"description": "User can't access email on mail.company.com.\n\t\t",
"state": "7",
"sys_id": "9c573169c611228700193229fff72400",
"hold_reason": "",
"sys_updated_on": "2018-02-25 20:16:07",
"made_sla": "false",
"sys_updated_by": "admin",
"priority": "1",
"approval_set": "",
"activity_due": "",
"notify": "1",
"group_list": "",
"reassignment_count": "1",
"knowledge": "false",
"work_end": "",
"expected_start": "",
"delivery_plan": "",
"resolved_at": "2018-02-25 19:56:12",
"sys_created_on": "2016-06-26 18:24:13",
"business_stc": "1892781",
"due_date": "",
"assignment_group": "d625dccec0a8016700a222a0f7900d06",
"approval_history": "",
"child_incidents": "",
"business_service": "",
"opened_at": "2017-11-25 23:09:51",
"caused_by": "",
"order": "",
"__status": "success",
"number": "INC0000001",
"contact_type": "",
"close_notes": "Closed before close notes were made mandatory\n\t\t",
"sys_class_name": "incident",
"business_duration": "1970-01-22 21:46:21",
"incident_state": "7",
"comments_and_work_notes": "",
"work_notes_list": "",
"reopen_count": "",
"location": "1083361cc611227501b682158cabf646",
"sys_created_by": "pat",
"parent": "",
"rfc": "",
"follow_up": "",
"user_input": ""
},

These fields are for single incident, like this i have received data for all the incidents available in my servicenow instance.
As you can see, all these are falling under a single field "record"

here is my configuration file.

input {
http_poller {
urls => {
url => "https://dev.service-now.com/incident_list.do?JSONv2&incident_list.do?JSONv2&display_value=True&sysparm_exclude_reference_link=True&sysparm_fields=resolved_by%2Cstate%2Ccmdb_ci%2Cpriority%2Ccaller_id%2Cshort_description%2Cassignment_group%2Cassigned_to%2Cseverity%2Clocation%2Cnumber&sysparm_limit=1sysparm_view=json_view"
}
request_timeout => 60
proxy => { host => "10.1.1.2" port => "8181" scheme => "http"}
user => "admin"
password => "********"
schedule => { cron => "* * * * *"}
codec => "json"
metadata_target => "http_poller_metadata"
}
}
filter
{
json
{
source => "records"
}
}
output {
elasticsearch {
hosts => ["10.1.1.4:9200"]
index => "servicenowinc"
}
stdout {
codec => rubydebug
}
}


(Magnus Bäck) #10

The records field is an array of objects. Kibana simply doesn't handle arrays of objects that well. Perhaps you should use the split filter on the records field and get one event per record? Then the fields will be available as I think you expect.


(Gautham) #11

@magnusbaeck I have not used split filter before, is there any documentation available for this, or can you gimme me some example config file, that will help me a lot.

Thanks
Gautham


(Magnus Bäck) #12

https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html


(Gautham) #13

Thank you very much @magnusbaeck for you guidance.....i'll go through docs.

Thanks
Gautham


(Gautham) #14

Bingo......It worked as i expected @magnusbaeck .. Thank you very much....

Thanks
Gautham


(system) #15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.