ServiceNow Incident Table data in ELK

Elastic Stack Logstash
1

Hey Guys,

I have been trying to get servicenow incident data into ELK through http_poller method using logstash, i'm getting the data but all the incident records are sitting under a single field.

Because of this i'm not able to create any visualization, as all the data are in single field i'm not able to break the data.
FYI...

image
image.png929×472 5.4 KB

All the incident data is falling under this record field.

Am i doing it rite, or do we need to use some other method to GET servicenow data into ELK.Please advice.

Thanks
Gautham

2

Make sure you have codec => "json" in your http_poller configuration. See the example in the plugin's documentation.

3

Thanks for the response @magnusbaeck I do have codec => "json"
Here is my config file.

input {
http_poller {
urls => {
url => "https://dev.service-now.com/api/208950/test_response"
}
request_timeout => 60
proxy => { host => "10..." port => "" scheme => "http"}
user => "*****"
password => "*"
schedule => { cron => " * * * "}
metadata_target => "http_poller_metadata"
codec => "json"
}
}
output {
elasticsearch {
hosts => ["10...:9200"]
index => "servicetest"
}
stdout {
codec => rubydebug
}
}

Thanks
Gautham

4

That's odd. What does the HTTP response actually look like, i.e. what do you get if you fetch the URL with e.g. curl?

5

@magnusbaeck Here is the output

curl "https://dev53030.service-now.com/api/now/table/incident?sysparm_display_value=false&sysparm_fields=number%2Cstate%2Cpriority&sysparm_limit=1" --request GET --user '*****':'*******'

{"result":[{"number":"INC0000001","state":"7","priority":"1"}]}

Thanks
Gautham

6

Okay, but what about the records field that's included in the screenshot you posted earlier?

7

That actually has a lot of data to be copied as i ran for whole Incident table.

Now for quick reference have ran for only one incident with limited fields as i posted the result above. still all the data are sitting under single field.

Thanks
Gautham

8

I still want to see the raw contents of the records field.

9

Sorry for the delay @magnusbaeck , here is the details you asked for.

records {
"impact": "1",
"assigned_to": "46b87022a9fe198101a78787e40d7547",
"closed_at": "2017-11-26 23:10:06",
"subcategory": "",
"work_start": "",
"sys_mod_count": "21",
"upon_reject": "",
"time_worked": "",
"additional_assignee_list": "",
"close_code": "Closed/Resolved by Caller",
"caller_id": "5137153cc611227c000bbd1bd8cd2005",
"upon_approval": "",
"sla_due": "",
"calendar_stc": "7937181",
"active": "false",
"calendar_duration": "1970-04-02 20:46:21",
"urgency": "1",
"work_notes": "",
"watch_list": "",
"approval": "",
"category": "network",
"resolved_by": "6816f79cc0a8016401c5a33be04be441",
"short_description": "Can't read email",
"severity": "1",
"escalation": "0",
"parent_incident": "",
"sys_domain_path": "/",
"company": "",
"sys_tags": "",
"closed_by": "9ee1b13dc6112271007f9d0efdb69cd0",
"opened_by": "681ccaf9c0a8016400b98a06818d57c7",
"problem_id": "9d3a266ac6112287004e37fb2ceb0133",
"correlation_display": "",
"cmdb_ci": "b0c4030ac0a800090152e7a4564ca36c",
"delivery_task": "",
"comments": "",
"sys_domain": "global",
"correlation_id": "",
"description": "User can't access email on mail.company.com.\n\t\t",
"state": "7",
"sys_id": "9c573169c611228700193229fff72400",
"hold_reason": "",
"sys_updated_on": "2018-02-25 20:16:07",
"made_sla": "false",
"sys_updated_by": "admin",
"priority": "1",
"approval_set": "",
"activity_due": "",
"notify": "1",
"group_list": "",
"reassignment_count": "1",
"knowledge": "false",
"work_end": "",
"expected_start": "",
"delivery_plan": "",
"resolved_at": "2018-02-25 19:56:12",
"sys_created_on": "2016-06-26 18:24:13",
"business_stc": "1892781",
"due_date": "",
"assignment_group": "d625dccec0a8016700a222a0f7900d06",
"approval_history": "",
"child_incidents": "",
"business_service": "",
"opened_at": "2017-11-25 23:09:51",
"caused_by": "",
"order": "",
"__status": "success",
"number": "INC0000001",
"contact_type": "",
"close_notes": "Closed before close notes were made mandatory\n\t\t",
"sys_class_name": "incident",
"business_duration": "1970-01-22 21:46:21",
"incident_state": "7",
"comments_and_work_notes": "",
"work_notes_list": "",
"reopen_count": "",
"location": "1083361cc611227501b682158cabf646",
"sys_created_by": "pat",
"parent": "",
"rfc": "",
"follow_up": "",
"user_input": ""
},

These fields are for single incident, like this i have received data for all the incidents available in my servicenow instance.
As you can see, all these are falling under a single field "record"

here is my configuration file.

input {
http_poller {
urls => {
url => "https://dev.service-now.com/incident_list.do?JSONv2&incident_list.do?JSONv2&display_value=True&sysparm_exclude_reference_link=True&sysparm_fields=resolved_by%2Cstate%2Ccmdb_ci%2Cpriority%2Ccaller_id%2Cshort_description%2Cassignment_group%2Cassigned_to%2Cseverity%2Clocation%2Cnumber&sysparm_limit=1sysparm_view=json_view"
}
request_timeout => 60
proxy => { host => "10.1.1.2" port => "8181" scheme => "http"}
user => "admin"
password => "********"
schedule => { cron => "* * * * *"}
codec => "json"
metadata_target => "http_poller_metadata"
}
}
filter
{
json
{
source => "records"
}
}
output {
elasticsearch {
hosts => ["10.1.1.4:9200"]
index => "servicenowinc"
}
stdout {
codec => rubydebug
}
}

10

The records field is an array of objects. Kibana simply doesn't handle arrays of objects that well. Perhaps you should use the split filter on the records field and get one event per record? Then the fields will be available as I think you expect.

11

@magnusbaeck I have not used split filter before, is there any documentation available for this, or can you gimme me some example config file, that will help me a lot.

Thanks
Gautham

12

https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html

1 Like
13

Thank you very much @magnusbaeck for you guidance.....i'll go through docs.

Thanks
Gautham

14

Bingo......It worked as i expected @magnusbaeck .. Thank you very much....

Thanks
Gautham

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.