Http_poller fails on subsets

[Callahan]

Hi,

So I'm trying to pull a json response from a site. The response has now changed as the API has been upgraded. Whereas before, the tickets would be returned as the sample shown below:

[
    {
        "account_id": 145344,
        "category": null,
        "cc_email": {
            "cc_emails": [],
            "fwd_emails": [],
            "reply_cc": [],
            "tkt_cc": []
        },
        "created_at": "2017-10-10T11:13:07+01:00",
        "deleted": false,
        "department_id_value": null,
        "display_id": 72,
        "due_by": "2017-10-12T11:13:07+01:00",
        "email_config_id": null,
        "frDueBy": "2017-10-11T08:43:07+01:00",
        "fr_escalated": true,
        "group_id": null,
        "id": 7000975397,
        "impact": 2,
        "import_id": null,
        "isescalated": false,
        "item_category": null,
        "owner_id": null,
        "priority": 1,
        "requester_id": 7000291444,
        "responder_id": null,
        "source": 4,
        "spam": false,
        "status": 2,
        "sub_category": null,
        "subject": "PC is broken",
        "ticket_type": "Incident",
        "to_email": null,
        "updated_at": "2017-10-10T11:13:07+01:00",
        "urgency": 1,
        "description": "Problem description",
        "description_html": "<div>Problem description<br>\n</div>",
        "status_name": "Open",
        "requester_status_name": "Being Processed",
        "priority_name": "Low",
        "source_name": "Chat",
        "requester_name": "Joe Bloggs",
        "responder_name": "No Agent",
        "to_emails": null,
        "department_name": "IT",
        "assoc_problem_id": null,
        "assoc_change_id": null,
        "assoc_change_cause_id": null,
        "assoc_asset_id": null,
        "urgency_name": "Low",
        "impact_name": "Medium",
        "custom_field": {}
    },
]

Whereas now the results returned are contained within a single subset as shown below:

{
    "tickets": [
        {
            "cc_emails": [],
            "fwd_emails": [],
            "reply_cc_emails": [],
            "fr_escalated": true,
            "spam": false,
            "email_config_id": null,
            "group_id": null,
            "priority": 1,
            "requester_id": 7000291444,
            "responder_id": null,
            "source": 4,
            "status": 2,
            "subject": "Broken PC",
            "to_emails": null,
            "department_id": 7000150395,
            "id": 72,
            "type": "Incident",
            "due_by": "2017-10-12T10:13:07Z",
            "fr_due_by": "2017-10-11T07:43:07Z",
            "is_escalated": false,
            "description": "<div>Problem description<br>\n</div>",
            "description_text": "Problem description",
            "category": null,
            "sub_category": null,
            "item_category": null,
            "custom_fields": {},
            "created_at": "2017-10-10T10:13:07Z",
            "updated_at": "2017-10-10T10:13:07Z",
            "deleted": false
        }
]
}

The end result is that when http_poller pulls this information into Logstash for processing, with V2 of the API, it only sees one line. That line is the combination of all jobs held under the "tickets" subset. The older V1 API would pull in the jobs line by line, i.e. line 1 contained job 1 details, line 2 contained line 2 details, etc. The older version would then see the list of jobs as separate entires.

Is there any clever way I can tell http_poller to expand the only line it pulls back?

[magnusbaeck (Magnus Bäck)]

What are these "lines" you're talking about? All entities are singular in your examples so I don't understand the problem. Instead of describing the problem in prose perhaps you can show us what Logstash currently produced and what you would like it to produce instead. Express your example in JSON.

[Callahan]

Hi Magnus,

The V2 API was too much hassle to work with so I've fallen back to their more stable V1. I'm still having issues with the http poller however. What was outputting, now seems not to be.
There are no errors in the logs, I just don't see the index ever appears in Elastic.

My code below.

input{
    http_poller {
	urls => {
			"Tickets" => {
                url => "http://192.168.0.1/report/12345.json"
				method => "get"
                headers => {
                "Accept" => "application/json"
					}
                auth => {
				user => "my_username"
				password => "my_password"
					}
				}
				}
                connect_timeout => 15
				automatic_retries => 5
				request_timeout => 60
				socket_timeout => 60
				schedule => { "every" => "10m"}
                codec           => "json"
                metadata_target => "http_poller_metadata"
                type            => "gb_tickets"
            }
}

output {
    if [type] == "gb_tickets" {
    elasticsearch {
	index => "tickets-%{+YYYY.MM.dd}"
	hosts => ["10.0.1.2:9200"]
		}
      }
    }

Looking at the URL in Postman or Firefox, I see what I expect to see, a bunch of tickets in JSON format. I start the service using this config and I get it gets passed (as well as passing the logstash -t -f check), but nothing turns up in Elastic (at least according to Kibana). Have tested Kibana with other new simple indexes and it works without issue.
Not sure what I've missed here....

[Callahan]

Scratch that. Turns out Logstash is case sensitive. I had uppercase letters in the index name and the "type" fields (not shown in the sample code above. Once removed, everything worked.

Painful hour's lesson....