Automatic fetch of data to index


(powgri) #1

I have a RESTFul webservice and a /store/list endpoint which responds
back with the following json:
How can I setup an elasticsearch instance to automatically pull data
in intervals from this endpoint based on the HTTP response headers set
by the service?

{
"stores": {
"@rows": "15",
"@total": "2239",
"store": [
{
"id": 8095,
"link": [
{
"@href": "http://dv1-cbsp4.abcd.int:8080/defg/
1/store/8095",
"@rel": "related",
"@title": "store"
},
{
"@href": "http://dv1-cbsp4.abcd.int:8080/defg/
1/promotion/store/8095/list",
"@rel": "related",
"@title": "store.promotion.list"
},
{
"@href": "http://dv1-cbsp4.abcd.int:8080/defg/
1/campaign/store/8095/list",
"@rel": "related",
"@title": "store.campaign.list"
},
{
"@href": "http://dv1-cbsp4.abcd.int:8080/defg/
1/store/8095/keyword/list",
"@rel": "related",
"@title": "store.keyword.list"
}
],
"name": "Abebooks",
"reward": {
"calculation": "Percentage",
"currency": "USD",
"amount": 5,
"display": "Percentage"
},
"totalReward": {
"calculation": "Percentage",
"currency": "USD",
"amount": 5,
"display": "Percentage"
},
"desc": "abebooks"
},
...
...


(vineeth mohan) #2

Create a river based plugin.

Thanks
Vineeth

On Fri, Sep 2, 2011 at 1:33 PM, JJ powgri@gmail.com wrote:

I have a RESTFul webservice and a /store/list endpoint which responds
back with the following json:
How can I setup an elasticsearch instance to automatically pull data
in intervals from this endpoint based on the HTTP response headers set
by the service?

{
"stores": {
"@rows": "15",
"@total": "2239",
"store": [
{
"id": 8095,
"link": [
{
"@href": "http://dv1-cbsp4.abcd.int:8080/defg/
1/store/8095",
"@rel": "related",
"@title": "store"
},
{
"@href": "http://dv1-cbsp4.abcd.int:8080/defg/
1/promotion/store/8095/list",
"@rel": "related",
"@title": "store.promotion.list"
},
{
"@href": "http://dv1-cbsp4.abcd.int:8080/defg/
1/campaign/store/8095/list",
"@rel": "related",
"@title": "store.campaign.list"
},
{
"@href": "http://dv1-cbsp4.abcd.int:8080/defg/
1/store/8095/keyword/list",
"@rel": "related",
"@title": "store.keyword.list"
}
],
"name": "Abebooks",
"reward": {
"calculation": "Percentage",
"currency": "USD",
"amount": 5,
"display": "Percentage"
},
"totalReward": {
"calculation": "Percentage",
"currency": "USD",
"amount": 5,
"display": "Percentage"
},
"desc": "abebooks"
},
...
...


(Shay Banon) #3

You can't, you need ot do it yourself. A river (as suggested in another
thread) is an option, but it might get too complex for you, a simple process
(in your fav lang) that does it is good enough.

On Fri, Sep 2, 2011 at 11:03 AM, JJ powgri@gmail.com wrote:

I have a RESTFul webservice and a /store/list endpoint which responds
back with the following json:
How can I setup an elasticsearch instance to automatically pull data
in intervals from this endpoint based on the HTTP response headers set
by the service?

{
"stores": {
"@rows": "15",
"@total": "2239",
"store": [
{
"id": 8095,
"link": [
{
"@href": "http://dv1-cbsp4.abcd.int:8080/defg/
1/store/8095",
"@rel": "related",
"@title": "store"
},
{
"@href": "http://dv1-cbsp4.abcd.int:8080/defg/
1/promotion/store/8095/list",
"@rel": "related",
"@title": "store.promotion.list"
},
{
"@href": "http://dv1-cbsp4.abcd.int:8080/defg/
1/campaign/store/8095/list",
"@rel": "related",
"@title": "store.campaign.list"
},
{
"@href": "http://dv1-cbsp4.abcd.int:8080/defg/
1/store/8095/keyword/list",
"@rel": "related",
"@title": "store.keyword.list"
}
],
"name": "Abebooks",
"reward": {
"calculation": "Percentage",
"currency": "USD",
"amount": 5,
"display": "Percentage"
},
"totalReward": {
"calculation": "Percentage",
"currency": "USD",
"amount": 5,
"display": "Percentage"
},
"desc": "abebooks"
},
...
...


(system) #4