Feed Elasticsearch directly from a web service URL?


(Marius Mathisen) #1

Hello,

Can I somehow feed Elasticsearch with the json file my web service produce? Like : nameofserver/api/thejsonfile ?

Or do I have to download it and feed it manually?


(Mark Walkom) #2

You cannot do it directly, no.

Something like Logstash will do it, of you can do a manual curl download and then upload to ES.


(Marius Mathisen) #3

OK. Thank you. :slight_smile: Will have to look into Logstash then.


(Marius Mathisen) #4

Are there any spesific formatting I need to use on the JSON file? When I download the JSON file locally and try to feed it to ES I seem to do something wrong.

If I use : curl XPOST localhost:9200/server/server/_bulk?pretty --databinary @Server.json I get the no request added error.

If I use: curl XPOST localhost:9200/server/server?pretty --databinary @Server.json it indexes it only as one object.

Here is a sample of my Server.json file, some info is changed for security reasons:

{
"machineID":1,
"isVirtualMachine":true,
"name":"Server1",
"lastActivity":"2015-06-27T06:47:03",
"domain":"NO",
"deviceOS":"Microsoft Windows Server 2012 Standard",
"lastBoot":"2015-06-11T03:11:31",
"patchePlan":"Auto",
"owner":1
},
{
"machineID":2,
"isVirtualMachine":true,
"name":"Server2",
"lastActivity":"2015-06-27T06:50:57",
"domain":"NO",
"deviceOS":"Microsoft Windows Server 2008 R2 Standard",
"lastBoot":"2015-06-10T22:14:27",
"patchePlan":"Auto",
"owner":1
},
{
"machineID":3,
"isVirtualMachine":false,
"name":"Server3",
"lastActivity":"2015-06-27T06:47:25",
"domain":"NO",
"deviceOS":"Microsoft Windows Server 2012 R2 Standard",
"lastBoot":"2015-06-11T04:11:37",
"patchePlan":"Auto",
"owner":1
},
{
"machineID":4,
"isVirtualMachine":true,
"name":"Server4",
"lastActivity":"2015-06-27T07:01:51",
"domain":"NO",
"deviceOS":"Microsoft Windows Server 2012 R2 Standard",
"lastBoot":"2015-06-11T03:12:31",
"patchePlan":"Auto",
"owner":1
},
{
"machineID":5,
"isVirtualMachine":true,
"name":"Server5",
"lastActivity":"2015-06-27T06:55:27",
"domain":"NO",
"deviceOS":"Microsoft Windows Server 2008 R2 Enterprise",
"lastBoot":"2015-06-11T17:43:17",
"patchePlan":"Manual",
"owner":1
}

Is this format in JSON wrong? If so, how should it be?


(David Pilato) #5

If you use bulk you need to respect bulk format:
Header
Payload
Header
Payload
..,

See https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html


(Marius Mathisen) #6

So basically the same format as in the accounts.json file in the guide?

{"index":{"_id":"1"}}
{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}

Is this something we have to do manually, or do we have a script/formatter to automatic format json-files correctly with Header, Payload, Header, Payload?


(David Pilato) #7

Well it depends on which client you are using.
In Java, you have some classes for example to simplify all that.

As @warkolm suggested, you could look at logstash. May be this plugin might help you: https://github.com/logstash-plugins/logstash-input-http-poller


(Marius Mathisen) #8

Yeah, seems like that´s the way to go. :smile:

Thank you. :slight_smile:


(system) #9