Input JSON file to elasticsearch via logstash

Elastic Stack Logstash
1

hi, im facing an issue with logstash while inserting json data to elasticsearch.

here is my logstash config file

input
{
file
{
codec => multiline
{
pattern => '^{'
negate => true
what => previous
}

    path => ["/usr/local/Cellar/logstash/5.1.1/test_payment.json"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"
}

}

filter
{
mutate
{
replace => [ "message", "%{message}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/
{
json { source => message }
}

}

output {
elasticsearch {
hosts => ["localhost:9200"]
user => elastic
password => techno2013
index => "test_payment"
}
stdout { codec => rubydebug }
}

then, this is my json file

{
"user_id":0,
"cart_id":"81746",
"error_status":"",
"error_message":"",
"data":{
"ord_id":218308,
"cart":{
"cart_id":"81746",
"session_id":"7afaaf7fac9bb934de4c6ecb2567461a6178495e",
"ip_address":"180.254.65.91",
"usr_id":"0",
"created_date":"2017-01-19 17:29:20",
"updated_date":"2017-01-19 18:08:35"
},
"cart_data":{
"coupon_code":"",
"ord_email":"shanti.swaragama@gmail.com",
"ord_firstname":"adhe",
"ord_lastname":"shanti",
"ord_address_line1":"Radio Swaragama, Komplek Bulaksumur Blok H No 5",
"ord_address_line2":"",
"country_id":"1",
"prov_id":"9",
"city_id":"45",
"ord_postcode":"55281",
"ord_phone":"082227369636",
"dif_addr":null,
"rec_name":"",
"rec_address_line1":"",
"rec_address_line2":"",
"rec_country":"1",
"rec_prov":"10",
"rec_city":"0",
"rec_postcode":"",
"rec_phone":"",
"wrap_type":"1",
"psncard_type":null,
"psncard_message":null,
"currency":"IDR",
"currency_value":1,
"language":"en",
"is_mobile":"0",
"ord_shp_fee":18000,
"ord_shp_mode":"reg",
"ord_py_mode":"transfer",
"newtoken":"",
"cc_credit_option":null,
"cc_saved_bank":"",
"cc_saved_period":"",
"cc_savecc":null,
"cc_saved_no":null,
"cc_savedccv":null,
"cc_new_bank":"",
"cc_new_period":"",
"cc_newcc1":"",
"cc_newcc2":"",
"cc_newcc3":"",
"cc_newcc4":"",
"cc_newexp1":"",
"cc_newexp2":"",
"cc_newccv":""
},
"cart_detail":[
{
"cadet_id":"179977",
"cart_id":"81746",
"stk_id":"5240",
"cadet_qty":"1",
"cadet_data":"",
"pd_id":"4890",
"pd_name":"Denim Odette",
"pd_shortdesc":"Cut-out Shoulders Top",
"pd_price":"249000.00",
"pd_img_primary":"DenimOdette_260916_04.jpg",
"color_id":"145",
"pd_special":"0",
"pd_sale":"0",
"pd_weight":"0.15",
"color_name":"Blue Jeans",
"color_thumbnail":"Blue-Jeans.jpg",
"size_id":"40",
"size_name":"Size L",
"size_shortname":"L",
"cat_name":"Tops",
"cat_id":"6",
"sub_id":"16",
"sub_name":"Short Sleeved",
"str_coll_id":"2"
}
],
"order_summary":{
"err_status":false,
"err_msg":"",
"subtotal":249000,
"promotion":0,
"wrapping":0,
"shipping":18000,
"total":267000,
"total_weight":0.15,
"coupon_code":"",
"coupon_type":"",
"boledisc":0,
"arrboledisc":[

     ],
     "bolefreeship":249000,
     "boleinstallment":249000,
     "shipping_code":"reg"
  }

}
}

when i try running logstash -f logstash.conf
all data successfully insert to elasticsearch, but the problem is
i only got the structure of json data, the value of its json not inserted to elasticsearch.

for example i have this json file
{"foo":"foo value", "bar": "bar value"}
{"hello":"world", "goodnight": "moon"}

when i insert to elasticsearch, i only got foo, and bar, im not get foo value and bar value.

the question is, how can i inserted object and value to elasticsearch.
im confused.
please help me.

2

Your file input configuration is incorrect. You can't specify two codecs (multiline and json). In this case you'll want to pick the multiline codec and then you can add a json filter to parse the resulting JSON string.

when i insert to elasticsearch, i only got foo, and bar, im not get foo value and bar value.

Please copy/paste the resulting document so it's more clear exactly what you mean.

3

the result is same sir.
either using json or multiline.

i only got the object data, not the value.

Screen Shot 2017-01-23 at 1.54.13 PM.png880×564 46.1 KB

what i want is like this :
"user_id":021
"order_id": 988982
"email_address": "gutasaputra@gmail.com"

instead of :
"order_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}

"email_address": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}

4

It seems like you're looking at the mapping of the index, not the documents in the index.

5

ya, its not there.
no value of myobject inserted to elasticsearch.

how can i insert all of my objects and values inside of my json file to elasticsearch via logstash ?

6

Judging by what you posted earlier you're not looking at the documents that Logstash inserts. You are looking at the mappings of the index, i.e. the collection of fields found in the documents (basically). Please show the actual contents of an example document.

7

this is my json file that i want to insert to elasticsearch via logstash sir

{
"user_id":0,
"cart_id":"81746",
"error_status":"",
"error_message":"",
"data":{
"ord_id":218308,
"cart":{
"cart_id":"81746",
"session_id":"7afaaf7fac9bb934de4c6ecb2567461a6178495e",
"ip_address":"180.254.65.91",
"usr_id":"0",
"created_date":"2017-01-19 17:29:20",
"updated_date":"2017-01-19 18:08:35"
},
"cart_data":{
"coupon_code":"",
"ord_email":"shanti.swaragama@gmail.com",
"ord_firstname":"adhe",
"ord_lastname":"shanti",
"ord_address_line1":"Radio Swaragama, Komplek Bulaksumur Blok H No 5",
"ord_address_line2":"",
"country_id":"1",
"prov_id":"9",
"city_id":"45",
"ord_postcode":"55281",
"ord_phone":"082227369636",
"dif_addr":null,
"rec_name":"",
"rec_address_line1":"",
"rec_address_line2":"",
"rec_country":"1",
"rec_prov":"10",
"rec_city":"0",
"rec_postcode":"",
"rec_phone":"",
"wrap_type":"1",
"psncard_type":null,
"psncard_message":null,
"currency":"IDR",
"currency_value":1,
"language":"en",
"is_mobile":"0",
"ord_shp_fee":18000,
"ord_shp_mode":"reg",
"ord_py_mode":"transfer",
"newtoken":"",
"cc_credit_option":null,
"cc_saved_bank":"",
"cc_saved_period":"",
"cc_savecc":null,
"cc_saved_no":null,
"cc_savedccv":null,
"cc_new_bank":"",
"cc_new_period":"",
"cc_newcc1":"",
"cc_newcc2":"",
"cc_newcc3":"",
"cc_newcc4":"",
"cc_newexp1":"",
"cc_newexp2":"",
"cc_newccv":""
},
"cart_detail":[
{
"cadet_id":"179977",
"cart_id":"81746",
"stk_id":"5240",
"cadet_qty":"1",
"cadet_data":"[]",
"pd_id":"4890",
"pd_name":"Denim Odette",
"pd_shortdesc":"Cut-out Shoulders Top",
"pd_price":"249000.00",
"pd_img_primary":"DenimOdette_260916_04.jpg",
"color_id":"145",
"pd_special":"0",
"pd_sale":"0",
"pd_weight":"0.15",
"color_name":"Blue Jeans",
"color_thumbnail":"Blue-Jeans.jpg",
"size_id":"40",
"size_name":"Size L",
"size_shortname":"L",
"cat_name":"Tops",
"cat_id":"6",
"sub_id":"16",
"sub_name":"Short Sleeved",
"str_coll_id":"2"
}
],
"order_summary":{
"err_status":false,
"err_msg":"",
"subtotal":249000,
"promotion":0,
"wrapping":0,
"shipping":18000,
"total":267000,
"total_weight":0.15,
"coupon_code":"",
"coupon_type":"",
"boledisc":0,
"arrboledisc":[

     ],
     "bolefreeship":249000,
     "boleinstallment":249000,
     "shipping_code":"reg"
  }

}
}

8

Yes, but what does that document look like in Elasticsearch?

9

do yo mean this ?

Screen Shot 2017-01-23 at 2.32.22 PM.png1250×764 72.1 KB

10

No! Use Kibana or the search API to look at the documents.

11

thats from kibana sir.

here is the full screenshot.

Screen Shot 2017-01-23 at 4.18.39 PM.png2560×1264 422 KB

12

That REST endpoint doesn't return any documents. Use the Discover tab instead or GET /test_payment/_search if you feel you need to use raw REST calls.

13

ohya, i got it.
i can see the document right now.

but i cant access it via kibana .
when i try to discover data using kibana, it return No results found.

oh should i create new threat for this issue?

thanks for help

14

Things to check:

  • Make sure the time span chosen in the Discover tab is correct and encompasses the @timestamp field (which is in UTC).
  • Make sure the index pattern used matches the test_payment index where your data seems to be stored.
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.