Input JSON file to elasticsearch via logstash


(Guta Saputra) #1

hi, im facing an issue with logstash while inserting json data to elasticsearch.

here is my logstash config file

input
{
file
{
codec => multiline
{
pattern => '^{'
negate => true
what => previous
}

    path => ["/usr/local/Cellar/logstash/5.1.1/test_payment.json"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"
}

}

filter
{
mutate
{
replace => [ "message", "%{message}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/
{
json { source => message }
}

}

output {
elasticsearch {
hosts => ["localhost:9200"]
user => elastic
password => techno2013
index => "test_payment"
}
stdout { codec => rubydebug }
}

then, this is my json file

{
"user_id":0,
"cart_id":"81746",
"error_status":"",
"error_message":"",
"data":{
"ord_id":218308,
"cart":{
"cart_id":"81746",
"session_id":"7afaaf7fac9bb934de4c6ecb2567461a6178495e",
"ip_address":"180.254.65.91",
"usr_id":"0",
"created_date":"2017-01-19 17:29:20",
"updated_date":"2017-01-19 18:08:35"
},
"cart_data":{
"coupon_code":"",
"ord_email":"shanti.swaragama@gmail.com",
"ord_firstname":"adhe",
"ord_lastname":"shanti",
"ord_address_line1":"Radio Swaragama, Komplek Bulaksumur Blok H No 5",
"ord_address_line2":"",
"country_id":"1",
"prov_id":"9",
"city_id":"45",
"ord_postcode":"55281",
"ord_phone":"082227369636",
"dif_addr":null,
"rec_name":"",
"rec_address_line1":"",
"rec_address_line2":"",
"rec_country":"1",
"rec_prov":"10",
"rec_city":"0",
"rec_postcode":"",
"rec_phone":"",
"wrap_type":"1",
"psncard_type":null,
"psncard_message":null,
"currency":"IDR",
"currency_value":1,
"language":"en",
"is_mobile":"0",
"ord_shp_fee":18000,
"ord_shp_mode":"reg",
"ord_py_mode":"transfer",
"newtoken":"",
"cc_credit_option":null,
"cc_saved_bank":"",
"cc_saved_period":"",
"cc_savecc":null,
"cc_saved_no":null,
"cc_savedccv":null,
"cc_new_bank":"",
"cc_new_period":"",
"cc_newcc1":"",
"cc_newcc2":"",
"cc_newcc3":"",
"cc_newcc4":"",
"cc_newexp1":"",
"cc_newexp2":"",
"cc_newccv":""
},
"cart_detail":[
{
"cadet_id":"179977",
"cart_id":"81746",
"stk_id":"5240",
"cadet_qty":"1",
"cadet_data":"[]",
"pd_id":"4890",
"pd_name":"Denim Odette",
"pd_shortdesc":"Cut-out Shoulders Top",
"pd_price":"249000.00",
"pd_img_primary":"DenimOdette_260916_04.jpg",
"color_id":"145",
"pd_special":"0",
"pd_sale":"0",
"pd_weight":"0.15",
"color_name":"Blue Jeans",
"color_thumbnail":"Blue-Jeans.jpg",
"size_id":"40",
"size_name":"Size L",
"size_shortname":"L",
"cat_name":"Tops",
"cat_id":"6",
"sub_id":"16",
"sub_name":"Short Sleeved",
"str_coll_id":"2"
}
],
"order_summary":{
"err_status":false,
"err_msg":"",
"subtotal":249000,
"promotion":0,
"wrapping":0,
"shipping":18000,
"total":267000,
"total_weight":0.15,
"coupon_code":"",
"coupon_type":"",
"boledisc":0,
"arrboledisc":[

     ],
     "bolefreeship":249000,
     "boleinstallment":249000,
     "shipping_code":"reg"
  }

}
}

when i try running logstash -f logstash.conf
all data successfully insert to elasticsearch, but the problem is
i only got the structure of json data, the value of its json not inserted to elasticsearch.

for example i have this json file
{"foo":"foo value", "bar": "bar value"}
{"hello":"world", "goodnight": "moon"}

when i insert to elasticsearch, i only got foo, and bar, im not get foo value and bar value.

the question is, how can i inserted object and value to elasticsearch.
im confused.
please help me.


Indexing JSON files from a local directory to elastic
(Magnus Bäck) #2

Your file input configuration is incorrect. You can't specify two codecs (multiline and json). In this case you'll want to pick the multiline codec and then you can add a json filter to parse the resulting JSON string.

when i insert to elasticsearch, i only got foo, and bar, im not get foo value and bar value.

Please copy/paste the resulting document so it's more clear exactly what you mean.


(Guta Saputra) #3

the result is same sir.
either using json or multiline.

i only got the object data, not the value.

what i want is like this :
"user_id":021
"order_id": 988982
"email_address": "gutasaputra@gmail.com"

instead of :
"order_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}

"email_address": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}


(Magnus Bäck) #4

It seems like you're looking at the mapping of the index, not the documents in the index.


(Guta Saputra) #5

ya, its not there.
no value of myobject inserted to elasticsearch.

how can i insert all of my objects and values inside of my json file to elasticsearch via logstash ?


(Magnus Bäck) #6

Judging by what you posted earlier you're not looking at the documents that Logstash inserts. You are looking at the mappings of the index, i.e. the collection of fields found in the documents (basically). Please show the actual contents of an example document.


(Guta Saputra) #7

this is my json file that i want to insert to elasticsearch via logstash sir

{
"user_id":0,
"cart_id":"81746",
"error_status":"",
"error_message":"",
"data":{
"ord_id":218308,
"cart":{
"cart_id":"81746",
"session_id":"7afaaf7fac9bb934de4c6ecb2567461a6178495e",
"ip_address":"180.254.65.91",
"usr_id":"0",
"created_date":"2017-01-19 17:29:20",
"updated_date":"2017-01-19 18:08:35"
},
"cart_data":{
"coupon_code":"",
"ord_email":"shanti.swaragama@gmail.com",
"ord_firstname":"adhe",
"ord_lastname":"shanti",
"ord_address_line1":"Radio Swaragama, Komplek Bulaksumur Blok H No 5",
"ord_address_line2":"",
"country_id":"1",
"prov_id":"9",
"city_id":"45",
"ord_postcode":"55281",
"ord_phone":"082227369636",
"dif_addr":null,
"rec_name":"",
"rec_address_line1":"",
"rec_address_line2":"",
"rec_country":"1",
"rec_prov":"10",
"rec_city":"0",
"rec_postcode":"",
"rec_phone":"",
"wrap_type":"1",
"psncard_type":null,
"psncard_message":null,
"currency":"IDR",
"currency_value":1,
"language":"en",
"is_mobile":"0",
"ord_shp_fee":18000,
"ord_shp_mode":"reg",
"ord_py_mode":"transfer",
"newtoken":"",
"cc_credit_option":null,
"cc_saved_bank":"",
"cc_saved_period":"",
"cc_savecc":null,
"cc_saved_no":null,
"cc_savedccv":null,
"cc_new_bank":"",
"cc_new_period":"",
"cc_newcc1":"",
"cc_newcc2":"",
"cc_newcc3":"",
"cc_newcc4":"",
"cc_newexp1":"",
"cc_newexp2":"",
"cc_newccv":""
},
"cart_detail":[
{
"cadet_id":"179977",
"cart_id":"81746",
"stk_id":"5240",
"cadet_qty":"1",
"cadet_data":"[]",
"pd_id":"4890",
"pd_name":"Denim Odette",
"pd_shortdesc":"Cut-out Shoulders Top",
"pd_price":"249000.00",
"pd_img_primary":"DenimOdette_260916_04.jpg",
"color_id":"145",
"pd_special":"0",
"pd_sale":"0",
"pd_weight":"0.15",
"color_name":"Blue Jeans",
"color_thumbnail":"Blue-Jeans.jpg",
"size_id":"40",
"size_name":"Size L",
"size_shortname":"L",
"cat_name":"Tops",
"cat_id":"6",
"sub_id":"16",
"sub_name":"Short Sleeved",
"str_coll_id":"2"
}
],
"order_summary":{
"err_status":false,
"err_msg":"",
"subtotal":249000,
"promotion":0,
"wrapping":0,
"shipping":18000,
"total":267000,
"total_weight":0.15,
"coupon_code":"",
"coupon_type":"",
"boledisc":0,
"arrboledisc":[

     ],
     "bolefreeship":249000,
     "boleinstallment":249000,
     "shipping_code":"reg"
  }

}
}


(Magnus Bäck) #8

Yes, but what does that document look like in Elasticsearch?


(Guta Saputra) #9

do yo mean this ?


(Magnus Bäck) #10

No! Use Kibana or the search API to look at the documents.


(Guta Saputra) #11

thats from kibana sir.

here is the full screenshot.


(Magnus Bäck) #12

That REST endpoint doesn't return any documents. Use the Discover tab instead or GET /test_payment/_search if you feel you need to use raw REST calls.


(Guta Saputra) #13

ohya, i got it.
i can see the document right now.

but i cant access it via kibana .
when i try to discover data using kibana, it return No results found.

oh should i create new threat for this issue?

thanks for help


(Magnus Bäck) #14

Things to check:

  • Make sure the time span chosen in the Discover tab is correct and encompasses the @timestamp field (which is in UTC).
  • Make sure the index pattern used matches the test_payment index where your data seems to be stored.

(system) #15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.