Can't get text on START_OBJECT

Mark_Bassett · January 4, 2019, 1:36am

Hi guys, I have some really bad source data that is giving me a mapping conflict when I try to ingest because sometimes it is a string value, and other times it is an array.

I've been fighting around with this for a while now, have tried many different things but nothing seems to be working.

{"m":"log_rgs","rgsResponseSummay":{"PrizeOutcome":[{"multiplier":"1","name":"BaseGame.Bonus","pay":"0","stage":"","totalPay":"0","type":"Pattern"},{"multiplier":"1","name":"BaseGame.Total","pay":"0","stage":"","totalPay":"0","type":"","Prize":{"betMultiplier":"1","multiplier":"1","name":"Total","pay":"0","payName":"","position":"0","symbolCount":"0","totalPay":"0","ways":"0"}},{"multiplier":"1","name":"BaseGame.Jackpot","pay":"0","stage":"","totalPay":"0","type":"Pattern"},{"multiplier":"1","name":"BaseGame.Lines","pay":"0","stage":"","totalPay":"0","type":"Pattern"},{"multiplier":"1","name":"SmartSound.Prize","pay":"0","stage":"","totalPay":"0","type":"Pattern"},{"multiplier":"1","name":"Game.Total","pay":"0","stage":"","totalPay":"0","type":"","Prize":{"betMultiplier":"1","multiplier":"1","name":"Total","pay":"0","payName":"","position":"0","symbolCount":"0","totalPay":"0","ways":"0"}}],"OutcomeDetail":{"TransactionId":"A10M@-15439533032755","Stage":"BaseGame","NextStage":"BaseGame","Balance":"5918261","GameStatus":"Start","Settled":"300","Pending":"0","Payout":"0"},"PopulationOutcome":[{"name":"CountryPicker.Picks","stage":"CountryPicker","Entry":[{"name":"L0C0R0","stripIndex":"0","Cell":"trigger,SubPicker1"},{"name":"L0C1R0","stripIndex":"1","Cell":"trigger,SubPicker2"},{"name":"L0C2R0","stripIndex":"2","Cell":"trigger,SubPicker3"}]},{"name":"OneToRule","stage":"OneToRule","Entry":[{"name":"L0C0R0","stripIndex":"0","Cell":"trigger,Nope"},{"name":"L0C1R0","stripIndex":"0","Cell":"trigger,Nope"},{"name":"L0C2R0","stripIndex":"0","Cell":"trigger,Nope"},{"name":"L0C3R0","stripIndex":"0","Cell":"trigger,Nope"},{"name":"L0C4R0","stripIndex":"0","Cell":"trigger,Nope"},{"name":"L0C5R0","stripIndex":"0","Cell":"trigger,Nope"},{"name":"L0C6R0","stripIndex":"0","Cell":"trigger,Nope"},{"name":"L0C7R0","stripIndex":"0","Cell":"trigger,Nope"},{"name":"L0C8R0","stripIndex":"0","Cell":"trigger,Nope"}]},{"name":"BaseGame.JackpotReels","stage":"BaseGame","Entry":[{"name":"Reel0","stripIndex":"163","Cell":["s08","s09","s10","s02"]},{"name":"Reel1","stripIndex":"193","Cell":["s02","s07","s10","j01"]},{"name":"Reel2","stripIndex":"573","Cell":["s02","s08","b01","s09"]},{"name":"Reel3","stripIndex":"48","Cell":["s07","s08","s03","s06"]},{"name":"Reel4","stripIndex":"91","Cell":["s07","s08","s04","s09"]}]},{"name":"BonusPicker.Picks","stage":"BonusPicker","Entry":[{"name":"L0C0R0","stripIndex":"0","Cell":"trigger,CountryPicker"},{"name":"L0C1R0","stripIndex":"1","Cell":"spins,5,trigger,FreeSpin"}]},{"name":"BaseGame.Reels","stage":"BaseGame","Entry":[{"name":"Reel0","stripIndex":"163","Cell":["s08","s09","s10","s02"]},{"name":"Reel1","stripIndex":"193","Cell":["s02","s07","s10","s20"]},{"name":"Reel2","stripIndex":"573","Cell":["s02","s08","b01","s09"]},{"name":"Reel3","stripIndex":"48","Cell":["s07","s08","s03","s06"]},{"name":"Reel4","stripIndex":"91","Cell":["s07","s08","s04","s09"]}]},{"name":"FreeSpin.Reels","stage":"FreeSpin","Entry":[{"name":"Reel0","stripIndex":"1","Cell":["s01","s01","s06","s10"]},{"name":"Reel1","stripIndex":"1","Cell":["s04","s07","s09","s02"]},{"name":"Reel2","stripIndex":"1","Cell":["w01","w01","w01","s06"]},{"name":"Reel3","stripIndex":"1","Cell":["b01","s05","s07","w01"]},{"name":"Reel4","stripIndex":"1","Cell":["s06","s03","s03","s08"]}]}]}}

my filter looks like (i've tried this many different ways and having no luck)

filter {
  mutate {
    id => "cell->CellArray"
    gsub => [ "message", "Cell:\"\[", 'CellArray:"[' ]
  }
  json { source => "message" }
}

I thought if i could rename the Cell to CellArray when its an array then i would get rid of the elasticsearch mapping issue.

Jan  4 00:55:52 logstash02 logstash: [2019-01-04T00:55:52,751][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"test-rgs-spin-2019-01-01", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x7d4277f6>], :response=>{"index"=>{"_index"=>"test-rgs-spin-2019-01-01", "_type"=>"doc", "_id"=>"MYpbFmgBNTQoEHRS-DU7", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [rgsResponseSummay.PopulationOutcome.Entry.Cell] of type [text]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:1126"}}}}}

Mark_Bassett · January 4, 2019, 7:13pm

The gsub compiles but it is not changing the source data, i still see events with Cell not CellArray field.

Mark_Bassett · January 11, 2019, 7:07pm

Bump... anyone? Bueller?

Chris_Lyons · January 11, 2019, 8:53pm

Check your " position on the pattern, should fall before the :

Mark_Bassett · January 11, 2019, 9:53pm

Oh my goodness. You are correct.

I've updated the gsub, but unfortunately my root issue still exists, it now just says the field is CellArray instead of Cell.


filter {
  mutate {
    id => "cell->CellArray"
    gsub => [ "message", "Cell\":\[", 'CellArray":[' ]
  }
  json { source => "message" }

I must have made an incorrect assumption about why the START_OBJECT message was being generated.


[2019-01-11T21:50:36,359][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"test-rgs-spin-2019-01-01", :_type=>"doc", :routing=>nil}, #], :response=>{"index"=>{"_index"=>"test-rgs-spin-2019-01-01", "_type"=>"doc", "_id"=>"otLlPmgBKn8SBCQ6OUIf", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [rgsResponseSummay.PopulationOutcome.Entry.CellArray] of type [text]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:1419"}}}}}

Chris_Lyons · January 11, 2019, 10:09pm

Might need to start a new index or delete your existing to ensure the type matches.

Mark_Bassett · January 11, 2019, 10:36pm

Got it working!! I had to update my mapping to say it is an object and not searchable. With it set to an object and still searchable I ran into another this same issue https://github.com/elastic/elasticsearch/issues/12366

Workaround/Fix in their thread fixed me as well.


"PopulationOutcome" : {
                "properties" : {
                  "Entry" : {
                    "properties" : {
                      "Cell" : {
                        "type" : "text",
                        "fields" : {
                          "keyword" : {
                            "type" : "keyword",
                            "ignore_above" : 256
                          }
                        }
                      },
                      "CellArray" : {
                        "type" : "object",
                        "enabled" : "false"
                      },

@Chris_Lyons Thank you for the eagle eye on the regex. I can't believe I missed that and I had several people looking at it lol!!

Chris_Lyons · January 12, 2019, 12:16am

Happy you got it working! When you have to deal with escape characters sometimes things get lost in the mix....

system · February 9, 2019, 12:16am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.