"Detected ambiguous Field Reference" warnings in Logstash 6.4.0

[vicpav (Victor)]

Hi,

After upgrading ELK cluster to 6.4.0 version, we have started to get Logstash warnings that make me puzzled:

{"level":"WARN","loggerName":"org.logstash.FieldReference","timeMillis":1537825989853,"thread":"Ruby-0-Thread-17@[main]>worker0: :1","logEvent":{"message":"Detected ambiguous Field Reference `products[0].id`, which we expanded to the path `[products, 0, .id]`; in a future release of Logstash, ambiguous Field References will not be expanded."}}
{"level":"WARN","loggerName":"org.logstash.FieldReference","timeMillis":1537825989853,"thread":"Ruby-0-Thread-17@[main]>worker0: :1","logEvent":{"message":"Detected ambiguous Field Reference `products[0].name`, which we expanded to the path `[products, 0, .name]`; in a future release of Logstash, ambiguous Field References will not be expanded."}}
{"level":"WARN","loggerName":"org.logstash.FieldReference","timeMillis":1537825989853,"thread":"Ruby-0-Thread-17@[main]>worker0: :1","logEvent":{"message":"Detected ambiguous Field Reference `products[0].quantity`, which we expanded to the path `[products, 0, .quantity]`; in a future release of Logstash, ambiguous Field References will not be expanded."}}
{"level":"WARN","loggerName":"org.logstash.FieldReference","timeMillis":1537825989854,"thread":"Ruby-0-Thread-17@[main]>worker0: :1","logEvent":{"message":"Detected ambiguous Field Reference `products[0].price`, which we expanded to the path `[products, 0, .price]`; in a future release of Logstash, ambiguous Field References will not be expanded."}}
{"level":"WARN","loggerName":"org.logstash.FieldReference","timeMillis":1537825989854,"thread":"Ruby-0-Thread-17@[main]>worker0: :1","logEvent":{"message":"Detected ambiguous Field Reference `products[0].totalPrice`, which we expanded to the path `[products, 0, .totalPrice]`; in a future release of Logstash, ambiguous Field References will not be expanded."}}
{"level":"WARN","loggerName":"org.logstash.FieldReference","timeMillis":1537825989854,"thread":"Ruby-0-Thread-17@[main]>worker0: :1","logEvent":{"message":"Detected ambiguous Field Reference `products[0].validity`, which we expanded to the path `[products, 0, .validity]`; in a future release of Logstash, ambiguous Field References will not be expanded."}} 

that are followed by the warnings incerting the logs into Elasticsearch

{"level":"WARN","loggerName":"logstash.outputs.elasticsearch","timeMillis":1537826018196,"thread":"Ruby-0-Thread-18@[main]>worker1: :1","logEvent":{"message":"Could not index event to Elasticsearch.","status":400,"action":["index",{"_id":"gke-XXX-/var/lib/docker/containers/XXX-json.log-227892","_index":"logs-2018.09.24","_type":"pumaska"},{"metaClass":{"metaClass":{"metaClass":{"action":"[\"index\", {:_id=>\"gke-XXX-/var/lib/docker/containers/XXX/XXX-json.log-227892\", :_index=>\"logs-2018.09.24\", :_type=>\"pumaska\", :_routing=>nil}, #<LogStash::Event:0x5592cf97>]","response":{"index":{"_index":"logs-2018.09.24","_type":"pumaska","_id":"gke-XXX-/var/lib/docker/containers/XXX/XXX-json.log-227892","status":400,"error":{"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"object field starting or ending with a [.] makes object resolution ambiguous: [.name]"}}}}}}}}]}}

The document in question is a log line with JSON content such that products is an array of Objects where every Object has keys id, name, quantity, etc.

The only Logstash filter that is touching products is a Ruby filter that gets the array, substitutes one of the key values to be type-consistent and sets it back:

        # If 'order.products.metadata.ride.leg.routeId' is not a number (but "new")
        if [message_json][order][products] {
          # Handle in ruby because logstash doesn't know what to do with arrays
          ruby {
            id => "order_metadata_ride_leg_routeId_type"
            code => "
              products = event.get('[message_json][order][products]')
              products.each { |p|
                # Logstash JRuby does not support dig, which means we have to do nested guards against nil...
                if (p['metadata'] && p['metadata']['ride'] && p['metadata']['ride']['leg'] && p['metadata']['ride']['leg']['routeId'])
                  routeId = p['metadata']['ride']['leg']['routeId']
                  unless routeId.nil? || routeId.is_a?(Integer)
                    p['metadata']['ride']['leg']['routeIdString'] = routeId.to_s
                    p['metadata']['ride']['leg']['routeId'] = -1
                  end
                end
              }
              event.set('[message_json][order][products]', products)
            "
          }
        }

Even though it is rather questionable if it is JRuby that somehow changes the array in a way so that Logstash cannot parse the field name as before.

I guess the actual warning comes down to elastic/logstash#9591 but I am not sure what am I supposed to do to make the logging flow to be Elastic 7.0 compatible.

Could somebody point me what would be the right way of fixing the system? Thank you!

[yaauie (Ry Biesemeyer)]

As the author of the pending strict-mode FieldReference parser, I'd be glad to help you chase this down.

The good news is that the problem here isn't the shape of the event, and that this will likely be fixed with a small change to how you're referencing the field in one of your filters or outputs.

Somewhere in your pipeline, the field reference products[0].id is being used to retrieve or store a value in an event. The legacy parser currently resolves that to something like "the product's 0th item's .id", but the strict syntax for this should be [products][0][.id].

The formal grammar documentation is currently a work in progress.

---

It is also possible to silence the warnings with the --field-reference-parser=LEGACY command-line flag or the config.field_reference.parser: LEGACY directive in logstash.yml.