Error in elasticsearch filter of logstash: Failed to query elasticsearch for previous event

[Vita_Rosenberg]

Hello.
I have an index in kibana called poc1*, it contains fields of machineId and timestamp
I need to get from that index a maximum value of timestamp to a new index for each machineId (which is deviceId in sql).
It is working if I specify a certain machine (ma1). But when trying to join machineId to deviceId it fails with - Failed to query elasticsearch for previous event.
Any ideas? What am I doing wrong?

this is the full error log:

 [3fc577f9e5c3f1bffc1199cd5f29ecdcd345d10b38736973b643f854c440f086] Failed to query elasticsearch for previous event {:index=>"poc1-2020.11.25", :error=>"[400] {\"error\":{\"root_cause\":[{\"type\":\"query_shard_exception\",\"reason\":\"Failed to parse query [MachineId:%{[deviceId]}]\",\"index_uuid\":\"-hEaWnQ-ThOunikXCMS9cg\",\"index\":\"poc1-2020.11.25\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"poc1-2020.11.25\",\"node\":\"p9GnWMV3RGiiE9AUHRD5Qg\",\"reason\":{\"type\":\"query_shard_exception\",\"reason\":\"Failed to parse query [MachineId:%{[deviceId]}]\",\"index_uuid\":\"-hEaWnQ-ThOunikXCMS9cg\",\"index\":\"poc1-2020.11.25\",\"caused_by\":{\"type\":\"parse_exception\",\"reason\":\"Cannot parse 'MachineId:%{[deviceId]}': Encountered \\\" \\\"]\\\" \\\"] \\\"\\\" at line 1, column 21.\\nWas expecting:\\n    \\\"TO\\\" ...\\n    \",\"caused_by\":{\"type\":\"parse_exception\",\"reason\":\"Encountered \\\" \\\"]\\\" \\\"] \\\"\\\" at line 1, column 21.\\nWas expecting:\\n    \\\"TO\\\" ...\\n    \"}}}}]},\"status\":400}"}

Thanks.

[tbennett]

I'd love to get an answer as well. I'm experiencing the same issue.

query => "organization.id: %{[organization][id]}"

"reason":"parse_exception: Encountered \\" \\"]\\" \\"] \\"\\" at line 1, column 31.\\nWas expecting:\\n    \\"TO\\" ...\\n    "

[Badger]

@Vita_Rosenberg

"reason":"Failed to parse query [MachineId:%{[deviceId]}]",

The elasticsearch filter does process sprintf references in the query option. If it does not get processed it is telling you that that event does not have a [deviceId] field.

[Vita_Rosenberg]

Hi.
I solved mine. It never worked with the query. Though i did manage to make this work with a query_template.json. Please show me your full code, maybe I can help you as well..

[tbennett]

I did try it without the sprintf references just in case, but that didn't appear to make a difference; which would make sense.

I think what got me on mine was, for debugging purposes I added an output to file plugin immediately after that, but I guess that plugin was not catching it until all the other filters were processed. So the output of my output plugin for debugging did not match the actual event itself.

specifically [organization][id] did not exist at that point, setting the debug level high enough to log the event as it crossed the pipeline showed the field had a different name [orgid].

[tbennett]

I suspect your events did not have [deviceId], as that's what my own debugging lead me to believe for my situation