Quotes are not converted correctly

Elastic Stack Kibana
1

Dear all =)

When creating Create Rule over the API, and want to search for host:"10.250.11.11", then I run into problems.

This is invalid JSON:

"esQuery": '{ "query": { "query_string": { "query": "host:"10.250.11.11" } } }'

Here Kibana removes the quotes around the IP address:

"esQuery": "{ \"query\": { \"query_string\": { \"query\": \"host:\"10.250.11.11\"\" } } }"

If I triple quote

"esQuery": "{ \"query\": { \"query_string\": { \"query\": \"host:\\\"10.250.11.11\\\"\" } } }"

then it becomes

{ "query": { "query_string": { "query": "host:\"10.250.11.11\"" } } }

but Kibana/Elastic doesn't unescape \", so the search it preforms is actually

host: \" 10 250 11 11 \"

Notice that searches for \". Where it should have been

host:"10.250.11.11"

If I had to guess, then the problem is that it isn't possible for Kibana/Elastic to convert "query": "host:\"10.250.11.11\"" into "query": "host:"10.250.11.11"" as that would be invalid JSON. Hence is it impossible to create rules over the API with searches that contain ".

Is that correct?

Hugs,
Sandra =)

2

JSON requires double-quotes around string values and keys. And in a JSON string, you can embed a double quote character by using "backslash double-quote" - \" (hoping that renders as the backslash character followed by the double quote character!)

So your first attempt seems closest. I think this is the value you want.

{"esQuery":{"query":{"query_string":{"query":"host:\"10.250.11.11\""}}}}

Triple quotes are largely a Kibana Dev Console thing, and aren't generally accepted in Kibana HTTP APIs, since they are not valid JSON - they're an extension of JSON.

3

If I send

{"esQuery":{"query":{"query_string":{"query":"host:\"10.250.11.11\""}}}}

then Kibana/Elastic treats the search as host:\" 10 250 11 11 \".

If I had to guess then the problem lies in Escaping_Special_Characters ie \" will never be unescaped, as Lucene sees \" as it should search for " and not for denoting start/end of a string?

I suppose the problem is that " can mean two different things. Eg.

host:" text with a " inside "
     ^             ^
     |             +-- char to search for
     +-- start of string

or

host:" text with a " inside "
     ^             ^
     |             +-- start of string
     +-- char to search for

and Lucene have no way to tell which is which and therefore treats all escaped chars a chars to search for.

A since JSON requires double quotes, all other inner double quotes have to be escaped, and hence Lucene sees them as chars to search for.

Or am I misunderstanding the situation?

4

Ah, I see what you're saying.

I tried several different ways of escaping a double quote in a search running from Dev Tools. And of course it might have it's own quoting issues. No luck when trying to include the quote in the query.

I did have some luck using a regexp though, and using . (wildcard) instead of the quote. For instance:

GET .kibana/_search
{
  "query": {
    "query_string": {
      "query": "/.*double quote . in the name.*/"
    }
  }
}

It found a document where a field had the text double quote " in the name. This query isn't great, since it uses regexp, searches default fields, and uses .* - I wasn't able to figure out how to give it a specific field name and still work, which should make it faster.

So I think we first need to figure out the query DSL for the query you want, not within alerting - either from Dev Console or via curl, using the _search API. Once we have that, hopefully we can figure out how to encode that in JSON so it gets passed along the way correctly.

5

That would be great =)

It would be

host:"10.250.11.11" and custom4:"errors occurred"

Whenever JSON is involved I can't see any solution. If only the the quotes could be written as %22.

6

I'd like to see the complete DSL, that you would use in a curl command or in the Dev Console. Otherwise I'm just guessing about the other parts :slight_smile:

7

Sorry, I misunderstood. Here it comes =)

{
   "params":{
      "esQuery":"{ \"query\": { \"query_string\": { \"query\": \"host:\\\"10.250.11.11\\\" and custom4:\\\"errors occurred\\\"\" } } }",
      "size":100,
      "timeWindowSize":60,
      "timeWindowUnit":"s",
      "threshold":[
         "1"
      ],
      "thresholdComparator":">=",
      "index":[
         "prod_log"
      ],
      "timeField":"@timestamp"
   },
   "consumer":"alerts",
   "schedule":{
      "interval":"60s"
   },
   "tags":[
      
   ],
   "name":"nonprod_test44",
   "enabled":true,
   "throttle":null,
   "rule_type_id":".es-query",
   "notify_when":"onActiveAlert",
   "actions":[
      {
         "group":"query matched",
         "id":"_REPLACE_THIS_",
         "params":{
            "body":"{\"system\":\"system2\",\"key\":\"nonprod_test44\",\"date\":\"{{context.date}}\",\"alert_id\":\"{{alertId}}\",\"alert_message\":\"{{alert.id}}\",\"matched_documents\":\"{{context.value}}\",\"documents\":\"[ {{context.hits}} ]\"}"
         }
      }
   ]
}
8

Sorry, I was not clear. I want to see a plain old Elasticsearch search, via curl or Dev Console, that performs the search you want to embed in the rule. Nothing to do with the rule :slight_smile:

Once we have that, I can see if we can figure out how to embed that inside the rule params.

I think what you pasted was your current attempt at trying to build the rule? Which doesn't work?

Another thought is this - especially since I will be mostly offline soon until Monday - once you have that query, you should be able to paste it into the Kibana "create alert" UI for the Elasticsearch query rule. Assuming the rule then works the way you want, you could use curl to get the rule data back out, to figure out how it was formatted. Get API here: Get rule API | Kibana Guide [master] | Elastic - there's also a Find API you can use, if you don't know the rule's ID which is required for the Get API - actually, you can just use the Find API unless you have lots of rules - it also returns the JSON definition of the rules.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.