How to find text contains special chars in a in a field?

ShayWeizman · December 26, 2021, 11:53am

Hi All,

I'm using 7.16.1

I have a log field that contains a lot of text, I want to find this log by looking part of his long text...
For example, this is the field content:
strong texttestcaseFailedReason" : """Test failed: Assertion for errors, warnings and sensitive data in log file failed: ["Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609164] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 13 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609164] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 23 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609165] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 2 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609167] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 23 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609167] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 23 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609167] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 36 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609168] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 23 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609183] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 7 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609229] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 13 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609229] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 13 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609230] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 23 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609230] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 4 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609253] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 33 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609296] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 33 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609345] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 24 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609345] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 23 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609346] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 4 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:30 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420610289] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 20 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:30 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420610340] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 3 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >","Found an unexpected error log. line: Dec 25 08:23:30 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420610343] [clif_worker_1[45]::requests_handler_ajax.js:39] [FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST] Ajax response with command 5 for clientNetworkId a36595b4e284f0411640420608978 . But the tab is not in the tabs list <ClientId: __FGL__698514294e3d8 TabId: a36595b4e284f041 Url: >"] """,

How can I query for text e.g.:
"""Test failed: Assertion for errors, warnings and sensitive data in log file failed: ["Found an unexpected error log. line: Dec 25 08:23:29 ubuntu18-10-2020-tester-pulb57 localhost /usr/bin/node[45]: [ERROR] [1640420609164] [clif_worker_1[45]::requests_handler_ajax.js:39]

Thanks!

ShayWeizman · December 26, 2021, 12:08pm

More issue:
I can find this field with this query:

GET testcases/_search
{
  "query": {
    "regexp": {
      "testcaseFailedReason": {
        "value": ".*[FGERROR:REQUESTS_HANDLER_AJAX_TAB_NOT_IN_LIST].*",
        "flags": "ALL",
        "case_insensitive": true,
        "max_determinized_states": 10000,
        "rewrite": "constant_score"
      }
    }
  },
  "sort": [
    {
      "timestamp": "desc"
    }
  ],
  "fields": [
    "buildNumber",
    "buildVersion.keyword"
  ],
  "_source": false
}

But not with this query:

GET testcases/_search
{
  "query": {
    "regexp": {
      "testcaseFailedReason": {
        "value": ".*Found an unexpected error log.*",
        "flags": "ALL",
        "case_insensitive": true,
        "max_determinized_states": 10000,
        "rewrite": "constant_score"
      }
    }
  },
  "sort": [
    {
      "timestamp": "desc"
    }
  ],
  "fields": [
    "buildNumber",
    "buildVersion.keyword"
  ],
  "_source": false
}

Tomo_M · December 26, 2021, 12:38pm

Have you tried wildcard field type with wildcard query or text field with n-gram tokenizer?

I'm not sure why the second query does not hit the document, but the first query seems different from your intention because "[" has special meaning in regular expression.

And please use formatting with codes, logs and scripts from next time. thanks. You can use formatting with the button:

ShayWeizman · December 26, 2021, 12:56pm

How do I use the wildcard field, how I defined it? (I'm using java script to save the data to elastic, sending post request).

Tomo_M · December 26, 2021, 1:15pm

You can use wildcard field with explicit mapping when creating an index (or reindex).

Another way is add multi-field to an existing field using the update mapping API.

Mark_Harwood · December 26, 2021, 2:06pm

This blog may be useful.

system · January 23, 2022, 2:06pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.