Querying log.file.path returns no results

[strshp_419]

Elasticsearch, Filebeat, and Kibana versions are 7.12.0 on Windows Server 2016

In Kibana, querying on log.file.path returns no results in certain scenarios.

Example:
In Kibana, I go to Observability > Logs and submit a query that successfully returns results. After the results are returned, I select "View details" on a given row and filter on log.file.path. No results are returned.

When I take the same log file and copy it to C:\Temp (also being crawled by filebeat), I can filter on that log.file.path with no issues.

Does anyone know why filtering/searching on log.file.path doesn't work in the first scenario above but works fine in the second? Any help would be appreciated.

[wayneseymour (Tre')]

@strshp_419 you know my first "knee jerk" question is: What is the length in characters of the file path in the first scenario? I'm guessing that in the second scenario, the file path is way shorter, yes?

[strshp_419]

Correct! The character length of the first file path is indeed much longer. That length is 103 characters. The second path is only 36 characters long.

I also had the same knee jerk reaction but I couldn't find any indicator that the file path length was an issue.

[wayneseymour (Tre')]

Hrmm, perhaps we can try placing it at a file path of around 80 characters or so? Truly, I'm not sure, but I wonder if there might be a limit within the software, and not just an os limit.

[stephenb (Stephen Brown)]

Wonder if the \U ... Is getting interpreted as unicode?

[wayneseymour (Tre')]

Oh good one @stephenb !

[strshp_419]

I think you are on the right path @stephenb but it appears to be \t and not \U.

I can reliably recreate this issue now when using a file path that contains \t. Below are my findings.

I can't filter or search on file path for the following:

• C:\Temp\toast\U6943acbex6074a4a2e3ex_m.log
• C:\Temp\task\U6943acbex6074a4a2e3ex_m.log
• C:\Temp\test\U6943acbex6074a4a2e3ex_m.log

These work fine:

• C:\Temp\coast\U6943acbex6074a4a2e3ex_m.log
• C:\Temp\kask\U6943acbex6074a4a2e3ex_m.log
• C:\Temp\best\U6943acbex6074a4a2e3ex_m.log
• C:\Temp\TyrannosaurusRex\U6943acbex6074a4a2e3ex_m.log

When I escape the \t like this when searching, results are returned as expected:
log.file.path:"C:\Temp\\toast\U6943acbex6074a4a2e3ex_m.log

I'm really not sure what to do with this info yet but I'm happy I can recreate the issue.

[stephenb (Stephen Brown)]

Good to know....

Wonder if lowercase \u would cause issue too

If you try KQL query bar and put the path in single quotes what happens?

log.file.path : 'C:\Temp\toast\U6943acbex6074a4a2e3ex_m.log'

[strshp_419]

\u seems to work fine.

When I search with single quotes, Kibana gets angry...

Error: Expected AND, OR, end of input but ":" found.
log.file.path : 'C:\Temp\toast\U6943acbex6074a4a2e3ex_m.log'
------------------^
    at Object.fromKueryExpression (http://ht-alpha-tcdsp1:5601/39309/bundles/plugin/data/data.plugin.js:1:386086)
    at http://ht-alpha-tcdsp1:5601/39309/bundles/plugin/infra/infra.chunk.1.js:103:17156
    at Object.useMemo (http://ht-alpha-tcdsp1:5601/39309/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:382:64610)
    at useMemo (http://ht-alpha-tcdsp1:5601/39309/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:353:5228)
    at useLogStream (http://ht-alpha-tcdsp1:5601/39309/bundles/plugin/infra/infra.chunk.1.js:103:17023)
    at Provider (http://ht-alpha-tcdsp1:5601/39309/bundles/plugin/infra/infra.chunk.0.js:3:75009)
    at da (http://ht-alpha-tcdsp1:5601/39309/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:382:59332)
    at $a (http://ht-alpha-tcdsp1:5601/39309/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:382:67554)
    at xs (http://ht-alpha-tcdsp1:5601/39309/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:382:105587)
    at fl (http://ht-alpha-tcdsp1:5601/39309/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:382:90018)

[stephenb (Stephen Brown)]

Where is that error shown?

How are you submitting that query ... Confused where you are seeing error

Is that in Discover?

[strshp_419]

That error is being returned in Observability > Logs. I just copied/pasted into the search bar like so...

And then I get this...

Discover also gives me the same...

Although the full message is a little different...

[stephenb (Stephen Brown)]

Hmmm Here is my test case with 7.11.1

Yeah I was wrong with the Single Quote.... I was not in front of my computer

PUT test/
{
  "mappings": {
    "properties": {
      "mypath" : {
        "type": "keyword"
      }
    }
  }
}

POST test/_doc
{
  "mypath" : "C:\\Temp\\toast\\U6943acbex6074a4a2e3ex_m.log"
}

POST test/_doc
{
  "mypath" : "C:\\Temp\\coast\\U6943acbex6074a4a2e3ex_m.log"
}

And now without filter

And now with filter... seems to work

Now using KQL seems to work...

[stephenb (Stephen Brown)]

So try in the KQL Bar of the logs stream.

Sorry wrong with the Single Quotes

log.file.path : "C:\\Temp\\toast\\U6943acbex6074a4a2e3ex_m.log"

it should do type ahead for you after you type

log.file.path :

if you wait it should suggest

And for your paths it should look something like

[strshp_419]

Sorry. I'm new here...

I know that the query works fine when I use \\t instead of \t, but I'm not following how that resolves my issue. Are you saying that I just need to make sure that I replace \ with \\ when posting Windows logs?

[stephenb (Stephen Brown)]

You do not need to change the ingested path just the way you search on it.

If that's what you're asking why the filter doesn't work I will have to try I don't have a log source with those types of paths that I can stream but the KQL should work and that's yes the \\t is how you will have to search on it.

And as you see the type of head should do that for you.

It is JSON an so the back slashes need to be escaped.

I will need to figure out how to try to get that into the Logs Viewer and see if the filter has a bug ... which perhaps it might

[stephenb (Stephen Brown)]

I think if you actually look at the source documents you will see it like that