How to filter by the nested values in the "message" field?

[Shiva_Subramaniyan]

Hi,

I have a "message" field in my "filebeat*" index.

This "message" field, particularly has nested fields like "httpRequest" and a "country" field in it. The value of this "country" field is 'US'

I want to use a query in Kibana to filter out the various values for this "country" field.

Can someone guide us here?

[cheiligers]

Hy @Shiva_Subramaniyan we added KQL support for querying nested fields in Kibana 7.6 and these are still supported in the current version (8.6). The examples should get you started: Kibana Query Language | Kibana Guide [8.6] | Elastic

[system]

Kibana 7.6 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

[Shiva_Subramaniyan]

Hi Cheiligers,

Many thanks for your response. When I use the filter as showm in the attachment, i get the following error

failed to create query: [nested] failed to find nested object under path [message]

Please advise.

[cheiligers]

We typically get that type of error when the nested fields you're querying on are either not mapped or not mapped as nested properties.
Can you check your mapping and see if the country is mapped as a nested property of httpRequest? if it is, then it also means it's mapped so the query might need to change a bit".

To check your mapping (you can use Kibana's Console app for these requests):

GET /filebeat*/_mapping

The mapping might have any fields, so you can narrow down the results to only include the httpRequest using:

GET /filebeat*/_mapping/field/httpRequest

or just the mapping for the country field:

GET /filebeat*/_mapping/field/httpRequest.country

Note: I've given links for the 8/6 version (the most recent), so you might need to change the stack version on the pages that the links points to.

There's way more info in the docs that including examples and things to watch out for, in case you get stuck.

[Shiva_Subramaniyan]

Hi Cheiligers,

Many thanks for your response.

The API request gives this response. Is this correct?

What should I do if it is not mapped?

[cheiligers]

Hi @Shiva_Subramaniyan you'll need to map it as a nested field in your index.

One can't change the mapping for a field that already exists in the index, so you'll first need to see if the parent (httpRequest) is mapped.

If it is, then you won't be able to change it to a nested field containing country. You'll need to create a new index with a nested field type for httpRequest and then reindex all your data from the current index into the new one.

There's a section in the Update mapping API docs that explains how to create a new index and reindex your data into the new one. The docs also gives examples.

What I recommend you do is:

1. Get the mapping of your existing index
2. Use the mapping that you got in 1 as the basis for the new index and then declare httpRequest as a nested field with the properties country and clientIp, headers and any others that are also nested in httpRequest.

You can use the same mappings for all the other fields in your index that you don't need to change.

I hope that helps!