Painless - check if a log contains a certain field

[yaelg]

Hey everyone,
I have the following fields in my logs:
"module": String
"audit": {"action": String, "user": String}

"audit" does not appear in all logs. I wanted to create a visualization of tag-cloud which would show the field "module" only for logs containing the field "audit". The problem is that you can't add a filter to a tag-clous visualization.

I've tried creating a scripted field called "audit-module" which returns doc["module"].value if and only if the field "audit" exists.

I've tried checking if the field exists with these methods:
doc.containsKey("audit") - for some reason always returns false
doc["audit"].size > 0
!doc["audit"].empty
!doc["audit.action.keyword"].empty
doc["audit"] != null
the last 4 do work, but they all cause the following runtime error for some logs that do not contain this field:

script_exception at shard 0 index sqlaudit-2020.12.15 node HKCyulOjSi--LtLScy34gQ

Type: script_exception

Reason: runtime error

Script stack:

org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:90)
org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)
if (doc['audit'] != null){
    
        ^---- HERE

Script:

if (doc['audit'] != null){
    return doc['module.keyword'].value;
}
else{
    return null;
}

Lang: painless

Position offset: 8

Position start: 0

Position end: 31

Caused by type: illegal_argument_exception

Caused by reason : No field found for [audit] in mapping with types [ ]

I would appreciate any help
Thanks in advance!

[thiago (Thiago Souza)]

First of all, it's important to understand that trying to address audit (either with [] or containsKey()) won't work since this field does not exists at index level as objects are flattened, so you can only access the underlying object fields using dot notation such as audit.action.keyword or audit.user.

@yaelg

the last 4 do work, but they all cause the following runtime error for some logs that do not contain this field:

The ones that refer to doc["audit"] won't work due to the reason I've explained above. But the reason that doc["audit.action.keyword"] should be because you are probably querying on sqlaudit-* which means that the query is hitting indices that does not contains a mapping for field audit.action.keyword such as sqlaudit-2020.12.15 (i.e. a document with "audit": { "action": "..." } was never indexed on day 2020.12.15 and the index template for sqlaudit also does not includes that mapping).

One way to workaround this issue, but I am not sure this solves your use case, is to filter documents that contains (or not) the audit.action field using an Exists query