Scripted field returns null on querying for logs

Elastic Stack Kibana
1

Hi community.

I am a newbie in Kibana and Elastic search. I am using scripted fields to search for logs containing certains strings and then visualize them. I have written my script in Painless. Here is the line that is returning null which it should not:

def m = doc['dissect.message.keyword'].value;

Basically I want to search for logs containing the string "logSearch" && "&fq=". And when I go to discover section and type in "logSearch" && "&fq=", the correct logs are returned, but scripted fields is unable to search for those logs. The very first line I mentioned above is returning null so further processing is not even possible.

But that above script line returns the correct logs when searching just for "logSearch". Meaning it returns null for logs that contains the both of the strings "&fq=" and "logSearch". I want to understand why it is returning null. I am attaching the log in json format for which the above script line returns null.

{
  "_index": "%{logplane}-2021.08.20",
  "_type": "doc",
  "_id": "di8DY3sByTEZYihPqwKx",
  "_version": 1,
  "_score": null,
  "_source": {
    "fields": {
      "logplane": "kubelog"
    },
    "@timestamp": "2021-08-20T10:03:07.756Z",
    "dissect": {
      "field_2": "log",
      "field_1": "var",
      "field_7": "be",
      "message": "stderr F 20-Aug-2021 10:03:07.182 INFO [http-nio-8080-exec-9] com.abcd.controller.HomeController.logSearch logSearch: signumid=9999_AND_userrole=SU_AND_username=UserName_AND_usergroup=2:name_AND_userquery=/solr/detail_view/select?&fq=abcd_name:searchTerm &q=*:*&rows=1000&wt=json",
      "field_3": "pods",
      "field_4": "prod",
      "field_6": "8e493f64-e2d4-4579-87b1-a6314ba5b5bb",
      "field_8": "0.log"
    },
    "severity": "info",
    "service_id": "UNKNOWN",
    "new_timestamp": "2021-08-20T12:03:07.183298076+02:00",
    "version": "0.2.0",
    "@version": "1",
    "timestamp": "2021-08-20T12:03:07.183298076+02:00"
  },
  "sort": [
    1629453787183
  ]
}
2

Hello @NavinKu10

Could you post the mapping for the index? I wonder if dissect.message.keyword doesn't exist but dissect.message does.

Thanks,
Matt

3

Hi, following is the mapping of one of the indices. All indices have the same mappings:

  "%{fields.logplane}-2021.08.10": {
    "mappings": {
      "doc": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "dissect": {
            "properties": {
              "field_1": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "field_2": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "field_3": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "field_4": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "field_6": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "field_7": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "field_8": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "message": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "fields": {
            "properties": {
              "logplane": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "message": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "new_timestamp": {
            "type": "date"
          },
          "pod_name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "service_id": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "severity": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "timestamp": {
            "type": "date"
          },
          "version": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    }
  }
4

This seems to confirm what I suspected - try dissect.message instead of dissect.message.keyword

5

Hi, actually I had tried dissect.message as well. When I create a scripted field using dissect.message the discover section just stops showing the logs. Like it doesn't show any logs. And as soon as I delete that scripted field, the discover section starts functioning normally. Also, aren't we suppose to use keyword when dealing with string type scripted fields? I am using Kibana 6.4.3.

6

EDIT:

Now it is working. I tried def m = params['_source']['message']instead of doc['dissect.message.keyword'].value; and now it returning me the desired logs. Thanks for the help anyways.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.