Create Kibana vizualiation based on keywords matched from message column

Elastic Stack Kibana
1

I know it's very easy to create visualizations based on predefined columns, but I wonder what is the proper way to create visualization based on some words from messages? Please share any useful guidance or tip about this

2

Could you provide some sample messages and a general idea of the output you're looking for?

3

Thanks @mattkime, this is mostly fluend stuff, but probably you have an idea how to deal with this.

Below is a fluend grok filter which creates additional fields: app_timestamp, app_log_level, app_message.
What I'd like to have is additionally parsed app_message field which seems to be json+raw data. So as a result it would be possible to create visualization based on values legalForm, legalName, from field app_message

<filter kubernetes.**>
    #@id filter_parser
    @type parser
    key_name log
    reserve_data true
    reserve_time true
    <parse>
      @type grok
      grok_failure_key grokfailure
      <grok>
        pattern %{TIMESTAMP_ISO8601:app_timestamp}" level\=%{DATA:app_log_level} msg\=%{GREEDYDATA:app_message}
      </grok>
    </parse>
</filter>

app_message field example:

Create data draft in db: DatasetDraftVO(id=d57f73fc547f, datasetVO=null, entityId=legalEntityId, datasetVersion=1, data=VerifiableCredential(contexts=[https://www.w3.org/2018/credentials/v1, https://app.com/api/v1], id=le_cd_1, types=[VerifiableCredential,credentialSubject={"corporateData":{"bylaws":{"value":""},"language":{"value":"English"},"legalForm":{"value":"Private limited company"},"legalName":{"value":"CompanyName1"},"companyEmail":{"value":"test@mail.com"},"companyWebsite":{"value":"MyProgram.fr"},"country":{"value":"France"},"mailbox":{"value":"444"},"zipCode":{"value":"2000"}} proofs=null), draftStatus=DRAFT)
4

There's a large number of ways this data could be visualized. Do you want to see a graph of documents over time?

If you'd like to browse some ideas you can look at kibana sample data - https://www.elastic.co/guide/en/kibana/current/add-sample-data.html

5

Thank you, @mattkime, should I parse below string into separate fields further in order to have option to create a dashboard based on keywords like
"country":{"value":"France"}
"country":{"value":"England"}

Create data draft in db: DatasetDraftVO(id=d57f73fc547f, datasetVO=null, entityId=legalEntityId, datasetVersion=1, data=VerifiableCredential(contexts=[https://www.w3.org/2018/credentials/v1, https://app.com/api/v1], id=le_cd_1, types=[VerifiableCredential,credentialSubject={"corporateData":{"bylaws":{"value":""},"language":{"value":"English"},"legalForm":{"value":"Private limited company"},"legalName":{"value":"CompanyName1"},"companyEmail":{"value":"test@mail.com"},"companyWebsite":{"value":"MyProgram.fr"},"country":{"value":"France"},"mailbox":{"value":"444"},"zipCode":{"value":"2000"}} proofs=null), draftStatus=DRAFT)

6

Yes, I'd recommend that. Generally speaking its a good idea to think of Kibana as displaying elasticsearch queries - if you can simplify the query by doing some upfront data parsing you'll likely save yourself a lot of work further down the line.

7

ok, so without fluent or logstash there is no way to do this, is that correct?

8

There may be other solutions but they become increasingly difficult or inflexible. Perhaps a scripted field is worth considering - https://www.elastic.co/guide/en/kibana/current/scripted-fields.html

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.