Filter by value on multiple events

Elastic Stack Kibana
1

Hi,
I'm starting with Elastic Stack.
My purpose is to handle log for many applications (stored in same index because i need to follow reference field across all applications), i got application in a json field as key/value.
I want to get a dashboard where i can see a table with all values of 'reference' field which are present or not in certain application.
Exemple:
-data:
timestamp | App | Reference | message
*** | X | 45 | "This is a log"
*** | Y | 45 | "This is a log"
*** | X | 46 | "This is a log"
*** | X | 47 | "This is a log"

what i would like to get :

A filter where i can select an App

   Reference       List of App where Reference appear
        46           |    X                                                               
        47           |    X

I tried by aggregating Reference and a filter removing "Y" App, but what i get is :

Reference List of App where Reference appear
45 | X
46 | X
47 | X

Is there anyway i could get the relation between multi events like saying i want all Reference value for which in all events it doesn't get any "Y" value for App.
Thanks in advance

2

@Unkownfo, welcome the the community and the Elastic stack!
You can use the data table visualization to show your logs in the categories you've described above, then add that to your dashboard.
Once you have the dashboard, you can set a global filter to exclude the values you're not interested in.
Here are some reference docs and tutorials to help guide you.

If you provide an example set of the logs, their mappings in the index and the version of the stack you are running, we'd be able to help you more.

Documentation and guides:
Getting started with Kibana: https://www.elastic.co/guide/en/kibana/7.9/getting-started.html
Creating a dashboard: https://www.elastic.co/guide/en/kibana/7.9/dashboard-create-new-dashboard.html
Lens (drag and drop): https://www.elastic.co/guide/en/kibana/7.9/lens.html
The visualize app: https://www.elastic.co/guide/en/kibana/7.9/visualize.html
Using a global filter:

Tutorials:
using lens: https://www.elastic.co/webinars/visualizing-your-observability-data-through-the-kibana-lens

3

Hi, thank for your response.
I'll try to be clearer about what i want.
What i want to do is to filter but it need "a context".
What happen is :
I got a reference value which is log through many Sub-applications, so i can pass by severeal different way (sometimes going through some sub-applications sometimes through some other one and there is several end to the process (depending on the sub-application). What i would like is to be able to see in a data table all references (which is seen on many events) minus all the one seen in all my endpoint of the process that way i could find which reference is stuck in my process.

Here is the mapping of my index :

{
  "This_is_my_index" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "App" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Script" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "SubApp" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "agent" : {
          "properties" : {
            "ephemeral_id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "asctime" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "ecs" : {
          "properties" : {
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "host" : {
          "properties" : {
            "architecture" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "containerized" : {
              "type" : "boolean"
            },
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "ip" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "mac" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "os" : {
              "properties" : {
                "codename" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "family" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "kernel" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "platform" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "version" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            }
          }
        },
        "input" : {
          "properties" : {
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "json" : {
          "properties" : {
            "App" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Script" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "SubApp" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "asctime" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "levelname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Reference" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "message" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "levelname" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "log" : {
          "properties" : {
            "file" : {
              "properties" : {
                "path" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "offset" : {
              "type" : "long"
            }
          }
        },
        "Reference" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "tags" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}
4

Here is an exemple of my event's data (i've cleared all the unnecessary fields) :

{
  "took" : 10,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2701,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "This_is_my_index",
        "_type" : "_doc",
        "_score" : 1.0,
        "_source" : {
          "Script" : "make_log",
          "@version" : "1",
          
          "input" : {
            "type" : "log"
          },
          "log" : {
            "offset" : 0,
            "file" : {
              "path" : "path_to_the_source_file"
            }
          },
          "message" : "talking about reference",
          
          "reference" : "string_reference_0",
          "levelname" : "INFO",
          "ecs" : {
            "version" : "1.5.0"
          },
          "asctime" : "2020-08-19 14:59:05,598",
          "App" : "Logmaker",
          "@timestamp" : "2020-08-19T12:59:05.598Z",
          "SubApp" : "analyse_app"
        }
      },
      {
        "_index" : "This_is_my_index",
        "_type" : "_doc",
        "_score" : 1.0,
        "_source" : {
          "Script" : "make_log",
          "@version" : "1",
        
          "App" : "Logmaker",
          "input" : {
            "type" : "log"
          },
         
          "message" : "This is a log",
          "levelname" : "INFO",
          "ecs" : {
            "version" : "1.5.0"
          },
          "reference" : "string_reference_0",
          "asctime" : "2020-08-19 14:59:05,625",
          "log" : {
            "offset" : 213,
            "file" : {
              "path" : "path_to_the_source_file"
            }
          },
          "@timestamp" : "2020-08-19T12:59:05.625Z",
          "SubApp" : "connect_app"
        }
      },
      {
        "_index" : "This_is_my_index",
        "_type" : "_doc",
        "_score" : 1.0,
        "_source" : {
          "Script" : "make_log",
          "@version" : "1",
          "log" : {
            "offset" : 460,
            "file" : {
              "path" : "path_to_the_source_file"
            }
          },
          "input" : {
            "type" : "log"
          },
          "message" : "final step my reference got out of the process",
          "levelname" : "INFO",
          "App" : "Logmaker",
          "reference" : "string_reference_0",
          "asctime" : "2020-08-19 14:59:05,649",
          "ecs" : {
            "version" : "1.5.0"
          },
          "@timestamp" : "2020-08-19T12:59:05.649Z",
          "SubApp" : "final_app"
        }
      }
    ]
  }
}
5

And here an exemple of what i would like at the end :

From :

Reference |        SubApp           | message
    A     |      First_app          | A gone through First_app
    A     |         Second_app      | A gone through Second_app
    B     |      First_app          | B gone through First_app
    B     |         Second_app      | B gone through Second_app       
    B     |      Final_app          | B gone through Final_app

so for those data i would like to get that type of data table :

 Reference | --
       A   | --

So as return i just want to get A as it isn't seen in the final_app while B is

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.