Creating a data view from logstash

Dor-Alter · December 24, 2023, 4:04pm

I have created a pipeline that loads my logs into elasticsearch. When I run the pipeline it works and in the developer console, in elasticsearch, I can see and run queries to the index. However, when I go to discover in Analytics I do not see the index as data view. I can create a data view out of this, but I was wondering if it is possible to configure the logstash config to save the logs directly as data view.

I am trying to automate my set up as much as possible, such that I will be able to run docker compose and get all my dashboards showing up. For that it will be convenient to have the logs save directly into a data view rather then needing to do it by hand.

My other question is, is it possible to create such an automatisation using docker that will run the logstash to get the data view and then run the dashboard graphs that I have created? If so, is there a tutorial or guide to how to do that?
I have graphs on a dashboard now but they are all configured and created by hand.

Any answer to either of the questions will be highly appreciated.

stephenb · December 24, 2023, 6:59pm

Hi @Dor-Alter

Logstash is not the proper way to Create Dataview and / or Load Dashboars you need to use the Kibana APIs for that.

Example Create Data View

Can you automate?... sure but there is nothing out of the box you will need to create your own custom scripts to do this and then call them within a "Setup Container" within your compose....

There is a sample setup container in the Example of How to Setup up an Elasticsearch Cluster with Docker

You can see the setup container here... I am not saying this is exactly what you want but perhaps it can get you started.

github.com

elastic/elasticsearch/blob/8.11/docs/reference/setup/install/docker/docker-compose.yml

version: "2.2"

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;

This file has been truncated. show original

system · January 21, 2024, 6:59pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.