Simultaneous data use of 2 indices differentes | Utilisation simultanée de 2 indices differents

kalache · August 17, 2021, 1:38pm

Hello,
I’m improving the IT inventory and I want to use the Elastic Stack to made it.
Logstash collect data into CSV file, Elasticsearch analyze this data and Kibana let us to visualize the data.

Why we want to use Elastic Stack ? I want a dynamic inventory, automated. Also, we can made more with the stack.

Already in use, I’ve got 2 data entries, with the name, the IP Address, OS..
Logstash collect data into 2 CSV files, and create 2 indices like « Inventory_1-date » and « Inventory_2-date ». I’ve create an Index also « Inventory* ».

I create annother index with the Dev Tools « referentiel » is a mapping of the differents indices.
I want to simultaneous use the data of both sources, with the IP Address like a reconciliation key. Is it possible ?

Actually, when i selection data of the source 1, source 2 display « No results found » with the « Inventory* » index and « referentiel » index.

Bonjour,

Je cherche à optimiser l’inventaire. J’ai pensé à utiliser la suite ELK dans le fonctionnement suivant :

Logstash : qui récupère les exports des fichiers CSV
Elasticsearch : qui s’occupe de traiter ces données
Kibana : qui va me permettre d’afficher ses données

Pourquoi utiliser la stack Elastic ? Pour avoir un inventaire dynamique, automatisé, se mettant à jour comme un grand. Et des nouvelles utilisations peuvent voir le jour grâce au fait que la stack est assez complète.

Pour l’instant j’ai 2 sources d’entrées, qui récupère des informations tel que le nom, l’adresse IP, l’OS etc.
Logstash est actuellement utilisés pour récupérer et créer 2 indices journaliers pourtant comme nom « inventaire_1-date » et « inventaire_2-date ». Un index a été créer avec comme nom « inventaire* ».
Un autre index portant le nom de « référentiel » a été créé via le mapping des diffèrent indices.

J’aimerais pouvoir faire en sorte d’utiliser dans Kibana, les données des deux sources simultanément, avec l’adresse IP comme clé de réconciliation. Est-ce possible ?
Actuellement, lorsque je sélectionne une donnée de la source 1, la source 2 m’affiche « no results found » avec les deux indexes.

warkolm · August 18, 2021, 10:02pm

Welcome to our community!
(@dadoonet might be able to help here with the French, in case I miss something.)

Are both of your index mappings are the same this should work, but it might help if you can share your mappings so we can take a closer look.

kalache · August 20, 2021, 7:51am

Hello @warkolm
Thanks for your reply.
So it's all the same ! I create un a alias with

POST _aliases
{
  "actions": [
    {
      "remove": {
        "index": "*",
        "alias": "inventaire"
      }
    }
  ]
}

And after i creat the index "inventaire" with the alias. And i map all in inventaire with

PUT inventaire
{ 
"mappings": {
"properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Application" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Backup_nbu" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Cluster" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Domaine" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Envirnoment" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Environement" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Evault_Agent" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Evault_Job" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Hypervisor" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "IPAddress" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Memory" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "NB_LAST_BACKUP" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Networks" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "NumCPU" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "OS" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "PMS" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "PMSInfo" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Powerstate" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Resp_app" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Site" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "StorageUsed" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "ToolsStatus" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "VMName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "cbt_reset_lock" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "host" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "path" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "tags" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            },
			 "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "BiosInfo" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "ClientVersion" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "ComputerRole" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "ComputerTargetGroupIds" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "FullDomainName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "IPAddress" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Id" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "LastReportedInventoryTime" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "LastReportedStatusTime" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "LastSyncResult" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "LastSyncTime" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Make" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Model" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "OSArchitecture" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "OSDescription" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "OSFamily" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "OSInfo" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "ParentServerId" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "RequestedTargetGroupName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "RequestedTargetGroupNames" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "SyncsFromDownstreamServer" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "UpdateServer" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "host" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "path" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "tags" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
		}
		}
		}
		}
		}

It doesn't work...

kalache · August 20, 2021, 7:55am

But I've got a warning with all fields !

#! At least one multi-field, [tags], was encountered that itself contains a multi-field. Defining multi-fields within a multi-field is deprecated and will no longer be supported in 8.0. To resolve the issue, all instances of [fields] that occur within a [fields] block should be removed from the mappings, either by flattening the chained [fields] blocks into a single level, or switching to [copy_to] if appropriate.
#! Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.13/security-minimal-setup.html to enable security.
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "inventaire"
}

We can see that it's work, but in my dashboard, when I took an information in the first data source, the second display "no results found"..

kalache · September 1, 2021, 8:32am

Hello everybody,
So I try it with more simple data.

My first data source is named : data1, and the seconde one : data2.

We have Fields 1,2,3,4 and IPAddress for row, and the data is like A1, A2,A3,B1,B2... and IPAddress.
I want to use the IP Address like a master key.

In this case, I just want to have A1, B1, C1 and D1 in the same line of IP Address 192.168.1.1.

warkolm · September 1, 2021, 9:54pm

Oh ok, Kibana can't do that at the moment (that I am aware of).

system · September 29, 2021, 9:54pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Querying in Elasticsearch Elasticsearch	4	313	May 23, 2018
Copy data from old index/indice over to new one Elasticsearch	5	1047	July 5, 2017
Matching data in two different indexes Kibana	5	861	January 23, 2023
Combining indices with same name but different content Elasticsearch	3	2422	December 18, 2017
Can someone please suggest better way here? Elasticsearch	8	555	March 18, 2018

Simultaneous data use of 2 indices differentes | Utilisation simultanée de 2 indices differents

Related topics