Elastic and Logstash pipeline for Customer history

[ratheeshkrishnan92 (Ratheesh Krishnan)]

Hello Team,

I am using Elasticsearch version 6.2.4.

I have below scenario and need a work around for this. Planning to have two indices with below properties.

EmployeeId, EmployeeName, Designation.

index 1 : Employee (Latest designation of each employee)
index 2 : Employee_History (History of designation of each employee)

There is an employee who joined as Software Engineer.So the index1 and index2 will be having the documents as below

Index : Employee
EmployeeId : 1001, EmployeeName : Sam, Designation: Software Engineer

Index : Employee_History

Later he promoted as Senior Software Engineer. So the indices records should be as follow.

Index : Employee
EmployeeId : 1001, EmployeeName : Sam, Designation: Senior Software Engineer

Index : Employee_History
EmployeeId : 1001, EmployeeName : Sam, Designation: Software Engineer

Later he promoted to Team Lead. So the indices records should be as follow.

Index : Employee
EmployeeId : 1001, EmployeeName : Sam, Designation: Team Lead

Index : Employee_History
EmployeeId : 1001, EmployeeName : Sam, Designation: Software Engineer
EmployeeId : 1001, EmployeeName : Sam, Designation: Senior Software Engineer

Here I want to use logstash to push the documents to elastic.

Could you please suggest the elastic/logstash configurations for the workaround.

I want to keep the data in this way as I have to show the designation history (Employee + Employee_History) for each employee but I will be using the index Employee for all the search functionality as this index will be having the latest designation for each employees.

Thanks.
Ratheesh

[Badger]

What do your inputs look like? Where is the data coming from?

Why bother with multiple indices? If you timestamp or date the records you can simple fetch the most recent to get the current designation. It sounds like you are micro-optimizing search performance before you have any idea whether the simpler setup gives acceptable performance.

[ratheeshkrishnan92 (Ratheesh Krishnan)]

@Badger

ou have any idea whether the simpler setup

The data is being coming from application. I need to show the designation history of each employee on clicking on changelog button for each user. There I have to show the history. But for all other place especially for search I need only the latest designation of each employee. Actually I used employee scenario as an example. In my real case, I have 3 million SKUs and I'm having 1.5 Cr records for all the revisions of these 3 million records. Currently we have only one index and has all the data in that index(latest and history). Each document has more than 250 attributes. This index is good for showing the changelog of SKU but for the search functionality, we have to do an aggregation on these 1.5 cr records to find the latest revision of 3 million records before applying the search filter as search is applicable only for the latest values of these 3 million records. So I would like to remove the Aggregation by putting the latest data and history data into two indices. So latest data index can use for search and combination of latest and history can use for changelog.