Advice on a use case needed

Elastic Stack Elasticsearch
1

Not sure if ELK stack is the right way to solve this problem, but let's just talk through it. Here is a new use case I have:

We get a daily playload with data which contains account and invoice data. We record this accounts data throughout the month into ES. As we
get closer to month end, we need to validate these accumulated accounts against a
remote database (JDBC query). Some of these accounts will be valid
(found), others will be invalid and we need to mark them as such in ES
so we can report on it. Ideally, I would like to re-trigger this
validation periodically, so as to get a more accurate picture of the
account statuses as we get closer ot month end billing run (around 21st
of each month).

How would we try to solve that using an ELK stack tools?! Is it possible to do that? Would it be practical? We have Java/JDBC skills on the team, as well as up and coming ELK skills. Please comment on the possible approach.

2

Why not simply use a RDBMS? If you do not have data that needs to be
sharded and do not need search capabilities, why use Elasticsearch?

Ivan

3

We need to keep track of, search and visualize the data coming into our system. The additional metadata needed is not in our RDBMS, it's in another remote system. Thus, the data needs to be 'enriched' in order to be fully meaningful.

I'm thinking of a Java application that can run from a command line and do the following tasks:

  1. Extract all unique account numbers from ES (using APIs)
  2. Craft a JDBC query to the remote system to validate these accounts
  3. Update all accounts in ES with appropriate status using bulk load that the java utility generates or using Java API (update)

There are approx 1600-1800 accounts each month...