I am new to ES. I have been installing it and working with its REST API. I like it and I would like to use it in an APP. However, I have started watching some tutorials to understand the full stack, including beats to be more comprehensive about it. More than one of the videos explained this isn't something to use in an APP as a DB like mongoDB, it should only be used for streams and log entry to search with.
I was looking forward to implementing this in an APP that will hold financial records like transactions and invoicing etc to learn how to use this and to leverage its awesome searching capabilities. Is this a good idea or should I really only focus using this for APP logs, linux platforms logs etc?
Elasticsearch doesn't have ACID guarantees like traditional databases and how you approach this depends on your risk profile. We recommend using things like replicas and snapshots to ensure you have availability and recoverability of your data if something goes wrong, which is pretty standard no matter what product you use.
I'm not sure what videos you watched, or how old they are, but there have been a huge numbers of improvements in data consistency in Elasticsearch since it had some known issues back on the 2.X and 5.X versions.
Thank you for your comment. I see, and I've read up on ACID.
So my understanding is:
Since for instance you are logging into ES logs from linux, they are already committed logs from actions that already happened with no risk or rollbacks, making it ideal for ES.
However, logging live transactions, that could fail on insert or have changes since DB's follow ACID does not make this ideal for ES?
There's zero reason you can't use Elasticsearch for transactional data. You treat each change as an event rather than updating the original one, as you then have a history of all the state changes (which you would want).
Think more about how you would recover from lost data though, you have backups, but do you also keep an archived file of all the transactions you can then replay? This isn't just an Elasticsearch problem though, this is something you should be doing for any system of record.
Yes, I see your point. The experience I wish to gain with ES is to hold past and present to future business transactions, also past invoicing and future invoicing I can then fetch as a REST API via PHP dynamic curl calls, which I would also want to updated invoices as paid etc.
If you're telling me ES is good for that kind of stuff, then I will be excited to start working on it in this perspective.
I guess I'm still a little confused for ACID and what ES should not be good for. So in a traditional APP, constantly doing DB inserts and updates on data is what ES wouldn't be ideal for and something like mongoDB is preferred?
An update to a document in Elasticsearch is expensive so if you are doing transactional data then that's not the right approach. Therefore don't update the original!
Log each change of state for the transaction as a new event in Elasticsearch. That means you have something like this as individual transactions;
PO created
Invoice created, with PO number stored in the document
Payment made, with invoice and PO number stored in the document
That way you can link it all together with the PO number.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
