# Elasticsearch as a primary database **URL:** \ **Category:** Elasticsearch\ **Created:** [May 15, 2017, 9:59am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733 "2017-05-15T09:59:33Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![hossein\_ey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hossein_ey/32/20117_2.png) [@hossein\_ey](https://discuss.elastic.co/u/hossein_ey)\ **Post date:** [May 15, 2017, 9:59am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/1 "2017-05-15T09:59:34Z")
we have a project with 50,000 rows insert per second in database. our tables has schema... after inserts, we process data and make some reports and analysis... is good solution using elasticsearch as a primary database for our case or we should use another database and use elasticsearch just for analysis. and if I want use another database with elasticsearch, what is the recommended database for use with elasticsearch? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 15, 2017, 10:17am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/2 "2017-05-15T10:17:23Z")
There's a lot of "it depends" here. What sort of data, what sort of analysis? ---
**Author:** ![hossein\_ey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hossein_ey/32/20117_2.png) [@hossein\_ey](https://discuss.elastic.co/u/hossein_ey)\ **Post date:** [May 15, 2017, 10:41am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/3 "2017-05-15T10:41:25Z")
our data is network packets of local networks, for example a small office network. we analyse this network data for detect trends and for monitor network usage for now, our analysis is aggregation and filter on some filed for define trends and draw some charts ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 15, 2017, 10:46am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/4 "2017-05-15T10:46:57Z")
How important is the data? ---
**Author:** ![hossein\_ey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hossein_ey/32/20117_2.png) [@hossein\_ey](https://discuss.elastic.co/u/hossein_ey)\ **Post date:** [May 15, 2017, 10:56am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/5 "2017-05-15T10:56:38Z")
it is important : ) in insert time, loosing 50 or 100 packet for 1000,000 is acceptable but after insert, loosing data is not acceptable ---
**Author:** ![bobby259](https://avatars.discourse-cdn.com/v4/letter/b/9de053/32.png) [@bobby259](https://discuss.elastic.co/u/bobby259)\ **Post date:** [May 15, 2017, 2:40pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/6 "2017-05-15T14:40:16Z")
Similar use case. But, we never use elasticsearch as a primary database. Once the data is there is our databases (mostly SQL) we transform and store it on elasticsearch cluster for analysis and some adhoc projects but we do not use ES as primary. It's because, our systems were built long back and they are critical. But if you are building completely new systems then you have your freedom. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 15, 2017, 9:01pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/7 "2017-05-15T21:01:16Z")
Make sure you read [https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html](https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html) ---
**Author:** ![hossein\_ey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hossein_ey/32/20117_2.png) [@hossein\_ey](https://discuss.elastic.co/u/hossein_ey)\ **Post date:** [May 16, 2017, 5:31am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/8 "2017-05-16T05:31:59Z")
I check this link before ask question. but I don't understand what exactly it is... ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 16, 2017, 5:45am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/9 "2017-05-16T05:45:52Z")
> Some customers use Elasticsearch as a primary datastore, some set-up comprehensive back-up solutions using features such as our Snapshot and Restore, while others use Elasticsearch in conjunction with a data storage system like Hadoop or even flat files. Elasticsearch can be used for so many different use cases which is why we have created this page to make sure you are fully informed when you are architecting your system. Basically you are aware that something can hoes wrong but we are trying hard to make that never happen ---
**Author:** ![hossein\_ey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hossein_ey/32/20117_2.png) [@hossein\_ey](https://discuss.elastic.co/u/hossein_ey)\ **Post date:** [May 16, 2017, 6:03am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/10 "2017-05-16T06:03:38Z")
the most important issue is lost data in Insert? or after data indexed in elasticsearch something can goes wrong ? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 16, 2017, 6:48am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/11 "2017-05-16T06:48:59Z")
> [@hossein\_ey](#): > > the most important issue is lost data in Insert? Yes. > [@hossein\_ey](#): > > or after data indexed in elasticsearch something can goes wrong ? Yes. The cluster can in some extreme conditions become RED which means that some primary shards are not available anymore. ---
**Author:** ![hossein\_ey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hossein_ey/32/20117_2.png) [@hossein\_ey](https://discuss.elastic.co/u/hossein_ey)\ **Post date:** [May 16, 2017, 7:03am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/12 "2017-05-16T07:03:25Z")
> [@dadoonet](#): > > Yes. The cluster can in some extreme conditions become RED which means that some primary shards are not available anymore so in this situation data completely lost ? we can not restore that ? what is your recommendation? do we use elasticsearch as a primary database or we use another database, if we should use another, what is the best database matched with elasticsearch? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 16, 2017, 7:30am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/13 "2017-05-16T07:30:24Z")
> [@hossein\_ey](#): > > so in this situation data completely lost ? No. Not completely. Some shards might be missing. Not all. > [@hossein\_ey](#): > > we can not restore that ? It depends but it might be hard to recover from that situation. Depending on how the RED actually occurred. > [@hossein\_ey](#): > > what is your recommendation? do we use elasticsearch as a primary database or we use another database The one I already pasted: > Some customers use Elasticsearch as a primary datastore, some set-up comprehensive back-up solutions using features such as our Snapshot and Restore, while others use Elasticsearch in conjunction with a data storage system like Hadoop or even flat files. Elasticsearch can be used for so many different use cases which is why we have created this page to make sure you are fully informed when you are architecting your system. So it really depends on your case. IMO you have 3 options: - You absolutely don't care about loosing part of your data. Let say "non critical logs". Then, may be you will loose 1 day of data at some point but may be it's not that critical for you and actually don't worth adding another server for storing - You absolutely care about your data and you want to be able to reindex in all cases. You need for that a datastore. A datastore can be a filesystem where you store JSON, HDFS, and/or a database you prefer and you are confident with. About how to inject data in it, you may want to read: [https://david.pilato.fr/blog/2015-05-09-advanced-search-for-your-legacy-application/](http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/). - Well. You know that you can loose data in some extreme corner cases but you don't want to pay the price of other servers. Do backups, use elasticsearch replicas (increase to 2)... But you know the risk. ---
**Author:** ![n.maire](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@n.maire](https://discuss.elastic.co/u/n.maire)\ **Post date:** [May 18, 2017, 8:28am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/14 "2017-05-18T08:28:02Z")
Hi, How are you getting your packet information into ES ? If you have it go through Logstash at some point in your pipeline you could easily just configure another output to a more robust datastore if you're afraid of losing data. If it's just for backup purposes I'd probably just dump it in a compressed file on my SAN through the File output plugin. PS : 50K packets per second hardly seems like a small office's activity, that's almost what we get on one of our datacenters for 7K users 😃 PPS : also, unless you specifically require to store a copy of each and every network packet, I think you should look into using Netflow/IPFIX instead, that'd probably make it easier on your ES cluster than having all those packets going into it 😉 ---
**Author:** ![Arun\_Palanisamy](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@Arun\_Palanisamy](https://discuss.elastic.co/u/Arun_Palanisamy)\ **Post date:** [May 19, 2017, 5:50pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/15 "2017-05-19T17:50:06Z")
I would say use Cassandra database as primary database and for analysis use elasticsearch ---
**Author:** ![geena.rollins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/geena.rollins/32/15661_2.png) [@geena.rollins](https://discuss.elastic.co/u/geena.rollins)\ **Post date:** [May 19, 2017, 8:21pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/16 "2017-05-19T20:21:15Z")
I've consulted on two projects with 15,000 events per second. Elasticsearch puts indexing requests in the Indexing Request Queue. When that overflows, ES will reject indexing requests and lose data. To prevent data loss it is prudent to use Kafka as a buffer to level out traffic. Read the resiliency page in the right context. Elastic is perfectionist and Data Stax is cavalier. Lack of file checksums was identified and now fixed by Elastic. I asked a Data Stax Solutions Architect if Cassandra used file checksums and he said he didn't know. Elasticsearch uses two-phase commit to update cluster state. Cassandra gossips cluster state among nodes making the eventual consistency model very complicated and needing repairs. For writing data, turning on WRITE\_CONSISTENCY=ALL differs. If not all shards can be written, the document-index operation is rolled back by Elasticsearch. Cassandra warns the application to do a roll back. Unlike Couchbase, Cassandra can not determine the most recent updates and can not update a parallel Elasticsearch cluster. Writing a Cassandra secondary-indexing plugin for Elasticsearch is very complicated, and will result in duplicate documents that must be filtered after search. It almost impossible to use Cassandra with Elasticsearch effectively. In any system, there is a long multi-minute latency to satisfy a query over 4.2 billion documents. A 7-day query would be 29 billion documents. The only systems capable of handling this use case are Hadoop/Spark and/or Elasticsearch. Exact choices depend on more details of the use case. - One architecture is HDFS or S3 as event storage with Spark programs that write results into Elasticsearch for visualization. - Another choice is Elasticsearch only, with Elasticsearch Scroll programs that run on periodic jobs. ( See [http://www.leapfire.com/elasticsearchjoin.html](http://www.leapfire.com/elasticsearchjoin.html) ) - For analysis on time periods for 24 hours or less, consider Spark Streaming for short-term disposable summaries. For one of my clients, I consulted for just 5 hours and they said I saved them weeks of work in trying to make a good decision. As part of that, I did a one-hour presentation on Time-Series Event Systems. Consider Kafka, Logstash, Elasticsearch, Hadoop, HDFS, S3, Spark, and Spark Streaming. ...Geena ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 19, 2017, 9:20pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/17 "2017-05-19T21:20:38Z")
> [@geena.rollins](#): > > When that overflows, ES will reject indexing requests and lose data It's up to the client to retry, ES has told it it cannot accept anything else, so it's not losing anything. ---
**Author:** ![geena.rollins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/geena.rollins/32/15661_2.png) [@geena.rollins](https://discuss.elastic.co/u/geena.rollins)\ **Post date:** [May 19, 2017, 9:37pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/18 "2017-05-19T21:37:03Z")
Well, ES is not losing the data, the ETL pipeline is losing the data. Kafka with a Logstash consumer is a lossless ETL pipeline. ---
**Author:** ![geena.rollins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/geena.rollins/32/15661_2.png) [@geena.rollins](https://discuss.elastic.co/u/geena.rollins)\ **Post date:** [May 19, 2017, 10:05pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/19 "2017-05-19T22:05:09Z")
Another reason Elasticsearch is a better primary store is that Snapshots are Consistent at a time point. This is a natural benefit of Lucene's immutable files. Cassandra backups are Eventually Consistent and are difficult to restore. I saw co-workers doing a Cassandra restore; it was not a good day for them. For complete disaster recovery, restore a Snapshot from time T, then have Logstash consume messages after time T from Kafka. This will result in reading some messages twice. If the application assigns document ids (or Logstash computes them from other fields), then indexing requests will just replace a duplicate in Elasticsearch. If a shard goes Red and can not be restored, this disaster recovery method will work well. As a distributed primary store for Time-Series Event data, nothing is more solid than Elasticsearch using these techniques. ...Geena ---
**Author:** ![hossein\_ey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hossein_ey/32/20117_2.png) [@hossein\_ey](https://discuss.elastic.co/u/hossein_ey)\ **Post date:** [May 20, 2017, 8:15am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/20 "2017-05-20T08:15:26Z")
> [@geena.rollins](#): > > For complete disaster recovery, restore a Snapshot from time T, then have Logstash consume messages after time T from Kafka. This will result in reading some messages twice. If the application assigns document ids (or Logstash computes them from other fields), then indexing requests will just replace a duplicate in Elasticsearch. I think Kafka does not support random access and filter messages by time, how you handle this issue? [Next page](https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733.md?page=2)