# Removing Data Without Deleting Index While Using Filebeat on Elasticsearch **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [February 6, 2023, 7:35am UTC](https://discuss.elastic.co/t/removing-data-without-deleting-index-while-using-filebeat-on-elasticsearch/324761 "2023-02-06T07:35:24Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![SFD13](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@SFD13](https://discuss.elastic.co/u/SFD13)\ **Post date:** [February 6, 2023, 7:35am UTC](https://discuss.elastic.co/t/removing-data-without-deleting-index-while-using-filebeat-on-elasticsearch/324761/1 "2023-02-06T07:35:24Z")
Hi everyone, I am using **filebeat-** \* index with some fields on **Elasticsearch**. _I want to remove all data on the elasticsearch which I used but that index remains_. It means that without deleting index name and contents (available fields) of it but data would be deleted. Some people say that > curl -XDELETE 0:9200/filebeat-\* could be useful. I suppose that the index will be deleted. How can I solve this problem? Thanks, ---
**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\ **Post date:** [February 6, 2023, 7:56am UTC](https://discuss.elastic.co/t/removing-data-without-deleting-index-while-using-filebeat-on-elasticsearch/324761/2 "2023-02-06T07:56:57Z")
Hello, You can try the delete\_by\_query API to delete specific documents. > **[Delete by query API | Elasticsearch Guide \[8.6\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html)** ---
**Author:** ![SFD13](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@SFD13](https://discuss.elastic.co/u/SFD13)\ **Post date:** [February 6, 2023, 8:24am UTC](https://discuss.elastic.co/t/removing-data-without-deleting-index-while-using-filebeat-on-elasticsearch/324761/3 "2023-02-06T08:24:09Z")
I am using Elasticsearch 7.17.6 version as offline. Might it causes any problem? Is it valid to delete data without index?: > curl -XDELETE 0:9200/filebeat/\* When I use the program (Zeek) to process the data, the data is uploaded to the Elasticsearch automatically. I want to delete data until processing a new data with new features in visualization. ---
**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\ **Post date:** [February 6, 2023, 8:52am UTC](https://discuss.elastic.co/t/removing-data-without-deleting-index-while-using-filebeat-on-elasticsearch/324761/4 "2023-02-06T08:52:31Z")
If you use the normal DELETE API you will delete the entire index, the `delete_by_query` API is also present in older versions. note that the `delete_by_query_api` is a slow way to do this.. ```auto POST /test-index/_delete_by_query { "query": { "match_all": {} } } ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 6, 2023, 11:03pm UTC](https://discuss.elastic.co/t/removing-data-without-deleting-index-while-using-filebeat-on-elasticsearch/324761/5 "2023-02-06T23:03:47Z")
Welcome to our community! 😃 Please note that delete-by-query is super inefficient. Why exactly do you want to do this? ---
**Author:** ![SFD13](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@SFD13](https://discuss.elastic.co/u/SFD13)\ **Post date:** [February 7, 2023, 7:49am UTC](https://discuss.elastic.co/t/removing-data-without-deleting-index-while-using-filebeat-on-elasticsearch/324761/6 "2023-02-07T07:49:12Z")
Actually, I am using Zeek to extract files from PCAP as offline. _Extracted data are visualized on the Elasticsearch using Kibana_. I have an one index name as '`filebeat`'. When I make content-related changes on my own Zeek, I can not see these changes on Elastic. Because this data was included (visualized) before. That's why I want to completely delete the extracted data on Elastic. I want to delete all the data **without deleting the index name** so that there is no problem in filebeat while deleting the data. @warkolm ---
**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\ **Post date:** [February 7, 2023, 8:08am UTC](https://discuss.elastic.co/t/removing-data-without-deleting-index-while-using-filebeat-on-elasticsearch/324761/7 "2023-02-07T08:08:59Z")
@SFD13 you can surely delete the index if you are re-ingesting the whole dump of data again into ES. When filebeat pushes the logs, ES will anyway check if the index is present or not, and if not, it is going to create a new index for you. The only issue here could be if there some specific index name you might be using in your application, in which case, the index name could be set on filebeat configuration. PS: there is no point in deleting all the data and keeping the index. ES only creates an index when the very first document/ event/ log for the index is ingested and stored. ---
**Author:** ![SFD13](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@SFD13](https://discuss.elastic.co/u/SFD13)\ **Post date:** [February 7, 2023, 8:29am UTC](https://discuss.elastic.co/t/removing-data-without-deleting-index-while-using-filebeat-on-elasticsearch/324761/8 "2023-02-07T08:29:50Z")
If I delete the index, will **yml files** like `filebeat.yml` or `filebeat.reference.yml` remain unchanged under **/etc/**? So, as said, deleting means that with (e.g) > curl -XDELETE [http://localhost:9200/filebeat](http://localhost:9200/filebeat) and then when I start the service with > sudo filebeat setup will it be updated in Elasticsearch? @Ayush_Mathur ---
**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\ **Post date:** [February 7, 2023, 8:50am UTC](https://discuss.elastic.co/t/removing-data-without-deleting-index-while-using-filebeat-on-elasticsearch/324761/9 "2023-02-07T08:50:55Z")
Deleting an index and restarting filebeat are two disparate things. You delete an index in Elasticsearch and it has none whatsover impact on Filebeat, the only thing impacted would be your logs which essentially would be deleted and won't be available in Kibana or any other client. When you start filebeat ,it will run normally, reading the logs, enriching them based on your processors (if any) and then sending them to Elasticsearch. By default, filebeat sets the value for `ctx._index` which tells ES in which index this particular document/event/log must be stored. Not that, filebeat itself is not storing the log or document into the index, it is Elasticsearch which writes and stores it. So essentially, when you delete the index, it is removed form ES, when you spin up your filebeat, it will send logs to ES for storing into `ctx._index` index. Elasticsearch will check if the index specified in the field `ctx._index` exists or not. If it exists, the document is added to that index, if not, it will create a new index with that name (value of `ctx._index` field) and store the document in that index. `sudo filebeat setup` essentially configures the index template (index setting and property mappings), ILM policy, etc. in ES that will be used when new index created based on that template. It is executed on ES when filebeat is started, but it won't change anything if templates and policies themselves have not been modified. Also, this command doesn't impact the reading and sending of logs, albeit it defines how the documents are stored and how indices are managed. ---
**Author:** ![SFD13](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@SFD13](https://discuss.elastic.co/u/SFD13)\ **Post date:** [February 13, 2023, 10:19am UTC](https://discuss.elastic.co/t/removing-data-without-deleting-index-while-using-filebeat-on-elasticsearch/324761/10 "2023-02-13T10:19:53Z")
After deleting an index, will Elasticsearch re-create that index? Is that right? For instance, assume that I delete the index that its name is filebeat. When I run that `sudo service filebeat start`, it should be recreate and the data goes on Elasticsearch. I suppose that the scenario would be after that if I delete an index. @Ayush_Mathur ---
**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\ **Post date:** [February 13, 2023, 10:26am UTC](https://discuss.elastic.co/t/removing-data-without-deleting-index-while-using-filebeat-on-elasticsearch/324761/11 "2023-02-13T10:26:18Z")
That's right, a new index will be re-created as soon as the first document is ingested for that index. ---
**Author:** ![SFD13](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@SFD13](https://discuss.elastic.co/u/SFD13)\ **Post date:** [February 13, 2023, 10:49am UTC](https://discuss.elastic.co/t/removing-data-without-deleting-index-while-using-filebeat-on-elasticsearch/324761/12 "2023-02-13T10:49:18Z")
Thanks for your help & time for me 🙂 @Ayush_Mathur ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 13, 2023, 12:49pm UTC](https://discuss.elastic.co/t/removing-data-without-deleting-index-while-using-filebeat-on-elasticsearch/324761/13 "2023-03-13T12:49:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.