# Logstash Sending Old Data to ES **URL:** \ **Category:** Logstash\ **Created:** [January 28, 2019, 9:53am UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974 "2019-01-28T09:53:12Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Aralex](https://avatars.discourse-cdn.com/v4/letter/a/e56c9b/32.png) [@Aralex](https://discuss.elastic.co/u/Aralex)\ **Post date:** [January 28, 2019, 9:53am UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974/1 "2019-01-28T09:53:12Z")
Hi, my pipeline consists of: filebeat -\> logstash -\> ES -\> kibana. I use it to read CSV files in a folder and index/mirror any changes or updates in real time on kibana. Problem 1: My test CSV file has 16 entries bus only 3 are reaching kibana, I see only 3 entries on kibana's ES index management section. Problem 2: I'm using one of the fields as a finger print to guarantee one instance of each entry being sent to ES. It hasn't worked so far. Problem 3: When I fire up the pipeline, logstash sends old data to ES, data that has been indexed before even though the CSV file is no longer present in the specified folder. This tells me that this old data is cached shomewhere in the pipeline. Even after I delete everything in the 'data' folder on filebeat, this still happens. I read a lot and experimented a lot about how filebeat and logstash work, and on how any why those files in the 'data' folder are created. Why old data is being sent from logstash to ES? Question: I terminate everything in my pipeline through ctr+c from the powershell window. That writes some files so logstash starts from where it left the next time it is initiated. Could this be the cause of problem 3? Is there a recommended way to terminate them? Thank you very much. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 28, 2019, 9:54am UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974/2 "2019-01-28T09:54:37Z")
Please show your full configuration. Without this it is very hard to help. ---
**Author:** ![Aralex](https://avatars.discourse-cdn.com/v4/letter/a/e56c9b/32.png) [@Aralex](https://discuss.elastic.co/u/Aralex)\ **Post date:** [January 28, 2019, 1:19pm UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974/3 "2019-01-28T13:19:35Z")
Hello Christian, thank you for the reply. Below are my configs: elasticsearch.yml: all commented out. kibana.yml: all commnted out. filebeat modules: all disabled. filebeat.yml: ############################ filebeat.inputs: - type: log paths: output.logstash: hosts: ["localhost:5044"] ######################### logstash.config: ######################### input { beats { port =\> 5044 } } filter { csv { columns =\> ["col name 1","col name 2","col name 3","col name 4"] autogenerate\_column\_names =\> false separator =\> ";" skip\_empty\_rows =\> true skip\_empty\_columns =\> true convert =\> {"col name 2" =\> "date\_time"} } date { match =\> ["col name 2","YYYY-MM-dd HH:mm:ss"] target =\> "col name 2" } } output { elasticsearch { hosts =\> ["[http://localhost:9200](http://localhost:9200)"] index =\> "mytestdata-%{+dd.MM.YYYY}" template\_overwrite =\> true document\_id =\> "%{[col name 1]}" } stdout {} } ############### Field mappings json file: { "template" : "mytestdata-%{+dd.MM.YYYY}", "version" : 1, "mappings" : { "_default_" : { "properties" : { "@timestamp": { "type": "date" }, "col name 2": { "type": "date", "format": "YYYY-MM-dd HH:mm:ss" }, "col name 1": { "type": "keyword" }, "col name 3": { "type": "keyword" }, "col name 4": { "type": "keyword" } } } } } #################### I then add those mappings to ES through: curl -XPUT -H "Content-Type: application/json" [http://localhost:9200/\_template/my\_index\_template?pretty](http://localhost:9200/_template/my_index_template?pretty) -d @mytemplate.json ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 28, 2019, 1:23pm UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974/4 "2019-01-28T13:23:25Z")
What does the CSV file look like? ---
**Author:** ![Aralex](https://avatars.discourse-cdn.com/v4/letter/a/e56c9b/32.png) [@Aralex](https://discuss.elastic.co/u/Aralex)\ **Post date:** [January 28, 2019, 1:27pm UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974/5 "2019-01-28T13:27:26Z")
I cannot share the data, but this is the structure: col name 1;col name 2; col name 3; col name 4 text 1;date;text 3;text 4 text 1;date;text 3;text 4 text 1;date;text 3;text 4 There are no numeric values in the CSV except a date field. Everything else is a string/text field, which is why it's critical to have each entry represented as a single ES document (no copies from logstash sending the same data multiple times to guarantee delivery), this is why I'm trying to use a specific field as a fingerprint. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 28, 2019, 1:35pm UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974/6 "2019-01-28T13:35:29Z")
How many unique values are there in the first column? As you are using this as document ID, only one document per unique ID will be created in Elasticsearch. ---
**Author:** ![Aralex](https://avatars.discourse-cdn.com/v4/letter/a/e56c9b/32.png) [@Aralex](https://discuss.elastic.co/u/Aralex)\ **Post date:** [January 28, 2019, 1:52pm UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974/7 "2019-01-28T13:52:24Z")
I see! So the order of columns is important? My first column has many repeated values. The one with unique values (IDs) are in column 3. So if I change it and put it as the first column, it would solve some issues? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 28, 2019, 1:54pm UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974/8 "2019-01-28T13:54:37Z")
Rows are processed one by one so only the number of unique values will affect the document count. If you remove the `document_id` parameter from the output, Elasticsearch will assign an id to each row and all should get inserted. ---
**Author:** ![Aralex](https://avatars.discourse-cdn.com/v4/letter/a/e56c9b/32.png) [@Aralex](https://discuss.elastic.co/u/Aralex)\ **Post date:** [January 28, 2019, 2:21pm UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974/9 "2019-01-28T14:21:21Z")
I exchanged the column orders and placed the one with the unique values as the first column, since I'm using it as the document\_id for fingerprinting. Also changed the field mapping order in logstash.config, and it worked!! So, column order IS important. Thank you Christian. Another question: my test data has 20 ES documents (entries) only, but it's consuming 121.3kb according to the index management section. If I drag and drop the file manually, the index consumption is much smaller. Why is that? How can I reduce memory consumption? If I use my full data with +45000 entries, mem consumption goes to 18mb which is pretty large for a csv file. ---
**Author:** ![Aralex](https://avatars.discourse-cdn.com/v4/letter/a/e56c9b/32.png) [@Aralex](https://discuss.elastic.co/u/Aralex)\ **Post date:** [January 28, 2019, 2:22pm UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974/10 "2019-01-28T14:22:30Z")
Also, is there a recommended order to fire up logstash and filebeat? or it doesn't matter? I tried both, seems similar for now, but asking just in case. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 28, 2019, 2:24pm UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974/11 "2019-01-28T14:24:38Z")
The size your data takes up on disk depends on [the mappings used](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/tune-for-disk-usage.html) but also the number of shards your index has, as compression typically improves as the shard size grows. ---
**Author:** ![Aralex](https://avatars.discourse-cdn.com/v4/letter/a/e56c9b/32.png) [@Aralex](https://discuss.elastic.co/u/Aralex)\ **Post date:** [January 28, 2019, 2:26pm UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974/12 "2019-01-28T14:26:38Z")
Thanks for the help Christian, I'll have a look at the link. We can consider this case closed. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 25, 2019, 2:26pm UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974/13 "2019-02-25T14:26:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.