"/var/ao/FileDrop/ecsv/3f0faf5f-1c1a-0a73-4751-2b266073e136.out.csv", :text=>"TlZWbU1VcDRUREk1YWpWcmRrVlBhUT0981FXHh6MObMq+4B85WRP0j+Wze04F+gAjMa+rZ0wCr0=,outlook.com,86,1,2020-06-23 01:57:01 AM,45,0.7,219,12,443142|446120|511210|448120\r"}
```
Nothing in the log indicated that records were being dropped. There were no errors. But imports a variable number of records each time with the same file.
Import 1: 440,668
Import 2: 211,840
Import 3: 417,153
Import 4: 740,142
I have tried it with different files arriving at the same out come with messages are being dropped. This is a single node instance. There are 8 cores. I have tried varying 1-8 workers. I have adjusted the batch size, but still messages are being dropped.
- The input files are generated in our analytics cluster and as such the format is controlled.
- The disk is 50% full.
- 500-ish indices on server
I AM STUMPED and about 24 hours into debugging this thing. I have tried everything I know to check.
Thought: The last field of the CSV is call "codes" sometimes it has values, sometimes it does not. There does not seem to be a correlation to the records that are kept/dropped.
Help.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [October 10, 2020, 9:23pm UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/2 "2020-10-10T21:23:20Z")
I would start by using a file output, see if all 1 million records get as far as the output section.
Do you have any ecsv-\* indexes you do not expect? Could the metadata be wrong?
---
### Author:  [@Paul\_Shriner](https://discuss.elastic.co/u/Paul_Shriner)
#### Post date: [October 10, 2020, 9:44pm UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/3 "2020-10-10T21:44:58Z")
OK, that is good idea in regards to the file output.
The indexes are set based on the inbound file name. I did check and I don't see any new "Surprise" indices.
---
### Author:  [@Paul\_Shriner](https://discuss.elastic.co/u/Paul_Shriner)
#### Post date: [October 10, 2020, 10:02pm UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/4 "2020-10-10T22:02:45Z")
We have crossed file output off the list of possible solutions.
File output had 569,336 lines in it.
I did not run it in debug mode, but do have a similar outcome.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [October 10, 2020, 10:14pm UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/5 "2020-10-10T22:14:25Z")
Next I would comment out the ruby filter and see if that fixes the problem.
---
### Author:  [@Paul\_Shriner](https://discuss.elastic.co/u/Paul_Shriner)
#### Post date: [October 10, 2020, 10:20pm UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/6 "2020-10-10T22:20:57Z")
Ruby code commented out.
647,444 records
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [October 10, 2020, 10:57pm UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/7 "2020-10-10T22:57:03Z")
Obviously if you are seeing DEBUG messages you could not have missed 300,000 mapping exceptions in the logstash log, but is elasticsearch logging anything of interest? I am about out of suggestions here.
---
### Author:  [@Paul\_Shriner](https://discuss.elastic.co/u/Paul_Shriner)
#### Post date: [October 11, 2020, 12:36am UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/8 "2020-10-11T00:36:22Z")
I don't think so, AND this is not new code, It has been running wtihout issue for the last 6-ish months.
This led me to the conclusion that it may some way related to the environment. We have ramped up usage and then suddenly started to see events drop.
On here, we were able to isolate this to being a Logstash and not ES issue.
From an environmental perspective, this is RAM usage.
```auto
total used free shared buff/cache available
Mem: 7970 4205 243 7 3522 3462
Swap: 1023 166 857
```
Because it is a small machine, we allocated 2G to LS
```auto
## JVM configuration
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms2g
-Xmx2g
```
^--- Does that look correct?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [October 11, 2020, 1:00am UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/9 "2020-10-11T01:00:55Z")
I routinely run logstash with a 400 MB heap. But that is with _very_ simple pipelines. If it ran out of memory I would expect exceptions to be logged.
Are all the records in the csv? Why do you have a tcp input? Could things be backed up enough that even a tcp input is losing data?
---
### Author:  [@Paul\_Shriner](https://discuss.elastic.co/u/Paul_Shriner)
#### Post date: [October 11, 2020, 1:20am UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/10 "2020-10-11T01:20:07Z")
Everything is coming in from the CSV file.
The only reason that that TCP input is there is because when I first wrote this file a year ago, the example included it. #embarassed
I just removed it and re-ran
539,416 records
So, while a cleaner config file, not the solution.
I have a pre-processor that i run to encrypt email addresses. I could wipe out "n/a" issue and then it would be a straight import.
---
### Author:  [@Paul\_Shriner](https://discuss.elastic.co/u/Paul_Shriner)
#### Post date: [October 11, 2020, 1:25am UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/11 "2020-10-11T01:25:00Z")
How do I discover the LS backlog?
I just found this thread, while older, I have been playing with batch size (100 - 3000 )
> [@Losing messages during high traffic rate](https://discuss.elastic.co/t/losing-messages-during-high-traffic-rate/56449):
>
> I am doing log analysis using Filebeat (1.2) -\> logstash(2.3) -\> Elasticsearch (2.3) I have 4 filebeat instances, 4 logstash instances (6 cores each) , and Elasticsearch cluster (8 cores, 64G RAM) of 2 nodes In Filebeat.yml logstash setting is as such: filebeat: prospectors: - paths: - /var/log/filebeat//.json encoding: utf-8 input\_type: log ignore\_older: 10m scan\_frequency: 1s exclude\_lines: ["^$"] spool\_size: 3072 registry\_file: .filebeat output: logstash: enabled: true hos…
That seems to be a dead end as well.
---
### Author:  [@Paul\_Shriner](https://discuss.elastic.co/u/Paul_Shriner)
#### Post date: [October 11, 2020, 1:38am UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/12 "2020-10-11T01:38:17Z")
Could queue type be a factor?
>
>
> We already have Persistent Queue but this queue type will still block if it's full.
> Another queue type for the purpose of...
---
### Author:  [@Paul\_Shriner](https://discuss.elastic.co/u/Paul_Shriner)
#### Post date: [October 11, 2020, 2:03am UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/13 "2020-10-11T02:03:35Z")
Think that you were right @Badger, It was the backpressure! I put a 50 ms delay between batches and got the full file import.
1,055,181
Who has two thumbs and is excited that this particular nightmare is over???
This guy!
---
### Author:  [@Paul\_Shriner](https://discuss.elastic.co/u/Paul_Shriner)
#### Post date: [October 11, 2020, 3:52am UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/14 "2020-10-11T03:52:09Z")
It worked once, and now back to the same place.
ARG
---
### Author:  [@Paul\_Shriner](https://discuss.elastic.co/u/Paul_Shriner)
#### Post date: [October 11, 2020, 2:56pm UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/15 "2020-10-11T14:56:26Z")
OK, so far today:
- Commented out entire conf file and still output to file
- Turned on persistent queue
- Cut number of workers in half
Still a variable output for line count on the output file
---
### Author:  [@Paul\_Shriner](https://discuss.elastic.co/u/Paul_Shriner)
#### Post date: [November 5, 2020, 3:41am UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/16 "2020-11-05T03:41:55Z")
So interesting update, I rewrote the ingestion using the Batch API. I was ingesting a file of 74MM records and it randomly dropping events. Sees like something other that LS...
---
### Author:  [@Paul\_Shriner](https://discuss.elastic.co/u/Paul_Shriner)
#### Post date: [November 5, 2020, 5:18pm UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/17 "2020-11-05T17:18:57Z")
To provide a little more detail, I was attempting to import 74MM through the batch API, only 60 million showed in the index. I dropped the index and reimported, this time 68 Million were imported, but still not 74. When I walk the reply for logging, no error are created. I pass all 74 MM records in, but still the drop.
Because this is happening in both simplified LS and the Batch API, it leads me to the conclusion it is likely a resource constrained server. Hard disk is 82%, and logs can be trimmed and some indices could be dropped.
SWAP is at 98%, which is concerning.
LS is no longer running on the box.
There are not a lot of operations running on the box at once.
Help.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 3, 2020, 5:19pm UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658/18 "2020-12-03T17:19:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.