This is a bit of a weird issue.
I'm using Logstash to ingest gziped CSVs from S3 into Elasticsearch. It's a known issue that our data has hidden ASCII control characters (EOF, Pause, Break, etc) in it. I'm not sure if this is the cause of my issue but it could be. I've seen similar problems with other data tools.
I've recently setup Logstash to move our data into Elasticsearch. When I start Logstash manually in a terminal, I see the CSV rows fly by, every once and a while it pauses and wont continue until I hit return it continues with the ingestion. If I do this enough times, it will complete without errors. I thought it was a issue with me starting it manually in the terminal. I figured it was sending a pause and the terminal session was picking that up. But I disconnected the terminal session and it still got paused.
Does anyone know of a way to override this behavior? Obviously I can't babysit every ingestion.
Much appreciated..