Logstash pauses on import and needs manual intervention

[Consolodated_Emails (Dustin)]

This is a bit of a weird issue.

I'm using Logstash to ingest gziped CSVs from S3 into Elasticsearch. It's a known issue that our data has hidden ASCII control characters (EOF, Pause, Break, etc) in it. I'm not sure if this is the cause of my issue but it could be. I've seen similar problems with other data tools.

I've recently setup Logstash to move our data into Elasticsearch. When I start Logstash manually in a terminal, I see the CSV rows fly by, every once and a while it pauses and wont continue until I hit return it continues with the ingestion. If I do this enough times, it will complete without errors. I thought it was a issue with me starting it manually in the terminal. I figured it was sending a pause and the terminal session was picking that up. But I disconnected the terminal session and it still got paused.

Does anyone know of a way to override this behavior? Obviously I can't babysit every ingestion.

Much appreciated..

[atira (Attila Boncok)]

Not sure what happens there, just wondering about this part:

@Consolodated_Emails

But I disconnected the terminal session and it still got paused.

Did you start Logstash as a service or as a regular process? If the latter, did you start the process with nohup/disown or without it?

[Consolodated_Emails (Dustin)]

I figured out the issue was using the stdout { } output. I expected that to just post to the terminal but it appears to be interactive. So what I think happened is there is a pause ASCII character (or similar) somewhere in my text. I guess when it posts to the terminal session that causes the script to pause (??)..

When I disabled stdout { } the ingestion runs until completion with no intervention necessary.