Logstash not stopping after completely reading the file


My logstash conf is as below.

input {
s3 {
access_key_id => ""
secret_access_key => ""
bucket => "tabprod-s3data"
region => "us-west-2"

filter {
json {
source => "message"
remove_field => ["message"]
#mutate {add_field =>{"channelId" => "indexid" }}

output {
amazon_es {
hosts => ["https://orlvialrzsqaufa5232n3rces.amazonaws.com"]
region => "us-west-2"
aws_access_key_id => ""
aws_secret_access_key => ""
index => "channel_%{channelId}"

I am writing a file from s3 to elasticsearch using logstash.

When I run logstash as a cli command, it works perfectly fine. ( Loads all 200k records from the file).

But If I run it as a service, the logstash keep on loading the same file again and again.
I have waited for long thinking some problem with the checkpointing, but it loaded till 9 folds and was still continuing.

Where and what should I be checking/modifying to avoid this behaviour?


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.