Looking for a way to ingest a file and remove it

Elastic Stack Logstash
1

i created a logstash conf file while back and it ingest the filr and removed it.
back at the time i did not wanted this behavior but now i do...
i only forgot how to do it :slight_smile:

can someone direct me to the correct syntax?

2

Use a file input in read (not tail) mode and set file_completed_action to "delete".

4

my config looks like this

input {
  file {
    path => "/data/elastic/*.json"
    mode => "read"
    file_completed_action => "delete"
    type => "json"
    codec => "json"
    start_position => "beginning"
    file_completed_log_path => "/var/lib/logstash/logged_file.list"
    sincedb_path => "/dev/null"
  }
}
output {
  stdout { codec => "dots"}
  elasticsearch {
    hosts => ["10.10.10.10:9200"]
    index => "jenkins"
    document_type => "_doc"
  }
}

i got a nasty error

warning: thread "Converge PipelineAction::Create" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>
create at org/logstash/execution/ConvergeResultExt.java:135
add at org/logstash/execution/ConvergeResultExt.java:60
converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:404
[ERROR] 2021-11-28 23:31:03.685 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>"}
[FATAL] 2021-11-28 23:31:03.696 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in create'", "org/logstash/execution/ConvergeResultExt.java:60:in add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:404:in `block in converge_state'"]}
[FATAL] 2021-11-28 23:31:03.704 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.19.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.19.0.jar:?]
at usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap

5

If you enable log.level debug do you get a more informative message?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.