Logstash - File output error - permission denied

Hello Team,

I am trying to collect a set of data from a URL and save it in a file as json format. below is my logstash configuration file . Kindly help me with troubleshooting where and what i am missing. I am getting error with permission for storing file in output i guess.

  • i am running logstash as root user
  • i gave chmod 777 permission to the folder where i am saving my output file.
  • I tried with both without creating the file i mentioned in output and also with file mentioned in the output. (along with highest permission - chmod 766)

input {
http_poller {
urls => {
minemeld => "https://x.x.x.x/feeds/DARP-output-feedgreen?tr=1"
user => "xxxx"
password => "xxxxx"
schedule => { "every" => "1h" }
}
}
}

output {
file {
path => "/root/ioc/minemeldoutput.json"
codec => "json"
}
}

I am getting below error

[2019-03-26T22:32:48,055][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<Errno::EACCES: Permission denied - /root/ioc/minemeld.json>, :backtrace=>["org/jruby/RubyDir.java:449:in mkdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:253:infu_mkdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:227:in block in mkdir_p'", "org/jruby/RubyArray.java:1818:inreverse_each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:225:in block in mkdir_p'", "org/jruby/RubyArray.java:1734:ineach'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:211:in mkdir_p'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:273:inopen'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:136:in block in multi_receive_encoded'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:135:in block in multi_receive_encoded'", "org/jruby/ext/thread/Mutex.java:148:insynchronize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:134:in multi_receive_encoded'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:87:inmulti_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:390:in block in output_batch'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:389:in output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:inworker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:304:in `block in start_workers'"]}
[2019-03-26T22:32:48,198][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

My ultimate aim is to collect the data from URL and store it in json/csv file. then use the file inside a translate filter in different logstash conf file.

I am stuck with first step of collecting data into a file.

If you take at look at the code, you will see that it is trying to create a directory. Immediately before that error message there should be an INFO message telling you which directory it is trying to create (I would expect /root/ioc/).

Are you using docker?

Hello badger,

No i am using just a standalone logstash instance. Ioc is a directory i created . Minemeldouput is the file i wantbthe data to be stored.

How does file output works ? Will it create file automatically or i should provide the empty file ? I tried both but no luck :frowning:

What should i provide in path field ?

My goal is to get the data in URL into either CSV or json file.

Thanks,

Haran

The file output will create the directory and the file if it needs to. You did not answer the other question I asked about the INFO message.

below is the complete error log. pls check and let me know what configuration i am missing.

  • I have created a folder IOC in /root
  • IOC folder has full permission chmod 777
  • i even tried path => /etc/logstash/minemeldoutput.json but still same permission error for logstash folder as well

[2019-03-26T22:30:39,066][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.2"}
[2019-03-26T22:30:45,967][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-03-26T22:30:46,107][INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :schedule=>{"every"=>"2m"}, :timeout=>nil}
[2019-03-26T22:30:46,154][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x1aaa6bfb run>"}
[2019-03-26T22:30:46,239][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2019-03-26T22:30:46,788][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-03-26T22:32:47,880][INFO ][logstash.outputs.file ] Opening file {:path=>"/root/ioc/minemeldoutput.json"}
[2019-03-26T22:32:47,885][INFO ][logstash.outputs.file ] Creating directory {:directory=>"/root/ioc"}
[2019-03-26T22:32:48,055][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<Errno::EACCES: Permission denied - /root/ioc>, :backtrace=>["org/jruby/RubyDir.java:449:in mkdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:253:infu_mkdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:227:in block in mkdir_p'", "org/jruby/RubyArray.java:1818:inreverse_each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:225:in block in mkdir_p'", "org/jruby/RubyArray.java:1734:ineach'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:211:in mkdir_p'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:273:inopen'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:136:in block in multi_receive_encoded'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:135:in block in multi_receive_encoded'", "org/jruby/ext/thread/Mutex.java:148:insynchronize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:134:in multi_receive_encoded'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:87:inmulti_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:390:in block in output_batch'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:389:in output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:inworker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:304:in `block in start_workers'"]}
[2019-03-26T22:32:48,198][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

You say you created the directory IOC inside /root, but UNIX filenames are case sensitive, so /root/IOC is not the same as /root/ioc.

I don't know about your server, but on mine I was surprised to find that /root is not owner-writeable.

$ ls -ld /root
dr-xr-x---. 5 root root 194 Mar 25 10:36 /root

I would expect id 0 to bypass those permissions. It works on my server.

Using mode 777 is almost always a bad idea, even for debugging, even in a discardable lab environment. I would suggest

chmod u+w /root

and if that does not help then perhaps

chmod g+w /root

but if that does not help then

chmod g-w /root

The fundamental problem is that logstash believes it needs to create the directory /root/ioc and it gets a permissions error when it tries to do so.

I assume you are not in a hardened UNIX install where "root" is not id 0, or id 0 has limited permissions. I think you would know if you were.

Changing the filename to /tmp/mm.json might work, but is potentially less secure. It could help narrow down the issue tho'.

Thank you Badger. I will check these things once I am back with my lab machine. Keep you posted on the outcome.

Regards,

Haran

Hello Badger,

I tried /tmp/minemeldoutput.json as my file output filter but could not narrow down the issue.
My logstash status is running , but there is no update of anything in logstash-plain.log and also in logstash-slowlog-plain.log

Also no output in tmp folder.

Kindly help with the issue. also i tried sending the collected data to elasticsearch but no luck.
below is my logstash conf file. anyways output to a file would be my required method.

input {
http_poller {
schedule => { "every" => "2s" }
urls => {
minemeld => "https://192.168.56.30/feeds/DARP-output-feedgreen?tr=1"
}
}
}

output {
elasticsearch {
hosts => ["192.168.56.10:9200"]
index => "logstash-threat-intel-%{+YYYY.MM.dd}"
document_type => "threatintel"
}
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.