Logstash - File output error - permission denied

Elastic Stack Logstash
1

Hello Team,

I am trying to collect a set of data from a URL and save it in a file as json format. below is my logstash configuration file . Kindly help me with troubleshooting where and what i am missing. I am getting error with permission for storing file in output i guess.

  • i am running logstash as root user
  • i gave chmod 777 permission to the folder where i am saving my output file.
  • I tried with both without creating the file i mentioned in output and also with file mentioned in the output. (along with highest permission - chmod 766)

input {
http_poller {
urls => {
minemeld => "https://x.x.x.x/feeds/DARP-output-feedgreen?tr=1"
user => "xxxx"
password => "xxxxx"
schedule => { "every" => "1h" }
}
}
}

output {
file {
path => "/root/ioc/minemeldoutput.json"
codec => "json"
}
}

I am getting below error

[2019-03-26T22:32:48,055][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<Errno::EACCES: Permission denied - /root/ioc/minemeld.json>, :backtrace=>["org/jruby/RubyDir.java:449:in mkdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:253:infu_mkdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:227:in block in mkdir_p'", "org/jruby/RubyArray.java:1818:inreverse_each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:225:in block in mkdir_p'", "org/jruby/RubyArray.java:1734:ineach'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:211:in mkdir_p'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:273:inopen'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:136:in block in multi_receive_encoded'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:135:in block in multi_receive_encoded'", "org/jruby/ext/thread/Mutex.java:148:insynchronize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:134:in multi_receive_encoded'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:87:inmulti_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:390:in block in output_batch'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:389:in output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:inworker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:304:in `block in start_workers'"]}
[2019-03-26T22:32:48,198][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

My ultimate aim is to collect the data from URL and store it in json/csv file. then use the file inside a translate filter in different logstash conf file.

I am stuck with first step of collecting data into a file.

2

If you take at look at the code, you will see that it is trying to create a directory. Immediately before that error message there should be an INFO message telling you which directory it is trying to create (I would expect /root/ioc/).

Are you using docker?

3

Hello badger,

No i am using just a standalone logstash instance. Ioc is a directory i created . Minemeldouput is the file i wantbthe data to be stored.

How does file output works ? Will it create file automatically or i should provide the empty file ? I tried both but no luck :frowning:

What should i provide in path field ?

My goal is to get the data in URL into either CSV or json file.

Thanks,

Haran

4

The file output will create the directory and the file if it needs to. You did not answer the other question I asked about the INFO message.

5

below is the complete error log. pls check and let me know what configuration i am missing.

  • I have created a folder IOC in /root
  • IOC folder has full permission chmod 777
  • i even tried path => /etc/logstash/minemeldoutput.json but still same permission error for logstash folder as well

[2019-03-26T22:30:39,066][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.2"}
[2019-03-26T22:30:45,967][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-03-26T22:30:46,107][INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :schedule=>{"every"=>"2m"}, :timeout=>nil}
[2019-03-26T22:30:46,154][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x1aaa6bfb run>"}
[2019-03-26T22:30:46,239][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2019-03-26T22:30:46,788][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-03-26T22:32:47,880][INFO ][logstash.outputs.file ] Opening file {:path=>"/root/ioc/minemeldoutput.json"}
[2019-03-26T22:32:47,885][INFO ][logstash.outputs.file ] Creating directory {:directory=>"/root/ioc"}
[2019-03-26T22:32:48,055][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<Errno::EACCES: Permission denied - /root/ioc>, :backtrace=>["org/jruby/RubyDir.java:449:in mkdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:253:infu_mkdir'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:227:in block in mkdir_p'", "org/jruby/RubyArray.java:1818:inreverse_each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:225:in block in mkdir_p'", "org/jruby/RubyArray.java:1734:ineach'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:211:in mkdir_p'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:273:inopen'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:136:in block in multi_receive_encoded'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:135:in block in multi_receive_encoded'", "org/jruby/ext/thread/Mutex.java:148:insynchronize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-file-4.2.6/lib/logstash/outputs/file.rb:134:in multi_receive_encoded'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:87:inmulti_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:390:in block in output_batch'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:389:in output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:inworker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:304:in `block in start_workers'"]}
[2019-03-26T22:32:48,198][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

6

You say you created the directory IOC inside /root, but UNIX filenames are case sensitive, so /root/IOC is not the same as /root/ioc.

I don't know about your server, but on mine I was surprised to find that /root is not owner-writeable.

$ ls -ld /root
dr-xr-x---. 5 root root 194 Mar 25 10:36 /root

I would expect id 0 to bypass those permissions. It works on my server.

Using mode 777 is almost always a bad idea, even for debugging, even in a discardable lab environment. I would suggest

chmod u+w /root

and if that does not help then perhaps

chmod g+w /root

but if that does not help then

chmod g-w /root

The fundamental problem is that logstash believes it needs to create the directory /root/ioc and it gets a permissions error when it tries to do so.

I assume you are not in a hardened UNIX install where "root" is not id 0, or id 0 has limited permissions. I think you would know if you were.

Changing the filename to /tmp/mm.json might work, but is potentially less secure. It could help narrow down the issue tho'.

7

Thank you Badger. I will check these things once I am back with my lab machine. Keep you posted on the outcome.

Regards,

Haran

8

Hello Badger,

I tried /tmp/minemeldoutput.json as my file output filter but could not narrow down the issue.
My logstash status is running , but there is no update of anything in logstash-plain.log and also in logstash-slowlog-plain.log

Also no output in tmp folder.

Kindly help with the issue. also i tried sending the collected data to elasticsearch but no luck.
below is my logstash conf file. anyways output to a file would be my required method.

input {
http_poller {
schedule => { "every" => "2s" }
urls => {
minemeld => "https://192.168.56.30/feeds/DARP-output-feedgreen?tr=1"
}
}
}

output {
elasticsearch {
hosts => ["192.168.56.10:9200"]
index => "logstash-threat-intel-%{+YYYY.MM.dd}"
document_type => "threatintel"
}
}

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.