I need to lunch the command logstash -f test.conf with python script from another terminal.
So I'd like to lunch the command logstash -f test.conf and then I'd like to have the possibility to lunch other command. Instead I'm staying in wait after insert the sudo password.
Is it possible to configure logstash.yml to lunch at the start of the service loading contain in test.conf?
Thank you in advance
Franco
what do you have, what do you expect, and how is it not working?
running logstash via the command line shouldn't require sudo:
bin/logstash -f test.conf
However, most input plugins will listen or search for additional input until interrupted (stdin is an exception -- it will close itself out when it reaches end-of-input; I personally don't know much about communicating to subprocesses in python, but I found this which may help).
Hi
I'm using this command line for debian server
sudo ./logstash -f /home/franco/apache.conf --path.settings /etc/logstash
the apache.conf is the following
input
{
file {
path => "/home/franco/POC/elastic_logstash/apache.log"
type => "logs"
start_position => "beginning"
}
}
filter
{
grok {
match => {
"message" => "%{COMBINEDAPACHELOG}"
}
}
mutate {
convert => { "bytes" => "integer" }
}
date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
locale => en
remove_field => "timestamp"
}
geoip {
source => "clientip"
}
useragent {
source => "agent"
target => "useragent"
}
}
output
{
stdout {
codec => dots
}
elasticsearch {
hosts => ["serverElk.example.com:9200"]
index => "prova1-%{+YYYY.MM.dd}"
}
}
If I launch the command in the debian after completed the loading the system doesn't give me the input line, but it remains in waiting mode, so I must give a CTRL+c to block the process.
How could I configure apache.conf or lunch logstash -f in order to load the file and give me again the input line?
Thank you
Franco
See my previous post; I indicated that the stdin input will close itself when it reaches end-of-file, and linked to documentation on how to send input to a process's stdin with python.
Hi
I will try to use stdin, configuring a port to read the file. I will post my result in the next hours
Thank you
Franco
Excuse me for the delay...
I did what you do and I have the first problem
/usr/share/logstash/bin/logstash -f apache_logstash_test.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-03-19 23:57:23.393 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[INFO ] 2018-03-19 23:57:23.420 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[FATAL] 2018-03-19 23:57:23.538 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:448:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:230:invalidate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:141:in block in validate_all'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:140:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:264:inexecute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:219:inrun'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:67:in'"]}
[ERROR] 2018-03-19 23:57:23.556 [main] Logstash - java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit
I add the --path.settings /etc/logstash and lunch again the command
I have a new error
/usr/share/logstash/bin/logstash -f apache_logstash_test.conf --path.settings /etc/logstash
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
2018-03-20 00:01:06,142 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.(FileOutputStream.java:213)
at java.io.FileOutputStream.(FileOutputStream.java:133)
at org.apache.logging.log4j.core.appender.rolling.RollingF.....
The problem is the permission so I need sudo
So I lunch again the command with sudo
sudo /usr/share/logstash/bin/logstash -f apache_logstash_test.conf --path.settings /etc/logstash and the process starts but I don't have the possibility to enter a new command
The configuration file has the stdin
Here the log file of logstash
franco@serverElk:~$ tail -f /var/log/logstash/logstash-plain.log
[2018-03-20T00:04:15,694][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/home/franco/scenario2_logstash/apache_template.json"}
[2018-03-20T00:04:15,712][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"apache_elastic_example", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"mapping"=>{"norms"=>false, "type"=>"text"}, "match_mapping_type"=>"string", "match"=>"message"}}, {"string_fields"=>{"mapping"=>{"norms"=>false, "type"=>"text", "fields"=>{"keyword"=>{"ignore_above"=>256, "type"=>"keyword"}}}, "match_mapping_type"=>"string", "match"=>"*"}}], "properties"=>{"geoip"=>{"dynamic"=>true, "properties"=>{"location"=>{"type"=>"geo_point"}, "ip"=>{"type"=>"ip"}, "continent_code"=>{"type"=>"keyword"}, "country_name"=>{"type"=>"keyword"}}, "type"=>"object"}, "@version"=>{"type"=>"keyword"}}}}}}
[2018-03-20T00:04:15,737][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/apache_elastic_example
[2018-03-20T00:04:15,924][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//10.0.2.10:9200"]}
[2018-03-20T00:04:16,276][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2018-03-20T00:04:17,199][INFO ][logstash.inputs.tcp ] Starting tcp input listener {:address=>"0.0.0.0:5000", :ssl_enable=>"false"}
[2018-03-20T00:04:17,838][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x537e73d6@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}
[2018-03-20T00:04:17,860][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:5000"}
[2018-03-20T00:04:17,906][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-03-20T00:04:17,963][INFO ][logstash.inputs.udp ] UDP listener started {:address=>"0.0.0.0:5000", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
And configuration file apache_logstash_test.conf
ranco@serverElk:~/scenario2_logstash$ more apache_logstash_test.conf
input {
stdin {
}
}
So in order to lunch date from the stdin I blocked this process and re lunch with cat name_file | /usr/share/logstash/bin/logstash -f apache_logstash_test.conf --path.settings /etc/logstash
With this, at the finished of loading I have the command line. Now how could I do not use the sudo?
Thank you
Franco
You may need super-user to correct the permissions on the directory, but you definitely do not need to run Logstash as the super user.
You need to modify the permissions of the directory to grant write access to the user account under which Logstash is running.
What are the permissions of /var/log/logstash and /var/log/logstash/logstash-plain.log if it exists? You can get a view of file metadata, including ownership and permissions, by using ls with the -la flags:
ls -la /var/log/logstash
ls -la /var/log/logstash/logstash-plain.log
Since your user, franco is running the command directly, we may need a list of the groups to which you are assigned with the groups command; setting permissions in such a way that the logstash user and franco both have access will enable us to run Logstash either as a service or directly.
Thank you.
I see that logstash user is the owner of the folder and files.
I could use logstash user, but what is the default password?
I try to logon with the logstash user and password equal to user but I received the message "incorrect password".
I think that if I reset the password of logstash, then it is possible that I have problem to the service or other component
Have a good day
Franco
Alternatively, you could add yourself to the logstash group, which will enable you to use the group permissions for the file:
sudo adduser franco logstashThis topic was automatically closed 28 days after the last reply. New replies are no longer allowed.