Hello everybody,
I've recently installed Elastic, Logstash and Kibana to parse apache's logs. But, logstash seems sending data only when I run the following command :
sudo /usr/share/logstash/bin/logstash --debug --path.settings /etc/logstash/ -f /etc/logstash/conf.d/01-xenilabslogs.conf
Logstash is enabled (sudo systemctl enable logstash.service) and started : (sudo systemctl start logstash.service). I've checked with ps aux | grep logstash and I've the following result :
logstash 4468 193 3.9 3643764 472628 ? SNsl 15:02 0:42 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 (... with several parameters)
Elastic reply with successfull at localhost:9200 (curl)
Elastic's indices in image:
Content of /etc/logstash/conf.d/01-xenilabslogs.conf :
input {
file {
path => ["/var/log/apache2/access_xenilabs.log"]
type => "apache_access"
}
file {
path => ["/var/log/apache2/error.log"]
type => "apache_error"
}
}
filter {
if [type] in [ "apache" , "apache_access" , "apache-access" ] {
grok {
match => [
"message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}",
"message" , "%{COMMONAPACHELOG}+%{GREEDYDATA:extra_fields}"
]
overwrite => [ "message" ]
}
mutate {
convert => ["response", "integer"]
convert => ["bytes", "integer"]
convert => ["responsetime", "float"]
}
geoip {
source => "clientip"
target => "geoip"
add_tag => [ "apache-geoip" ]
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
remove_field => [ "timestamp" ]
}
useragent {
source => "agent"
}
}
if [type] in ["apache_error","apache-error"] {
grok {
match => ["message", "[%{WORD:dayname} %{WORD:month} %{DATA:day} %{DATA:hour}:%{DATA:minute}:%{DATA:second} %{YEAR:year}] [%{NOTSPACE:loglevel}] (?:[client %{IPORHOST:clientip}] ){0,1}%{GREEDYDATA:message}"]
overwrite => [ "message" ]
}
mutate
{
add_field =>
{
"time_stamp" => "%{day}/%{month}/%{year}:%{hour}:%{minute}:%{second}"
}
}
date {
match => ["time_stamp", "dd/MMM/YYYY:HH:mm:ss"]
remove_field => [ "time_stamp","day","dayname","month","hour","minute","second","year"]
}
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "apache-%{+YYYY.MM.dd}"
document_type => "system_logs"
}
stdout { codec => rubydebug }
}
-- end of file --
The rights of /var/log/apache2/access_xenilabs.log ( -rw-r----- 1 root adm )
Content of tail /var/log/logstash/logstash-plain.log
[2020-05-25T16:41:24,935][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-05-25T16:41:24,998][INFO ][filewatch.observingtail ][main][0e84060726dd1047b88ea007bcb3f18880b9ab9327b3a03db5bf174ba548b8b2] START, creating Discoverer, Watch with file and sincedb collections
[2020-05-25T16:41:25,004][INFO ][filewatch.observingtail ][main][cb78811e9c064f0f8b08e8b2aa16d7cb541bd3ad9a82814e9105a335d7ac390b] START, creating Discoverer, Watch with file and sincedb collections
[2020-05-25T16:41:25,065][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2020-05-25T16:41:25,693][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-05-25T16:44:27,404][WARN ][logstash.runner ] SIGINT received. Shutting down.
[2020-05-25T16:44:27,527][INFO ][filewatch.observingtail ] QUIT - closing all files and shutting down.
[2020-05-25T16:44:27,566][INFO ][filewatch.observingtail ] QUIT - closing all files and shutting down.
[2020-05-25T16:44:28,817][INFO ][logstash.javapipeline ] Pipeline terminated {"pipeline.id"=>"main"}
[2020-05-25T16:44:28,959][INFO ][logstash.runner ] Logstash shut down.
-- end of content : tail --
Could you help me to automaticly to send data please ? Thank in advance,
Lucas