Hi Hussein,
As said by Mark, your logs needs to be parsed, each line being analyzed and broken in fields
before being sent to Elasticsearch, this way your queries should be more straightforward.
Usually you run a "log shipper" ( a 'filebeat' process for log files in ELK stack) on each production machine, this
shipper will send your logs to a logstash process running on one of your computers, preferably near your ES cluster.
The filebeat will handle all tomcat logs produced by your application, "remembers" where it stops, can be restarted and handles log file rotations.
The logstash process on it's side will parse all log data it receives, broke lines in fields before sending them to your ES cluster.
So, to recap, log ingestion is two processes : filebeat on production machine to send logs, logstash to receive logs,
parse them and send to ES cluster.
With each filebeat process you have a yaml config file like this one (change paths),
assuming logstash is configured to listen on port 5044 (see logstash.conf config filelater)
Filebeat.yaml
# run with filebeat-1.2.3-x86_64/filebeat -c filebeat.yml
filebeat:
prospectors:
-
paths:
- "/xxxFullPathHerexx-app/logs/server.log*"
multiline:
#pattern: '\]$'
pattern: '^\['
negate: true
match: after
input_type: log
document_type: beat
registry: /xxxFullPathHerexx/registry
output:
logstash:
hosts: ["MyLogstashComputerHere:5044"]
logging:
to_files: true
files:
path: /xxxFullPathHerexx-app/
name: filebeat
rotateeverybytes: 10485760
level: error
The logstash config file should be something like the following
(the grok pattern is for a glassfish application log, not tomcat,
you have to change it to suit your log format,
plus it depends on the log pattern used with log4j in your application,
and you should also change the field names on the match => line)
logs-glassfish.conf
see ES template in logs-template.json
to run it ... logstash-2.3.4/bin/logstash -f logs-glassfish.conf
nb: send logs with netcat:
nc localhost 4560 < logs/server.log
input {
beats {
type => "engine"
port => 5044
}
}
filter {
customize the match line
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:cTime}|%{LOGLEVEL:logLevel}|%{DATA:application}|%{DATA:class}|_ThreadID=%{NUMBER:threadID};_ThreadName=%{DATA:threadName};|%{DATA:message}" }
}
}
output {
uncomment to debug or see dots
#stdout { codec => rubydebug }
#stdout { codec => dots }
elasticsearch {
hosts => "http://elk1:9200"
index => "logs-%{+YYYY.MM}"
template => "./logs-template.json"
template_name => "logs"
template_overwrite => true
flush_size => 10000
}
}
the template.json is the mappings template to use with your logs.
you can remove the lines if you don’t want to overwrite it each time
the idea is to extract fields you want (like User ID ) in match pattern.
HTH
regards,
Alain
