hi,
i'm new with ELK and is to be able to configure it to receive syslog messages. Ultimately we would like logstash to receive cloudtrail files, to do this I tried to follow Parsing Amazon CloudTrail JSON Logs with a Customized Logstash Build but for testing purposes we locally stored a json file and use this as an input file. but when i ran:
java -jar logstash-1.4.07.dev.medidata-flatjar.jar agent -f /etc/logstash/conf.d/combined.conf --debug
this resulted into and gets stuck with the last line
Automatic template management enabled {:manage_template=>"true", :level=>:info, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/logstash/outputs/elasticsearch_http.rb", :line=>"104"}
Template Search URL: {:template_search_url=>"http://localhost:9200/_template/*", :level=>:debug, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/logstash/outputs/elasticsearch_http.rb", :line=>"112"}
_discover_file_glob: /etc/logstash/conf.d/cloudtrail.json: glob is: ["/etc/logstash/conf.d/cloudtrail.json"] {:level=>:debug, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/filewatch/watch.rb", :line=>"117"}
researching i found this to be related to permission and deletig .sincedb* file but this didn't help. below is my is my config
input {
file {
type => "json"
path => "/etc/logstash/conf.d/cloudtrail.json"
start_position => beginning
}
}
filter {
if !("splitted" in [tags])
{
json {
source => "message"
}
split {
field => "Records"
reuse_element => false
refilter => true
add_tag => ["splitted"]
}
}
if ("splitted" in [tags])
{
date {
match => ["eventTime", "ISO8601"]
remove_tag => ["splitted"]
}
# workaround for the problem that responseElements chokes Elasticsearch. It expects a string but responseElements is sometimes a hash.
ruby {
code => "event['null'] = event['null'].inspect"
}
}
}
output {
elasticsearch_http { host => localhost }
stdout {
debug => true
codec => rubydebug
}
}
any help or guidance is very much appreciated.
Just to be clear: what problem do you want help with? The message in the topic's subject ('glob is []' etc) isn't the same as the rest of your question ('glob is: ["/etc/logstash/conf.d/cloudtrail.json"]').
To have Logstash read a file from the beginning, make sure you have start_position => beginning and that no sincedb state is saved about the file. Deleting ~logstash/.sincedb* should do but assumes that Logstash isn't running. Logstash will log information about the sincedb file and the offsets used.
Instead of using a special (outdated) Logstash build from a third party I'd look into the standard cloudtrail codec.
hi magnus,
appreciate your response!
i just deleted the ones inside the bracket to shorten the subject line. i'm really concerned and stuck with
discoverfile_glob: /etc/logstash/conf.d/cloudtrail.json: glob is: ["/etc/logstash/conf.d/cloudtrail.json"] {:level=>:debug, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/filewatch/watch.rb", :line=>"117"}
in my config i already have the start_position => beginning and have deleted the .sincedb* that is being created at the root directory whenever i ran logstash. but still same behavior was encountered.
as for cloudtrail codec, do you know any working config of cloudtrail codec that i can use as a reference? i have been searching for its documentation but fails to find one.
in my config i already have the start_position => beginning and have deleted the .sincedb* that is being created at the root directory whenever i ran logstash. but still same behavior was encountered.
Was Logstash shut down when you deleted the sincedb files? What do the logs say about the sincedb files used?
as for cloudtrail codec, do you know any working config of cloudtrail codec that i can use as a reference? i have been searching for its documentation but fails to find one.
I don't know. I get the impression that it's zero-config and supposed to be used together with an s3 input.
yes the logstash service is stopped when .sincedb* is deleted. at the debug it is creating a new .sincedb file and checking if the json file grew before proceeding.
Adding type with date config {:type=>"", :field=>"eventTime", :format=>"ISO8601", :level=>:debug, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/logstash/filters/date.rb", :line=>"165"}
Pipeline started {:level=>:info, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/logstash/pipeline.rb", :line=>"78"}
_sincedb_open: /root/.sincedb_cc8d6ba6fe00dea7817c06dc10aeeac2: No such file or directory - /root/.sincedb_cc8d6ba6fe00dea7817c06dc10aeeac2 {:level=>:debug, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/filewatch/tail.rb", :line=>"195"}
_discover_file_glob: /etc/logstash/conf.d/cloudtrail.json: glob is: ["/etc/logstash/conf.d/cloudtrail.json"] {:level=>:debug, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/filewatch/watch.rb", :line=>"117"}
_discover_file: /etc/logstash/conf.d/cloudtrail.json: new: /etc/logstash/conf.d/cloudtrail.json (exclude is ) {:level=>:debug, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/filewatch/watch.rb", :line=>"126"}
_open_file: /etc/logstash/conf.d/cloudtrail.json: opening {:level=>:debug, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/filewatch/tail.rb", :line=>"98"}
/etc/logstash/conf.d/cloudtrail.json: initial create, no sincedb, seeking to beginning of file {:level=>:debug, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/filewatch/tail.rb", :line=>"133"}
writing sincedb (delta since last write = 1438908755) {:level=>:debug, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/filewatch/tail.rb", :line=>"177"}
/etc/logstash/conf.d/cloudtrail.json: file grew, old size 0, new size 16721 {:level=>:debug, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/filewatch/watch.rb", :line=>"81"}
Automatic template management enabled {:manage_template=>"true", :level=>:info, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/logstash/outputs/elasticsearch_http.rb", :line=>"104"}
Template Search URL: {:template_search_url=>"http://localhost:9200/_template/*", :level=>:debug, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/logstash/outputs/elasticsearch_http.rb", :line=>"112"}
_discover_file_glob: /etc/logstash/conf.d/cloudtrail.json: glob is: ["/etc/logstash/conf.d/cloudtrail.json"] {:level=>:debug, :file=>"/etc/logstash/conf.d/logstash-1.4.07.dev.medidata-flatjar.jar!/filewatch/watch.rb", :line=>"117"}
Okay, so the file input is reading from the beginning of the file. But you're saying that you're not getting anything on stdout or to Elasticsearch? What if you comment out the elasticsearch output so that you only have the stdout output?
Hi Magnus,
We can now close this, I tweak my config and eliminate the filter and instead point codec to json.