Logstash service behaving different than command line

chrisan · October 15, 2016, 12:54pm

Hello, I have 2 cloudfront distributions I am looking to analyze. The system is a fresh VM of Ubuntu 16.04 with latest Elasticsearch and Logstash installed via apt-get

I have the following 2 configs:

gist.github.com

https://gist.github.com/chrisan/7312f5a894a49e0a3164699ccd4ed4a0

companylogs.conf

# /etc/logstash/conf.d/userlogs.conf
input {

  s3 {
     bucket => "example-logs"
     delete => false
     interval => 60 # seconds
     prefix => "user-logs/"
     region => "us-east-1"
     type => "cloudfront"

This file has been truncated. show original

userlogs.conf

#/etc/logstash/conf.d/companylogs.conf
input {

  s3 {
     bucket => "example-logs"
     delete => false
     interval => 60 # seconds
     prefix => "company-logs/"
     region => "us-east-1"
     type => "cloudfront"

This file has been truncated. show original

In both cases I started with the CLOUDFRONT_ACCESS_LOG from https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/aws and added in some matches to the URL to perform some aggregates on.

When I run both of these via the command line

$ /opt/logstash/bin/logstash -f /etc/logstash/conf.d/userlogs.conf
and
$ /opt/logstash/bin/logstash -f /etc/logstash/conf.d/companylogs.conf

There are no parse errors and everything is coming out fine in elasticsearch queries.

I then reset everything with curl -XDELETE localhost:9200/* && sudo rm -f /var/lib/logstash/.sincedb*

I start the service sudo service logstash start and wait a few moments then try my search again, but this time I have tons of _grokparsefailure and _dateparsefailure listed in tags.

gist.github.com

https://gist.github.com/chrisan/91d6877876630c2c36dcb36218f8b767

post.json

{
  "size": 10,
  "query": {
    "filtered": { 
      "filter": {
          "bool": {
              "must": {"term": { "tags": "_grokparsefailure" }}
          }
        
      }

This file has been truncated. show original

results.json

{
   "took": 5,
   "timed_out": false,
   "_shards": {
      "total": 10,
      "successful": 10,
      "failed": 0
   },
   "hits": {
      "total": 1766851,

This file has been truncated. show original

In the above you can see the search returned a type of company_logs but appears to be trying to grok with logs from the user cdn. There is 100% no path of /v1/users on the company CDN

Additionally while trying to debug the service you can see I added a raw_data field that just returns the original message add_field => [ "raw_data", "%{message}" ] What is odd is in the parse failures the raw_data is spitting back out "%{message}" while when I run these configs from the command line that doesn't happen.

Right now I can only assume running these configs from the command line work because they are in isolation of each other being run one at a time, while the logstash service is loading both of the configs files and doing them in tandem.

What am I missing with setting up logstash as a service? I do not see any other config files in /etc/logstash

Christian_Dahlqvist · October 15, 2016, 2:20pm

When you run Logstash as a service, it will concatenate both configuration files into one and all events will go through all filters as you are not using conditionals. Add a tag in each input so you can use this to apply the correct filters and output to each event using conditionals.

There are quite a few examples of this type of configuration in these forums, e.g. this one.

chrisan · October 15, 2016, 2:45pm

Thank you Christian for the explanation and links. Adding a conditional to each block worked as desired

Much appreciated