Logstash parses configs but not as service

[Callahan]

Hi,

So this is very odd. I have a small conf file that pulls data from log file and does a little mutating, and key/value pair matching then outputs to elastic. That conf is shown below.

input {
      file {
        type => "log"
    	path => "C:\ELK\Import_Logs_Folder\Events.log"
    	start_position => beginning
    	sincedb_path => "/dev/null"
      }
    }

    filter {
      if [type] == "log" {
      kv {
      field_split => ";"
      trimkey => "\s"
         }
      }
      
    mutate {
    	gsub => [ "UserName", "^(.*)[\\\\]", " " ]
    	remove_field => [ "message" ]
    	strip => [ "UserName" ]
    	gsub => [ "ComputerIPAddress", "\r", "" ]
    }
    date {
               match => [ "InsertedAt", "YYYY-MM-dd HH:mm:ss" ]
               target => "@timestamp"
         }
    }
    output {
    	if [type] == "sophoslog" {
        elasticsearch {
        index => "log-%{+YYYY.MM.dd}"
        hosts => ["172.16.30.12:9200"]
    	user => elastic
        password => changeme
        }
    }
    #stdout { codec => rubydebug }
    }

If I remove the # from stdout and run it manually, I can see the output all formatted correctly and it is inserted into Elastic. If I then put the conf file into my configs folder along with my other confs and restart the Logstash service, Logstash dumps the index into Elastic but it looks like this:

{
  "log-2017.06.14": {
    "aliases": {},
    "mappings": {
      "log": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "host": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "path": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "type": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1497465239325",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "MczS4oYyQi2-lyL4XnT65A",
        "version": {
          "created": "5020299"
        },
        "provided_name": "log-2017.06.14"
      }
    }
  }
}

I'm at a loss as to explain why it works when run manually ( logstash -f C:\path_to_folder), but not as a service. It's calling the same exe....

If you have any insight, please share!

Thanks.

[mick66 (Mick Brennan)]

On your input filter you have:

type => "log"

and on your output filter you have:

if [type] == "sophoslog" {

These do not match therefore your output filter will not process the message and it will not get added to ES.

Change this line to:

if [type] == "log" {