Two types under one filter

[rachelyang]

Can I have two types (plain txt log & json )under one filter in the logstash.yml file?

input {
    beats {
      port => "5044"
    }
}

filter {
  if [fields=='access'] {
    grok {
	  remove_field => "message"
    }
  }
  if [fields=='cloud'] {
  json {
        remove_field => [ "ClientIP","ResponseBytes" ]
      }
  }
}

output {
    if [fields=='access'] {
      elasticsearch {
	    hosts => ["localhost:9200"]
        index => "access-%{+YYYY.MM.dd}"
        }
	}
    else if [fields=='cloud'] {
      elasticsearch {
	    hosts => ["localhost:9200"]
        index => "cloud-%{+YYYY.MM.dd}"
        }
	}
}

[Rios]

I assume that you have a log on FB side which sends data to LS. By default, all data which has been arrived to LS will be in the message field, as text. Inside LS, you will do conversion to JSON fields, which you are set in the code.
I assume that you want to write different log in different index.You can use "fields.name" in FB which you will have in LS. For instance, fields.log: access and fields.log: cloud in filebeat.yml

Your code would be:

input {
    beats {
      port => "5044"
    }
}

filter {
  if [fields][log]=="access" {
    mutate {
	  remove_field => "message"
    }
  }
  else if [fields][log]=="cloud" {
  json {  }
    mutate {
	  remove_field => [ "ClientIP","ResponseBytes", "fields" ] # remove "fields" before Elasticsearch
    }
  }
}

output {
  if [fields][log]=="access" {
      elasticsearch {
	    hosts => ["localhost:9200"]
        index => "access-%{+YYYY.MM.dd}"
        }
	}
  else if [fields][log]=='cloud' {
      elasticsearch {
	    hosts => ["localhost:9200"]
        index => "cloud-%{+YYYY.MM.dd}"
        }
	}
}

Another option to check how logs start, or containing a keyword, with regex you can check does a line starts with date in format yyyy-MM-dd:
if ([message] =~ /^(\d{4,})-(\d{2})-(\d{2})$/ ) { ...
This is useful for other streams like syslog.
Do whatever is easier for you.

[rachelyang]

Thank you, Rios. I have two types on the input side of Filebeat, text & Json. I will push both of them to Logstash port 5044. Can I do that?
Here is my input part of the filebeat.yml:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/access/access.log
  fields: 
    access: true
  fields_under_root: true
  
- type: filestream
  enabled: true
  paths:
    - /var/log/cloud/cloud.log
  fields: 
    cloud: true
  fields_under_root: true

[Rios]

@Rios

if [fields][log]

For your filebeat.yml will be

`if [fields][access]` ==true {... # or maybe will be under quotes =="true"
`if [fields][cloud]` ==true {...

Add in output debugger and will see values:

output {
 stdout { codec => rubydebug }
...
  }