Split JSON array doesn't work with multiple config files

Elastic Stack Logstash
1

Hi All,
I have two config files under the /etc/logstash/conf.d directory.
Each config is listening to a different HTTP port.
When I run the logstash as a daemon(systemctl start logstash) with both config files, I see this Warning: Only String and Array types are splittable. field:items is of type = Hash
However, if i run the logstash with just one config file, I don't see any Warning. It's working well. What is the wrong configuration with multi config files?

This my simple Json Array;

    {
    	"items": [
    		{	
    			"name": "alican",
    			"surname": "uzunhan",
    			"displayName": "sample-Mas1s6"
    		},
    		{
    			"name": "bruce",
    			"surname": "lee",
    			"displayName": "sample-Cas1s3"
    		},
    		{
    			"name": "brandon",
    			"surname": "lee",
    			"displayName": "sample-MAS-astr-2"
    		}
    	]
    }

my content of the /etc/logstash/conf.d/north.conf file

    input {
      http {
        port => 9605
        type => "north"
        codec => "json"
      }
    }

    filter {
      json {
         source => "messages"
      }

      split {
        field => "items"
      }

      mutate {
        remove_field => ["headers", "host", "@version"]
      }

      if [items][displayName] =~ /([M-m]as|MAS)[\d]*/ or [items][displayName] =~ /([E-e]lk|ELK)/ {
        mutate {
          add_field => {"regex" => "Matched"}
        }
      }
      else {
        mutate {
          add_field => {"regex" => "NOT Matched"}
        }
      }
    }

    output {
      if[type] == "north" {
        elasticsearch {
          hosts => ["http://<masterIP>:9200"]
          user => "XXXX"
          password => "XXXXXX"
          index => "nort-logstash-%{+YYYY.MM.dd}"
        }
      }
      stdout { codec => rubydebug }
    }

my content of the /etc/logstash/conf.d/south.conf file

    input {
      http {
        port => 9608
        type => "south"
        codec => "json"
      }
    }

    filter {
      json {
         source => "messages"
      }

      split {
        field => "items"
      }

      mutate {
        remove_field => ["headers", "host", "@version"]
      }

      if [items][displayName] =~ /([M-m]as|MAS)[\d]*/ or [items][displayName] =~ /([E-e]lk|ELK)/ {
        mutate {
          add_field => {"regex" => "Matched"}
        }
      }
      else {
        mutate {
          add_field => {"regex" => "NOT Matched"}
        }
      }
    }

    output {
      if [type] => "south" {
        elasticsearch {
          hosts => ["http://<masterIP>:9200"]
          user => "XXXX"
          password => "XXXXXXX"
          index => "south-logstash-%{+YYYY.MM.dd}"
        }
      }    
      stdout { codec => rubydebug }
    }

content of /etc/logstash/pipelines.yml

    - pipeline.id: north
      path.config: "/etc/logstash/conf.d/north.conf"

    - pipeline.id: south
      path.config: "/etc/logstash/conf.d/south.conf"

content of /etc/logstash/logstash.yml

    path.data: /var/lib/logstash
    pipeline.ordered: auto
    path.config: /etc/logstash/conf.d
    http.host: "127.0.0.1"
    http.port: 9600-9700
    path.logs: /var/log/logstash

Sending Json Array on 9605 HTTP port to point "north.conf";

curl -XPOST http://<logstashIP>:9605 -d '{"items": [{"name": "alican","surname": "uzunhan","displayName": "sample-Mas1s6"},{"name": "bruce","surname": "lee","displayName": "sample-Cas1s3"},{"name": "brandon","surname": "lee","displayName": "sample-MAS-astr-2"}]}'

Warning Messages tail -f /var/log/logstash/logstash-plain.log

    [2021-02-24T19:32:27,082][WARN ][logstash.filters.split   ][main][57ad9cac6065ceff5c766fdf2db03131bcea3316edc24480866722690ae98c7d] Only String and Array types are splittable. field:items is of type = Hash
    [2021-02-24T19:32:27,097][WARN ][logstash.filters.split   ][main][57ad9cac6065ceff5c766fdf2db03131bcea3316edc24480866722690ae98c7d] Only String and Array types are splittable. field:items is of type = Hash
    [2021-02-24T19:32:27,101][WARN ][logstash.filters.split   ][main][57ad9cac6065ceff5c766fdf2db03131bcea3316edc24480866722690ae98c7d] Only String and Array types are splittable. field:items is of type = Hash

However, if i set single conf rather then multi conf, there is no WARN as above. it's working well.
Please show me where is the mistake? Many thanks.

2

You made a small configuration mistake, if you want to use the pipelines.yml file, you cannot set the path.config option in the file logstash.yml.

Setting path.config in your logstash.yml will make logstash ignore the pipelines.yml, and since you have two config files in the directory /etc/logstash/conf.d it will concatenate those files to create one single pipeline.

Now that your two pipelines are just one, logstash is trying to split the items field twice, on the first time it will work because it is an array, but on the second time it will fail and give you the warning you are getting.

Look in your logstash logs and you probably have a line similar to this one:

[2021-02-24T21:01:31,040][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

And another one like this:

[2021-02-24T21:01:37,402][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}

The solution is to remove the path.config line from your logstash.yml, this way logstash will start running using your pipelines.yml and you will have two independent pipelines.

In the logstash log you will have a line like this one:

[2021-02-24T21:08:18,435][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:south, :north], :non_running_pipelines=>[]}
2 Likes
3

I am appreciated Leandro. it's working. I got it. Thank you so much. :pray:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.