Configuring the new logstash version - issues on the output Elasticsearch template for the mapping of my logs

Elastic Stack Logstash
1

Hi Guys

I'm trying to configure the new version logstash (7.x)

Everything is working as prefect, logstash is running and I can see my logs on kibana :slight_smile: . BUT the mapping/parsing elasticsearch isn't working :frowning:

Following, the config of the new logstash :

My config file : logstash.conf
(This works fine but the template part isn't well configured I guess).

input {
	kafka {
		topics => ["nifi-xxx-fab","nifi-yyy-fab"]
		client_id => "logstash-FAB"
		group_id => "Logs-FAB"
		bootstrap_servers => '10.200.175.xxx:15011,10.200.175.xxx:15012,10.200.xxx.152:15013'
		consumer_threads => 3
	}
}

filter {
if [type] == "weblogic-xxx" {
	date {
		match => [ "date", "MMM d, yyyy, H:mm:ss,SSS a" ]
		}
	}
}

output {
    elasticsearch {
        hosts => ["elastixxxx:xxxx", "elastixxxx:xxxx", "elastixxxx:xxxx"]
	index => "logstash-%{+YYYY.MM.dd}"
	**template => "/usr/share/logstash/pipeline/elasticsearch-template.json"**
	template_overwrite => "true"
}
}

my elasticsearch-template.json looks like this :

{
  "index_patterns" : "logstash-*",
  "settings" : {
    "index" : {
	  "refresh_interval" : "5s"
    }
	},
	"mappings" : {
    "dynamic_templates" : [
    {
	  "message_field" : {
	    "match" : "message",
	    "match_mapping_type" : "string",
	    "mapping" : {
	      "type" : "string"
	    }
      }
    },
    {
      "string_fields" : {
        "match_mapping_type" : "string",
        "mapping" : {
	      "type" : "string",
	      "fields" : {
            "raw" : {
              "type": "string",
              "ignore_above" : 256
            }
		  }
		}
      }
    }
    ]
  }
}

My config of the docker logstash .yml

logstash:
    image: docker-virtual.artefact-repo.pole-emploi.intra/logstash/logstash:7.0.1
    ports:
      - "xxxx:xxxx"
      - "xxxx:xxxx"
    volumes:
      - /nas/donapp/docke/_fsb/swmxor10/oxxx/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
      - /nas/donapp/docke/_fsb/swmxor10/oixxx/logstash/pipeline/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
      - /nas/donapp/docke/_fsb/swmxor10/oi071/logstash/pipeline/elasticsearch-template.json:/usr/share/logstash/pipeline/elasticsearch-template.json

So when I deploy logstash. I got the following erreur :

message=>"Expected one of #, input, filter, output at line 1,

and without the template thing, it works so I am pretty sure that the error come from the configuration of my elasticsearch-template

my old config of the template is as follow ( and this was working so fine)


{
  "template" : "logstash-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
       "_all" : {"enabled" : true, "omit_norms" : true},
       "dynamic_templates" : [ {
         "message_field" : {
           "match" : "message",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true
           }
         }
       }, {
         "string_fields" : {
           "match" : "*",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true,
               "fields" : {
                 "raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
               }
           }
         }
       } ],
       "properties" : {
         "@version": { "type": "string", "index": "not_analyzed" },
         "geoip"  : {
           "type" : "object",
             "dynamic": true,
             "properties" : {
               "location" : { "type" : "geo_point" }
             }
         }
       }
    }
  }
}

Thank you guys for your help.

Regards

2

Hi @soufian.eldouqe
Could you try this ?

 output {
elasticsearch {
    hosts => ["elastixxxx:xxxx", "elastixxxx:xxxx", "elastixxxx:xxxx"]
	index => "logstash-%{+YYYY.MM.dd}"
	manage_template => true
	template => "/usr/share/logstash/pipeline/elasticsearch-template.json"
	template_name => "elasticsearch-template"
	template_overwrite => true
			}
}
3

Hello,

Thank you for your answer but no it doesn't work and I still got the same error

message=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1)",

regards,

4

Hello

Could you change, in your template file ?

"template" : "logstash-*", with "index_patterns": ["logstash-*"],
5

Hello,

That's what I already have. I think you were watching the old config.

6

The error show that you have a wildcard in the beginning of your logstash config file
Byte 1, line 1 means that your config start with hidden special char ...
Could you make sure you config file in UTF-8 and that's clean (use special file editor to show all special caracters like notepad++)

7

Thank you for your help.

I have checked my file in notepad. and I'm sure that I don't have any special char on it. My file is on ASCII format as is my old one .

8

Just double check with Notepad++

9

What is path.config set to? What command line are you using to start logstash?

10

docker stack deploy oixxx -c /applis/xdocke/pur/par/oixxx.yml

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.