Logstash 6.8.23 Cannot put data into elastic index, mapping issue

Elastic Stack Logstash
1

HI, I am trying to pass data from logstash to ES index, but it throws error as shown below

2022-07-28T10:28:17,909][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"100", :_index=>"stm-staff-profile-index-v120", :_routing=>nil, :_type=>"doc"}, #<LogStash::Event:0x75fd1305>], :response=>{"index"=>{"_index"=>"stm-staff-profile-index-v120", "_type"=>"doc", "_id"=>"100", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Mapper for [can_display_user_weight] conflicts with existing mapper:\n\tCannot update parameter [index] from [false] to [true]"}}}

Input plugin is as shown below

input {
  jdbc {
    tags => "staff"
    jdbc_connection_string => "jdbc:postgresql://${DB_HOST}:${DB_PORT}/${DB_NAME}"
    jdbc_user => "${DB_USER}"
    jdbc_password => "${DB_PASSWORD}"
    schedule => "* * * * *"
    jdbc_driver_class => "org.postgresql.Driver"
    statement_filepath => "/staffboard-logstash/queries/stm-staff-query.sql"
    use_column_value => true
    tracking_column => "last_updated"
    tracking_column_type => "timestamp"
    last_run_metadata_path => "/usr/share/logstash/logstash_jdbc_last_run_stm_staff_profile"
    jdbc_paging_enabled => true
    jdbc_page_size => 10000
    jdbc_fetch_size => 10000
  }
}

Output plugin is as shown below

  if "staff" in [tags] {
    stdout {
      codec => json
    }
    elasticsearch {
      hosts => ["${ELASTIC_HOST}"]
      ssl => true
      index => "stm-staff-profile-index-v125"
      document_id => "%{sb_id}"
      user => "${ELASTIC_USER_NAME}"
      password => "${ELASTIC_PASSWORD}"
      ilm_enabled => false
      manage_template => true
      template => "/staffboard-logstash/mappings/stm/stm-staff-profile-index.json"
      template_name => "stm-staff-profile-index-mapping-125"
      template_overwrite => true
    }
  }

Mapping file is as shown below

{
  "template": "stm-staff-profile-index-v125",
  "version": 1,
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "@version": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "account_type": {
        "type": "keyword"
      },
      "user_name_kana": {
        "type": "keyword"
      },
      "shop_code": {
        "type": "keyword"
      },
      "shop_number": {
        "type": "keyword"
      },
      "store_account_id": {
        "type": "keyword"
      },
      "shop_name": {
        "type": "keyword"
      },
      "user_code": {
        "type": "keyword"
      },
      "sb_id": {
        "type": "integer",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      },
      "user_name": {
        "type": "keyword"
      },
      "user_nickname": {
        "type": "keyword"
      },
      "crowned_staff": {
        "type": "boolean"
      },
      "store_visit": {
        "type": "boolean"
      },
      "status": {
        "type": "keyword"
      },
      "branch_cd": {
        "type": "keyword"
      },
      "role_name": {
        "type": "keyword"
      },
      "label_code": {
        "type": "keyword"
      },
      "group_id": {
        "type": "keyword"
      },
      "sns_approval_status.approval_status": {
        "type": "keyword"
      },
      "sns_approval_status.sns_type": {
        "type": "keyword"
      },
      "sns_approval_status.sns_handle": {
        "type": "text",
        "index": false
      },
      "age_range": {
        "type": "text",
        "index": false
      },
      "bio": {
        "type": "text",
        "index": false
      },
      "body_characteristics": {
        "type": "text",
        "index": false
      },
      "branch_name": {
        "type": "text",
        "index": false
      },
      "can_display_age": {
        "type": "boolean",
        "index": false
      },
      "can_display_bio": {
        "type": "boolean",
        "index": false
      },
      "can_display_body_characteristics": {
        "type": "boolean",
        "index": false
      },
      "can_display_body_shape": {
        "type": "boolean",
        "index": false
      },
      "can_display_date_of_birth": {
        "type": "boolean",
        "index": false
      },
      "can_display_fav_styling": {
        "type": "boolean",
        "index": false
      },
      "can_display_front_label": {
        "type": "boolean",
        "index": false
      },
      "can_display_gender": {
        "type": "boolean",
        "index": false
      },
      "can_display_have_child": {
        "type": "boolean",
        "index": false
      },
      "can_display_interest": {
        "type": "boolean",
        "index": false
      },
      "can_display_personal_color": {
        "type": "boolean",
        "index": false
      },
      "can_display_profile_on_st": {
        "type": "boolean",
        "index": false
      },
      "can_display_shoe_size": {
        "type": "boolean",
        "index": false
      },
      "can_display_shoulder_stature": {
        "type": "boolean",
        "index": false
      },
      "can_display_size": {
        "type": "boolean",
        "index": false
      },
      "can_display_store": {
        "type": "boolean",
        "index": false
      },
      "can_display_user_bottom_size": {
        "type": "boolean",
        "index": false
      },
      "can_display_user_height": {
        "type": "boolean",
        "index": false
      },
      "can_display_user_name": {
        "type": "boolean",
        "index": false
      },
      "can_display_user_top_size": {
        "type": "boolean",
        "index": false
      },
      "can_display_user_weight": {
        "type": "boolean",
        "index": false
      },
      "coubic_url": {
        "type": "text",
        "index": false
      },
      "created_at": {
        "type": "date",
        "index": false
      },
      "csvideo_count": {
        "type": "integer",
        "index": false
      },
      "cumulation_count": {
        "type": "integer",
        "index": false
      },
      "dob": {
        "type": "date",
        "index": false
      },
      "fav_styling_1": {
        "type": "text",
        "index": false
      },
      "fav_styling_2": {
        "type": "text",
        "index": false
      },
      "fav_styling_3": {
        "type": "text",
        "index": false
      },
      "follower_count": {
        "type": "integer",
        "index": false
      },
      "front_display_label_code": {
        "type": "text",
        "index": false
      },
      "front_display_label_name": {
        "type": "text",
        "index": false
      },
      "have_child": {
        "type": "text",
        "index": false
      },
      "interests": {
        "type": "text",
        "index": false
      },
      "is_post_approval_required": {
        "type": "boolean",
        "index": false
      },
      "label_name": {
        "type": "text",
        "index": false
      },
      "label_name_kana": {
        "type": "text",
        "index": false
      },
      "last_updated": {
        "type": "date",
        "index": false
      },
      "last_updated_stm": {
        "type": "date",
        "index": false
      },
      "personal_color": {
        "type": "text",
        "index": false
      },
      "sb_permission_id": {
        "type": "integer",
        "index": false
      },
      "sb_role_id": {
        "type": "integer",
        "index": false
      },
      "shop_name_kana": {
        "type": "text",
        "index": false
      },
      "shoulder_stature": {
        "type": "text",
        "index": false
      },
      "store_prefecture_code": {
        "type": "text",
        "index": false
      },
      "styling_count": {
        "type": "integer",
        "index": false
      },
      "tags": {
        "type": "text",
        "index": false
      },
      "user_body_shape": {
        "type": "text",
        "index": false
      },
      "user_bottom_size": {
        "type": "text",
        "index": false
      },
      "user_gender": {
        "type": "text",
        "index": false
      },
      "user_height": {
        "type": "text",
        "index": false
      },
      "user_image_url": {
        "type": "text",
        "index": false
      },
      "user_shoe_size": {
        "type": "text",
        "index": false
      },
      "user_top_size": {
        "type": "text",
        "index": false
      },
      "user_weight": {
        "type": "text",
        "index": false
      },
      "weight_range": {
        "type": "text",
        "index": false
      }
    }
  }
}

I understand that the mapping that I provided is not taken into consideration. I think it is trying to apply a default mapping different from the one I have given while passing data from logstash to ES.

Any help is appreciated. Thank you in advance

2

What is the content of the template file?

This one : /staffboard-logstash/mappings/stm/stm-staff-profile-index.json ?

Does the index_pattern match your index name? Also, did you indexed anything else in this index before creating the template?

3

The json file show in the question is the content of /staffboard-logstash/mappings/stm/stm-staff-profile-index.json

Yes, I have given the same name as the index in the index pattern(shown in json data provided in question)
I tried running logstash after deleting all the existing template and index and it still throws the error

4

The json file you shared does not have any index_pattern setting, are you sure that this is the same file?

5

I have tried with both template as well as index_pattern in place of it but the end result is the same issue @leandrojmp

I had this working in 3 other environments, but in a new instance created from a clone of another instance this issue is happening.

In the instances that the code is working, we had reindex from an index having default mapping to this custom mapping and sync works well there. However it does not work in the new clone instance.

Let me share with you the docker file, if that helps

FROM openjdk:8u265-jdk-slim-buster

WORKDIR /folder
RUN apt-get update; apt-get install -y curl \
  && curl -sL https://deb.nodesource.com/setup_14.x | bash - \
  && apt-get install -y nodejs \
  && curl -L https://www.npmjs.com/install.sh | sh


RUN apt-get install -y wget; wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add - \
    && apt-get install -y apt-transport-https \
    && echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list \
    && apt-get update \
    && apt-get install -y logstash=1:6.8.23-1 \
    && /usr/share/logstash/bin/logstash-plugin install --version '10.8.6' logstash-output-elasticsearch 

RUN apt-get -y install cron

RUN echo "config.support_escapes: true" >> /etc/logstash/logstash.yml
RUN echo "pipeline.batch.size: 1000" >> /etc/logstash/logstash.yml
RUN echo "pipeline.batch.delay: 50" >> /etc/logstash/logstash.yml
RUN echo "queue.type: persisted" >> /etc/logstash/logstash.yml
RUN echo "queue.max_bytes: 2000mb" >> /etc/logstash/logstash.yml
RUN echo "queue.drain: true" >> /etc/logstash/logstash.yml

RUN apt-get install -y jq python-pip libpython-dev

RUN apt-get install -y jq
RUN pip install --upgrade awscli

ADD . /folder

RUN mkdir  /folder/es
ADD folder.conf /folder/es

RUN mkdir /root/.aws
ADD config /root/.aws
ADD credentials /root/.aws

RUN  cp postgresql-42.2.18.jar /usr/share/logstash/logstash-core/lib/jars \
  && chown logstash:logstash /usr/share/logstash/logstash-core/lib/jars/postgresql-42.2.18.jar


RUN ["chmod", "+x", "/folder/entrypoint.sh"]

CMD /folder/entrypoint.sh
#ENTRYPOINT /batch/entrypoint.sh
6

You need to use index_pattern, that's why I asked for you to share the exactly json you are using.

Also, since you are using 6.8.X, your template needs a mapping type in this version, which your json file do not have, I'm not sure it works without a mapping type, it's been years since I've used 6.X.

You would need something like this:

{
    "index_patterns": ["index-name"],
    "version": 1,
    "mappings": {
        "_doc": {
            "properties": {
                "your mappings"
            }
        }
    }
  }

Another thing is, your index name in the output is stm-staff-profile-index-v125, the index name in your error log is stm-staff-profile-index-v120, so your index_pattern should be stm-staff-profile-index-*.

One important thing is that you mentioned that this you have this working in 3 environments, but that in the instances that the code is working you did a reindex from a previous index.

Did you reindex first and then started the Logstash pipeline, or you started the logstash pipeline and after that you did the reindex?

The order is important because the template is only applied when the index is created, so if you reindex from an index with the correct mapping and after that you started the sync with Logstash, the template you configured is not used.

7

We are using open search 1.2.0 instead of Elastic search. Could this cause issue with Logstash version 6.8.23 ?

Also we are using different version of Elasticsearch output plugin, as specified in this line in docker

   && /usr/share/logstash/bin/logstash-plugin install --version '10.8.6' logstash-output-elasticsearch
8

From the documentation, the version for this plugin in Logstash 6.8 is 9.4.X, from version 7.X the version of the plugin 10.X, but should not be an issue since Logstash 6.8.X is compatible with elasticsearch until version 7.17.

Opensearch is not supported here since Amazon has made some changes in the code and it is not possible to know if your issue is related to some of these changes or not, you will need to check with the Opensearch forum for further help.

But, what about this question?

Since you didn't answered it, it seems that your template never worked and the index where it seems to have worked got the mappings from the reindex, not the template.

I would suggest that you try to create an index manually with your mapping and see if it is working or not.

9

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.