# 哪位兄弟知道，怎么在 filebeat 的 json 模板里使用自定义的 analyzer

**URL:** <https://discuss.elastic.co/t/filebeat-json-analyzer/304444>\
**Category:** 中文提问与讨论\
**Created:** [May 11, 2022, 9:26am UTC](https://discuss.elastic.co/t/filebeat-json-analyzer/304444 "2022-05-11T09:26:52Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![fanybook](https://avatars.discourse-cdn.com/v4/letter/f/258eb7/32.png) [@fanybook](https://discuss.elastic.co/u/fanybook)\
**Post date:** [May 11, 2022, 9:26am UTC](https://discuss.elastic.co/t/filebeat-json-analyzer/304444/1 "2022-05-11T09:26:52Z")

</div>

filebeat version: 7.17.3

json 模板是这样的

```json
{
  "index_patterns": ["test_logs*"],
  "template": {
    "settings": {
      "number_of_shards": 1
    },
    "mappings": {
      "properties": {
        "message123": {
          "type": "text",
          "analyzer": "ik_max_word"
        },
        "level123": {
          "type": "keyword"
        }
      }
    },
    "aliases": {}
  }
}

```

但是生成的索引里，message123 字段没有 analyzer，level123字段的类型也不对

```json
{
  "version": 4,
  "mapping_version": 2,
  "settings_version": 1,
  "aliases_version": 1,
  "routing_num_shards": 1024,
  "state": "open",
  "settings": {
    "index": {
      "creation_date": "1652257343314",
      "number_of_shards": "1",
      "number_of_replicas": "1",
      "uuid": "Aut0taBNSFqe0oH0tR_hIQ",
      "version": {
        "created": "7080199"
      },
      "provided_name": "test_logs"
    }
  },
  "mappings": {
    "_doc": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message123": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "log": {
          "properties": {
            "file": {
              "properties": {
                "path": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "ignore_above": 256,
                      "type": "keyword"
                    }
                  }
                }
              }
            },
            "offset": {
              "type": "long"
            }
          }
        },
        "level123": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      }
    }
  },
  "aliases": [],
  "primary_terms": {
    "0": 1
  },
  "in_sync_allocations": {
    "0": [
      "Sesp4gsmSRCSHxG6nJPX5w"
    ]
  },
  "rollover_info": {}
}

```

网上有教程说改 fields.yml，但 GitHub 上，明显还不支持这么做，貌似 8.1 才可以

> <https://github.com/elastic/beats/pull/28926>
>
> \<!-- Type of change
> Please label this PR with one of the following labels, depe…nding on the scope of your change:
> \- Bug
> \- Enhancement
> \- Breaking change
> \- Deprecation
> \- Cleanup
> \- Docs
> \--\>
> 
> \## What does this PR do?
> 
> \<!-- Mandatory
> Explain here the changes you made on the PR. Please explain the WHAT: patterns used, algorithms implemented, design architecture, message processing, etc.
> \--\>
> 
> This adds support for defining custom text analyzers in fields.yml files. For example:
> \`\`\`
> \- key: powershell
> title: PowerShell module
> description: \>
> These are the event fields specific to the module for the Microsoft-Windows-PowerShell/Operational and Windows PowerShell logs.
> release: beta
> analyzer:
> powershell\_script\_analyzer:
> type: pattern
> pattern: "\[\\\\W&&\[^-\]\]+"
> fields:
> ...
> \`\`\`
> 
> \## Why is it important?
> 
> \<!-- Mandatory
> Explain here the WHY, or the rationale/motivation for the changes.
> \--\>
> 
> Not being able to define custom analyzers is a blocker for processing some documents containing syntactically meaningful non-standard token structures (for example captured script text as shown above).
> 
> \## Checklist
> 
> \<!-- Mandatory
> Add a checklist of things that are required to be reviewed in order to have the PR approved
> 
> List here all the items you have verified BEFORE sending this PR. Please DO NOT remove any item, striking through those that do not apply. (Just in case, strikethrough uses two tildes. ~~Scratch this.~~ )
> \--\>
> 
> \- \[x\] My code follows the style guidelines of this project
> \- \[x\] I have commented my code, particularly in hard-to-understand areas
> \- \[\] I have made corresponding changes to the documentation
> ~~- \[\] I have made corresponding change to the default configuration files~~
> \- \[x\] I have added tests that prove my fix is effective or that my feature works
> \- \[x\] I have added an entry in \`CHANGELOG.next.asciidoc\` or \`CHANGELOG-developer.next.asciidoc\`.
> 
> \## Author's Checklist
> 
> \<!-- Recommended
> Add a checklist of things that are required to be reviewed in order to have the PR approved
> \--\>
> No specific recommendations.
> 
> \## How to test this PR locally
> 
> \<!-- Recommended
> Explain here how this PR will be tested by the reviewer: commands, dependencies, steps, etc.
> \--\>
> 
> Running \`go test\` in the relevant packages tests this change.
> 
> \## Related issues
> 
> \<!-- Recommended
> Link related issues below. Insert the issue link or reference after the word "Closes" if merging this should automatically close it.
> 
> \- Closes #123
> \- Relates #123
> \- Requires #123
> \- Superseds #123
> \--\>
> \- Closes #28540
> \- Relates #28483
> \- Relates elastic/integrations#1931
> 
> \## Use cases
> 
> \<!-- Recommended
> Explain here the different behaviors that this PR introduces or modifies in this project, user roles, environment configuration, etc.
> 
> If you are familiar with Gherkin test scenarios, we recommend its usage: https://cucumber.io/docs/gherkin/reference/
> \--\>
> 
> Use case is shown above.
> 
> \## Screenshots
> 
> \<!-- Optional
> Add here screenshots about how the project will be changed after the PR is applied. They could be related to web pages, terminal, etc, or any other image you consider important to be shared with the team.
> \--\>
> 
> N/A
> 
> \## Logs
> 
> \<!-- Recommended
> Paste here output logs discovered while creating this PR, such as stack traces or integration logs, or any other output you consider important to be shared with the team.
> \--\>
> 
> N/A

---

<div class="post-metadata">

**Author:** ![martinliu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinliu/32/40279_2.png) [@martinliu](https://discuss.elastic.co/u/martinliu)\
**Post date:** [May 11, 2022, 11:07am UTC](https://discuss.elastic.co/t/filebeat-json-analyzer/304444/2 "2022-05-11T11:07:23Z")

</div>

尝试下这个做法，在摄入日志前，先做 filebeat setup ； 在 setup 完了以后，在修改 filebeat 的索引 mapping ，加入你需要的 analyzer，

也可以通过修改 fields.yml 的方式在修改 mapping 设置，加上你所需要的。7.17 应该是 ok 的。

---

<div class="post-metadata">

**Author:** ![fanybook](https://avatars.discourse-cdn.com/v4/letter/f/258eb7/32.png) [@fanybook](https://discuss.elastic.co/u/fanybook)\
**Post date:** [May 12, 2022, 2:58am UTC](https://discuss.elastic.co/t/filebeat-json-analyzer/304444/3 "2022-05-12T02:58:53Z")

</div>

感谢帮助 😀

我大概试了一下您说的：

1. 我关闭了 json 模板（不关就不会走 fields.yml），去修改 fields.yml，并没有得到我想要  
fields.yml，注意：一个 text 类型，一个 keyword 类型

```nohighlight
  - name: message123
    level: core
    type: text
    description: 'one message.'
    example: Hello World
  - name: level123
    level: core
    type: keyword
    description: 'one level.'
    example: error

```

我得到的索引类型：两个都是 keyword 类型

```json
{
  "message123": {
    "ignore_above": 1024,
    "type": "keyword"
  },
  "level123": {
    "ignore_above": 1024,
    "type": "keyword"
  }
}

```

1. 手动使用 filebeat setup，这种方式能说的再清晰一些么？我做了这个，但完全没有作用

```bash
./filebeat setup --index-management -E 'output.elasticsearch.hosts=["172.16.100.128:9200"]' -E 'output.elasticsearch.username="elastic"' -E 'output.elasticsearch.password="temppwd"' -E 'setup.template.json.enabled=true' -E 'setup.template.json.path="/opt/filebeat/index/template/golang_logs_index_template.json"' -E 'setup.template.json.name="golang_logs_index_template"'

```

我知道 filebeat setup 是手动加载索引模板的方法，但是我不知道它是怎么和 ./filebeat -e -c test01.yml 中的 test01.yml 关联的

---

<div class="post-metadata">

**Author:** ![fanybook](https://avatars.discourse-cdn.com/v4/letter/f/258eb7/32.png) [@fanybook](https://discuss.elastic.co/u/fanybook)\
**Post date:** [May 12, 2022, 6:00am UTC](https://discuss.elastic.co/t/filebeat-json-analyzer/304444/4 "2022-05-12T06:00:54Z")

</div>

我知道原因了，是 json 模板里多了一级 template

---

<div class="post-metadata">

**Author:** ![martinliu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinliu/32/40279_2.png) [@martinliu](https://discuss.elastic.co/u/martinliu)\
**Post date:** [May 12, 2022, 7:31am UTC](https://discuss.elastic.co/t/filebeat-json-analyzer/304444/5 "2022-05-12T07:31:35Z")

</div>

不错，定位到原因就好。

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 9, 2022, 7:31am UTC](https://discuss.elastic.co/t/filebeat-json-analyzer/304444/6 "2022-06-09T07:31:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.