Filebeat send to ES ingest pipeline but can't create index

Elastic Stack Beats
1

I would like to send log to to ingest node pipeline this is my config
before I send the log via filebeat I was created pipeline on ES and see filebeat log send the log to ES.
I have question

  • Why I can't see index name "abc" on ES
  • Is this true on output.elasticsearch should be set to ingest node only ? I have 3 master,data node and 2 ingest node

Environment:
filebeat version 7.3.2 (amd64), libbeat 7.3.2
Elastisearch version 7.3

filebeat.inputs:
- type: log
  enabled: true
  tail_files: true
  paths:
  - /tmp/service.log
  pipeline: "testpipeline"
setup.template.enabled: false
#================================ Outputs =====================================
output.elasticsearch:
  hosts: ["10.235.163.97:9200","10.235.163.98:9200"]
  index: abc

filebeat.registry.path: /var/lib/filebeat/registry/test

Elasticsearch pipline

PUT _ingest/pipeline/testpipeline
{
        "description": "Pipeline for test",
        "processors": [{
                "grok": {
                    "ignore_missing": true,
                    "field": "message",
                    "patterns": ["""TIMESTAMP\|%{TIMESTAMP_ISO8601:TIMESTAMP}\|LOG_TYPE\|%{GREEDYDATA:LOG_TYPE}\|NODE\|%{GREEDYDATA:NODE}\|CACHE_STATUS\|%{GREEDYDATA:CACHE_STATUS}\|REQUEST_ID\|%{GREEDYDATA:REQUEST_ID}\|TRAN_ID\|%{GREEDYDATA:TRAN_ID}\|USER_ID\|%{GREEDYDATA:USER_ID}\|ACTION\|%{GREEDYDATA:ACTION}\|COMMAND\|%{GREEDYDATA:COMMAND}\|REQUEST_PARAM\|%{GREEDYDATA:REQUEST_PARAM}\|RESPONSE_PARAM\|%{GREEDYDATA:RESPONSE_PARAM}\|RESULT\|%{GREEDYDATA:RESULT}\|RESULT_CODE\|%{GREEDYDATA:RESULT_CODE}\|RESULT_DESC\|%{GREEDYDATA:RESULT_DESC}\|RESP_TIME\|%{GREEDYDATA:RESP_TIME}"""
                    ]
                }
            }
        ]
}
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.