Something is wrong with pipeline

Elastic Stack Beats
1

I have Stack without Logstash. But I need to correct some fields in kibana (timestamp for example). I try to use pipeline for this purpose. This is my config file:
filebeat.yml


filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log

multiline.pattern: '^\['
multiline.negate: true
multiline.match: after


output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: "rename_hostname"

setup.template.overwrite: true
setup.dashboards.overwrite: true
output.elasticsearch.index: "docker-one"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.dashboards.index: "docker-*"

setup.kibana:
  host: "localhost:5601"

And I execute this command in devtools in Kibana:

 PUT _ingest/pipeline/rename_hostname
{
  "processors": [
    {
      "rename": {
        "field": "hostname",
        "target_field": "host",
        "ignore_missing": true
      }
    }
  ]
}

... and nothing changes in kibana interface. What I did wrong?

2

Are there any errors in your Filebeat logs? Such as a mapping error for host? You might need to customize your index template if you intent to make host a keyword because in the latest version host is mapped as an object. It does have a host.name field though.

3

Filebeat working fine and I dont get any errors. It does not matter for me what type of processors I have used. For example if my pipeline like:

PUT _ingest/pipeline/examplepipe
{
  "description" : "Change smth",
  "processors": [
    ]
}

then how I can understand: are me log files go through this pipeline? Because other processors seems do not work.

4

Elasticsearch provides stats about each pipeline that you can use to check if any events are passing through.

GET _nodes/stats/ingest

Your config appears correct to me. But how about trying to put pipeline as part of the filebeat input config instead. reference

filebeat.inputs:
- type: log
  paths:
    - /var/log/*.log
  pipeline: my_pipeline
5

My problems seems to be solved. But I can't describe what exactly I was doing wrong. Thanks for some helpfull advices.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.