Hi,
I am stuck on configuring filebeat to use a custom elasticsearch ingest pipeline.
For some reasons the pipeline is now applied for logs shipped by filebeat.
If a document is indexed manually, everything works as expected. Please see the details below.
Pipeline
PUT _ingest/pipeline/test
{
"description": "Test",
"processors": [
{
"set": {
"field": "foo",
"value": "bar"
}
}
]
}
Create Index
PUT test
Testdocument
POST test/_doc?pipeline=test
{
"@timestamp" : "2020-07-07T14:21:52.000+02:00"
}
Filebeat config
output:
elasticsearch:
hosts:
- https://elastic01-t-bfl.intra.realstuff.ch
index: test
password: xxxxxxxxxxx
ssl.certificate_authorities:
- /etc/pki/tls/certs/rs-ansible-ca.crt
username: beats_writer
pipelines:
- pipeline: test
Index Query first document
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "E-xcKXMBiIdD4nFHVd6o",
"_score" : 0.13353139,
"_source" : {
"@timestamp" : "2020-07-07T14:56:52.000+02:00",
"foo" : "bar"
}
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "e-xcKXMBiIdD4nFHfN9m",
"_score" : 1.0,
"_source" : {
"agent" : {
"hostname" : "elastic01-t-bfl.intra.realstuff.ch",
"name" : "elastic01-t-bfl.intra.realstuff.ch",
"id" : "5f20d6ab-fc29-4b6d-90e7-5777c5381fa5",
"type" : "filebeat",
"ephemeral_id" : "73f7b3d0-51aa-47bd-9e9c-1d04d7cc9d07",
"version" : "7.8.0"
},
"process" : {
"name" : "journalbeat"
},
"log" : {
"file" : {
"path" : "/var/log/messages"
},
"offset" : 244192259
},
"fileset" : {
"name" : "syslog"
},
"message" : """2020-07-07T14:59:50.921+0200#011INFO#011[input]#011input/input.go:141#011journalbeat successfully published 1 events#011{"id": "cca32660-ba22-4298-87ac-2548992b4a3d"}""",
"input" : {
"type" : "log"
},
"@timestamp" : "2020-07-07T14:59:50.000+02:00",
"system" : {
"syslog" : { }
},
"ecs" : {
"version" : "1.5.0"
},
"service" : {
"type" : "system"
},
"host" : {
"hostname" : "elastic01-t-bfl",
"name" : "elastic01-t-bfl.intra.realstuff.ch"
},
"event" : {
"timezone" : "+02:00",
"module" : "system",
"type" : "event",
"dataset" : "system.syslog"
}
}
}
]
Does anybody know the problem?