Hi Folks,
This is a fairly basic question so i'm sure i'm missing something straightforward.
We are using out of the box everything in a brand new POC environment.
I have created an enrichment policy and associated pipeline that works great when i run data through the ingest node locally in Elasticsearch (via dev console). *pipeline details below
Example of success with enrichment pipeline
POST metricbeat-7.6.2-2020.04.13/_update_by_query?pipeline=pipeline_appcmdb_host_application&wait_for_completion=false
Works great! I can see our enrichment data fields populated based on the matching fields
POST metricbeat-7.6.2-2020.04.21/_doc?pipeline=pipeline_appcmdb_host_application
{doc data here}
This also works just fine.
We have beats 7.6.x running and sending to Logstash. Logstash in turn sends to Elastic via the Elasticsearch Output Plugin (https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-pipeline)
I'm trying to put our logstash sent data through the same pipeline but we are not getting any enrichment. The tag associated with the pipeline is not being added either suggesting it isn't even trying the pipeline.
here is our Logstash pipeline definition which includes the Elasticsearch plugin with the target pipeline defined.
# MonitorDevPipe-1 Logstash configuration
# Beats -> Logstash -> Elasticsearch pipeline.
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["http://gtselkdev01:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
pipeline => "pipeline_appcmdb_host_application"
}
}
Thanks for any help!
Michael
pipeline details
PUT _ingest/pipeline/pipeline_appcmdb_host_application
{
"description" : "adds application name to incoming beats data based on beats agent host name",
"processors" : [
{
"enrich": {
"policy_name": "enrich_appcmdb_host_application",
"field": "host.hostname",
"target_field": "appcmdb_application",
"ignore_missing": true,
"tag": "enrich_appcmdb_host_application"
}
}
]
}