How to attach pipeline to index

Elastic Stack Elasticsearch
1

I want to use the pipeline use a pipeline as parameter on an index. Going with info on the ingest node page I was using

curl -XPUT 'http://localhost:9200/demo_old/_type/0?pipeline=old_index&pretty' -H 'Content-Type: application/json' -d'
{}'

Pipeline was created using:

curl -XPUT -k 'http://localhost:9200/_ingest/pipeline/_simulate?pretty' -H 'Content-Type: application/json' -d'
{
"processors" : [
{
"remove" : {
"field" : "geoip"
		}
	}
    
 ] }'

Error I am getting:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "exception",
        "reason" : "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [geoip] not present as part of path [geoip]",
        "header" : {
          "processor_type" : "remove"
        }
      }
    ],
    "type" : "exception",
    "reason" : "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [geoip] not present as part of path [geoip]",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "java.lang.IllegalArgumentException: field [geoip] not present as part of path [geoip]",
      "caused_by" : {
        "type" : "illegal_argument_exception",
        "reason" : "field [geoip] not present as part of path [geoip]"
      }
    },
    "header" : {
      "processor_type" : "remove"
    }
  },
  "status" : 500

Any help with would be great.

2

In your source document you apparently don't have a field named geoip.

3

The log that is harvested by filebeat do not have goip, it is added by Logstash on its way to ES. I am trying to add pipeline on the ES node with the index that have the field added.

I do not know if I put right parameters /_type/0? here

curl -XPUT 'http://localhost:9200/demo_old/_type/0?pipeline=old_index&pretty' -H 'Content-Type: application/json' -d'
{}'
4

So filebeat collects data, then send it to Logstash which enrich it with geoip filter then you send that to elasticsearch through an ingest pipeline. Do I understand correctly?

If so, why are you doing that? If you are using logstash you probably don't need ingest node features as you can do all that in logstash.

FWIW sending an empty document to the ingest pipeline as you did will generate the error you saw.
You need to send something like

curl -XPUT 'http://localhost:9200/demo_old/type/0?pipeline=old_index&pretty' -H 'Content-Type: application/json' -d'{
  "geo": "foo"
}'
5

Yes, this is how we are operating at the moment. We want to get rid of Logstash and send data filebeat to ingest node and I am testing it on the non production env at the moment. is there any more docs that I can read about how to work with pipelines and ingest nodes? If you can provide a link it would be great.

All I wanted to test was if I can attach the pipeline to a specific index so the data that is coming from Logstash for this particular one can be stripped.

Another question I have is (I saw that there is a way of telling filebeat about pipeline):

If not attach pipeline to an index, can I specify pipeline in filebeat for the specific log and it will be applied by the ingest node by request from filebeat?

6

I'd not try to mix things here.

If you want to remove LS then just create a filebeat instance, connect to ES and define the pipeline you want to use.
Then start elasticsearch, create the pipeline and start filebeat.

7

Ok, what about old data I want to restore first? I am using elasticdump to move it from old env to new env and also want to strip geoip from the index. Is reindexing with pipeline my only option? Will the feature backup and restore with pipeline work for me?

8

To do that, I'd run the reindex from remote API on the new cluster to fetch data from the old one. Reindex API has support for ingest pipelines so removing the geo field should be easy.

9

If you need help using reindex API+ingest, there's an example at https://www.elastic.co/guide/en/elasticsearch/reference/5.2/docs-reindex.html . You could also use a simple Painless script at reindex (example also on that page) if you don't want to set up an ingest pipeline/node.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.