emi_rose
(user858708178411)
September 29, 2023, 3:21pm
1
Hi there,
When I test my ingest pipeline, which replaces a delimiter with a whitespace, with a document that has the exact field I'm trying to transform, I keep getting this error: [Field [field] not present as part of path [field.query] for Elasticsearch ingest pipeline]. I'm not understanding why the field would not be present. Using ignore_missing or ignore_failure just skips over the field completely as if it's not there.
Any help would be appreciated.
Thanks
dadoonet
(David Pilato)
September 29, 2023, 4:27pm
2
It would be easier if you share:
Your pipeline
A sample document
Event better, if you could share a simple _simulate
call , like:
POST /_ingest/pipeline/_simulate
{
"pipeline" :
{
"description": "_description",
"processors": [
{
"set" : {
"field" : "field2",
"value" : "_value"
}
}
]
},
"docs": [
{
"_index": "index",
"_id": "id",
"_source": {
"foo": "bar"
}
}
]
}
emi_rose
(user858708178411)
September 29, 2023, 5:23pm
3
POST /_ingest/pipeline/_simulate
{
"pipeline" :
{
"processors": [
{
"gsub": {
"field": "foo.bar.foobar",
"pattern": "\\|x\\|",
"replacement": " "
}
}
]
},
"docs": [
{
"_index": "my_index",
"_id": "1",
"_version": 1,
"_score": 0,
"_ignored": [
"message.keyword"
],
"_source": {
"foo.bar.foobar": "foo|x|bar"
}
}
]
}
stephenb
(Stephen Brown)
September 29, 2023, 6:02pm
4
Pipelines do not support "flattened" objects today
POST /_ingest/pipeline/_simulate
{
"pipeline": {
"processors": [
{
"gsub": {
"field": "foo.bar.foobar",
"pattern": """\|x\|""",
"replacement": " "
}
}
]
},
"docs": [
{
"_source": {
"foo.bar.foobar": "foo|x|bar"
}
},
{
"_source": {
"foo": {
"bar": {
"foobar": "foo|x|bar"
}
}
}
}
]
}
#results
{
"docs": [
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "field [foo] not present as part of path [foo.bar.foobar]"
}
],
"type": "illegal_argument_exception",
"reason": "field [foo] not present as part of path [foo.bar.foobar]"
}
},
{
"doc": {
"_index": "_index",
"_id": "_id",
"_version": "-3",
"_source": {
"foo": {
"bar": {
"foobar": "foo bar"
}
}
},
"_ingest": {
"timestamp": "2023-09-29T17:55:13.057682074Z"
}
}
}
]
}
if you actually have that syntax then you will need to dot_expander
POST /_ingest/pipeline/_simulate
{
"pipeline": {
"processors": [
{
"dot_expander": {
"field": "foo.bar.foobar"
}
},
{
"gsub": {
"field": "foo.bar.foobar",
"pattern": """\|x\|""",
"replacement": " "
}
}
]
},
"docs": [
{
"_source": {
"foo.bar.foobar": "foo|x|bar"
}
},
{
"_source": {
"foo": {
"bar": {
"foobar": "foo|x|bar"
}
}
}
}
]
}
# results
{
"docs": [
{
"doc": {
"_index": "_index",
"_id": "_id",
"_version": "-3",
"_source": {
"foo": {
"bar": {
"foobar": "foo bar"
}
}
},
"_ingest": {
"timestamp": "2023-09-29T18:00:41.097144482Z"
}
}
},
{
"doc": {
"_index": "_index",
"_id": "_id",
"_version": "-3",
"_source": {
"foo": {
"bar": {
"foobar": "foo bar"
}
}
},
"_ingest": {
"timestamp": "2023-09-29T18:00:41.097175694Z"
}
}
}
]
}
1 Like
emi_rose
(user858708178411)
September 29, 2023, 6:09pm
5
I was wondering if that might be the case. Unfortunately, the actually document has many foo.bar* fields so I'm not sure if I'd need to expand all of them or not. Or perhaps would this automatically expand all of them?
emi_rose
(user858708178411)
September 29, 2023, 6:14pm
6
Nevermind, I tried it out and it only expands the one field. This makes the pipeline execute and the gsub processor work. Thank you very much!
1 Like
emi_rose
(user858708178411)
September 29, 2023, 8:52pm
7
@stephanb For some reason, the simulation works, but the pipeline does not work upon ingestion. I've tried many different combinations of ignore_failure, ignore_missing, and override with the two processors but the foo.bar.foobar field looks untouched in the documents.
stephenb
(Stephen Brown)
September 29, 2023, 10:08pm
8
@emi_rose
You need to show us an end to end repeatable example
Sample Documents
Sample Pipeline
Sample Mapping (schema) are you creating one?
Execute to ingest / write the document and the failed result.
Show all the commands and all the result
otherwise we can not help.
The writing most likely fails because the document being written does not match the expected mapping (think schema)
system
(system)
Closed
October 27, 2023, 10:09pm
9
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.