# Cannot seem to get ingest pipelines to work, please help!
**URL:** https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259
**Category:** Elasticsearch
**Created:** [February 14, 2021, 11:09pm UTC](https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259 "2021-02-14T23:09:47Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@mhare](https://discuss.elastic.co/u/mhare)
#### Post date: [February 14, 2021, 11:09pm UTC](https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259/1 "2021-02-14T23:09:47Z")
I am trying to understand ES ingest pipelines and Filebeat.
I am using version 7.11 ES and K and FB in docker containers.
If I do not add the pipeline, I get the log lines in ES/K from FB, but they are not formatted very well, (It's all indexed into a 'message' keyword. So I want to create a pipeline to further breakdown the values in the log lines into keywords using 'tab' separators.
I have created a pipeline in Kibana Dev Tools as:
```auto
PUT _ingest/pipeline/rcp_log_pipeline_tab
{
"description" : "rcp log tab pattern",
"processors" : [
{
"csv" : {
"field" : "message",
"target_fields" : [
"timestamp",
"relativeTime",
"thread",
"processName",
"sourceName",
"logType",
"logMessage"
],
"separator" : " "
}
},
{
"rename" : {
"field" : "timestamp",
"target_field" : "@timestamp"
}
},
{
"rename" : {
"field" : "@timestamp",
"target_field" : "index_timestamp"
}
}
]
}
```
and I have simulated it with
```auto
POST _ingest/pipeline/rcp_log_pipeline_tab/_simulate
{
"docs": [
{
"_source": {
"message": "2021-01-02T00:01:00.134-08:00 1047176101054 0x0017 US-W10L2.Axxion.ToolSpud. IoProvider Background Performing BankReadIOPoints"
}
}
]
}
```
And I get the results I am expecting. Perfect!
So I updated my filebeat.yml to use this pipeline
```auto
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST_PORT}']
username: '${ELASTIC_USERNAME}'
password: '${ELASTIC_PASSWORD}'
pipeline: 'rcp_log_pipeline_tab'
```
deleted the filebeat-\* index in Kibana, removed the registry in Filebeat so it will resend the log files, restarted the Filebeat container and I get nothing from Filebeat. The index is created, but it is empty. ☹
What I find scouring the web seems to indicate this is what I should be doing.. so why is it not working?
Thanks for taking a look!
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [February 15, 2021, 12:53am UTC](https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259/2 "2021-02-15T00:53:23Z")
What do your Filebeat logs show?
---
### Author:  [@mhare](https://discuss.elastic.co/u/mhare)
#### Post date: [February 15, 2021, 1:43am UTC](https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259/3 "2021-02-15T01:43:05Z")
oooo... where are they on a Mac? Not /var/log as far as I can tell.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [February 15, 2021, 1:43am UTC](https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259/4 "2021-02-15T01:43:52Z")
Installed via Homebrew?
---
### Author:  [@mhare](https://discuss.elastic.co/u/mhare)
#### Post date: [February 15, 2021, 1:44am UTC](https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259/5 "2021-02-15T01:44:49Z")
I'm running in Docker containers
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [February 15, 2021, 1:45am UTC](https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259/6 "2021-02-15T01:45:36Z")
The `docker log` would be what you are after.
---
### Author:  [@mhare](https://discuss.elastic.co/u/mhare)
#### Post date: [February 15, 2021, 2:02am UTC](https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259/7 "2021-02-15T02:02:49Z")
seems there is a problem with screen and accessing the tty on the docker vm and so access to the docker logs are not available (I am still searching for alternative ways to get to the logs) I do appreciate your patience and help
---
### Author:  [@mhare](https://discuss.elastic.co/u/mhare)
#### Post date: [February 15, 2021, 2:25am UTC](https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259/8 "2021-02-15T02:25:06Z")
ok, so 'docker logs ' is working
---
### Author:  [@mhare](https://discuss.elastic.co/u/mhare)
#### Post date: [February 15, 2021, 2:48am UTC](https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259/9 "2021-02-15T02:48:13Z")
I was so bent on finding the Filebeat logs I completely forgot to actually look at the docker logs (oh, well)
That was a big help. It sems to be not parsing the same from Filebeat it did with Dev Tools, but at least I am getting something I can work with now. Thank you so much for knocking me in the direction of Docker logs=
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [February 15, 2021, 3:01am UTC](https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259/10 "2021-02-15T03:01:37Z")
Hi @mhare
Perhaps Add some failure handling to the ingest pipeline. See [here](https://www.elastic.co/guide/en/elasticsearch/reference/7.11/handling-failure-in-pipelines.html#accessing-error-metadata)
It will help you debug.. ahh I see you just found the logs... I was typing this up...
2nd When you just simulate pipelines, types are not checked as well, its a good first check but it does not actually try to insert a doc so if there is a type mismatch it will not throw an error, if it is throwing and error when the docs is inserted you will not see it AND then doc will not be inserted.
3rd I am not quite sure what you are trying to accomplish
You rename `timestamp` to `@timestamp` then rename `@timestamp` to `index_timestamp` this mean `@timestamp` will not be actually set in the doc by this pipeline, these all happen **before** the doc is written.
4th doing a rename on the timestamp field is actually doing some smarts under the covers which you can not always count on you really should use the [date processor,](https://www.elastic.co/guide/en/elasticsearch/reference/7.11/date-processor.html) turns out your date has a good format so the rename will work (I think)
5th be super careful with that tab in the processor make sure you smart tabs is not turning into a space
So I suggest (besides finding the logs... which will probably say the doc is not being inserted, I would add error handline something like this
```
PUT _ingest/pipeline/rcp_log_pipeline_tab
{
"description": "rcp log tab pattern",
"processors": [
{
"csv": {
"field": "message",
"target_fields": [
"timestamp",
"relativeTime",
"thread",
"processName",
"sourceName",
"logType",
"logMessage"
],
"separator": " ",
"on_failure": [
{
"set": {
"field": "error.message_csv",
"value": "error in csv processor"
}
}
]
}
},
{
"date": {
"field": "timestamp",
"target_field": "@timestamp",
"formats": ["date_optional_time||strict_date_optional_time"],
"on_failure": [
{
"set": {
"field": "error.message_date",
"value": "error in date processor"
}
}
]
}
}
]
}
# Create a mapping with an actual @timestamp field
DELETE my-test-data
PUT my-test-data
{
"mappings": {
"properties": {
"@timestamp" : {
"type": "date"
}
}
}
}
# Now actual test writing a doc
POST my-test-data/_doc?pipeline=rcp_log_pipeline_tab
{
"message": "2021-01-02T00:01:00.134-08:00 1047176101054 0x0017 US-W10L2.Axxion.ToolSpud. IoProvider Background Performing BankReadIOPoints"
}
# See what it looks like
GET my-test-data/_search
```
---
### Author:  [@mhare](https://discuss.elastic.co/u/mhare)
#### Post date: [February 15, 2021, 1:56pm UTC](https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259/11 "2021-02-15T13:56:57Z")
Awesome stuff. Got this too late last night to try anything, but this afternoon looks like a good time to get to it. Thank you for taking the time and offering these suggestions
---
### Author:  [@mhare](https://discuss.elastic.co/u/mhare)
#### Post date: [March 9, 2021, 3:16am UTC](https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259/12 "2021-03-09T03:16:25Z")
OK Finally got the cycles to get back to this. @stephenb you're response led me to a correct implementation. Only real difference was using the ISO for the date format. Appreciate the help!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 6, 2021, 3:17am UTC](https://discuss.elastic.co/t/cannot-seem-to-get-ingest-pipelines-to-work-please-help/264259/13 "2021-04-06T03:17:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.