# Elastic agent with Ingest pipeline
**URL:** https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400
**Category:** Elastic Agent
**Created:** [July 2, 2024, 6:02pm UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400 "2024-07-02T18:02:15Z")
**Posts on this page:** 19
**Page:** 1
### Author:  [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)
#### Post date: [July 2, 2024, 6:02pm UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/1 "2024-07-02T18:02:15Z")
Hi Team,
We are using fleet-managed elastic agents to send log events to elasticsearch. The custom log integration has been done as part of this process and the log events are passed through an ingest pipeline before the events are ingested into Elasticsearch. The pipeline uses a Grok processor with %{COMBINEDAPACHELOG} pattern for the apache access logs however the events are not getting parsed. Using other processors works just fine but the GROK one. I am attaching a sample of log event and ingest pipeline screenshot below. Can anyone help me with this issue?
---
### Author:  [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)
#### Post date: [July 3, 2024, 6:23am UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/2 "2024-07-03T06:23:23Z")
Hi @Ankita_Pachauri, Could you provide more information how configuration looks like ?
Also I guess you are trying to monitor apache web server?
---
### Author:  [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)
#### Post date: [July 3, 2024, 6:56am UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/3 "2024-07-03T06:56:54Z")
Hi Ashish,
Can you please tell me which configuration do you need?
---
### Author:  [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)
#### Post date: [July 3, 2024, 9:54am UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/4 "2024-07-03T09:54:11Z")
on elastic agent side ? Or you configured from fleet?
Also in case apache monitoring, worth checking [apache integration](https://www.elastic.co/docs/current/en/integrations/apache).
---
### Author:  [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)
#### Post date: [July 3, 2024, 10:22am UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/5 "2024-07-03T10:22:07Z")
Hi Ashish,
Thanks for your response. However, i need to use the custom log integration as i have few custom logs in addition to the apache access logs.
---
### Author:  [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)
#### Post date: [July 3, 2024, 4:54pm UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/6 "2024-07-03T16:54:33Z")
Could you share your grok processor request which you have created on Ingest pipeline?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [July 3, 2024, 10:20pm UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/7 "2024-07-03T22:20:07Z")
Hi @Ankita_Pachauri
Please share samples of the raw log lines 3-5
And your ingest pipeline.
---
### Author:  [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)
#### Post date: [July 4, 2024, 3:35pm UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/8 "2024-07-04T15:35:53Z")
Hi Ashish,
Please find the information below:
Sample Log:
```auto
122.161.52.27 - - [02/Jul/2024:08:48:17 +0000] "GET / HTTP/1.1" 503 299 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
```
Request:
```auto
PUT _ingest/pipeline/apache
{
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{COMBINEDAPACHELOG}"
]
}
}
]
}
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [July 4, 2024, 5:46pm UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/9 "2024-07-04T17:46:07Z")
Hi @Ankita_Pachauri
BTW that log message parses with the Built In Apache Integration as @ashishtiwari1993 suggested ... I would perhaps try starting with that.
```auto
POST _ingest/pipeline/logs-apache.access-1.20.0/_simulate
{
"docs": [
{
"_source" : {
"@timestamp" : "2024-07-04T17:42:06.606248379Z",
"message" : "122.161.52.27 - - [02/Jul/2024:08:48:17 +0000] \"GET / HTTP/1.1\" 503 299 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36\""
}
},
{
"_source" : {
"@timestamp" : "2024-07-04T17:42:06.606248379Z",
"message" : """122.161.52.27 - - [02/Jul/2024:08:48:17 +0000] "GET / HTTP/1.1" 503 299 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" """
}
},
{
"_source" : {
"@timestamp" : "2024-07-04T17:42:06.606248379Z",
"message" : """127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)" """
}
}
]
}
```
It Also Parses with the pipeline you supplied but you will need to additional work to properly set timestamp etc.. etc... perhaps try the Apache Integration first
```auto
POST _ingest/pipeline/apache/_simulate
{
"docs": [
{
"_source" : {
"message" : "122.161.52.27 - - [02/Jul/2024:08:48:17 +0000] \"GET / HTTP/1.1\" 503 299 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36\""
}
},
{
"_source" : {
"message" : """122.161.52.27 - - [02/Jul/2024:08:48:17 +0000] "GET / HTTP/1.1" 503 299 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" """
}
},
{
"_source" : {
"message" : """127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)" """
}
}
]
}
```
---
### Author:  [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)
#### Post date: [July 5, 2024, 3:54am UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/10 "2024-07-05T03:54:00Z")
Hi Ashish,
I have custom logs to integrate with this process. I have taken apache logs just for testing purpose.
---
### Author:  [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)
#### Post date: [July 9, 2024, 7:15am UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/11 "2024-07-09T07:15:44Z")
Could you share custom logs then ? You can check complete log pattern behind the [COMBINEDAPACHELOG](https://github.com/logstash-plugins/logstash-patterns-core/blob/4ba9bf573583ad510aaf4bd0b3418bdbe3402585/patterns/httpd).
---
### Author:  [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)
#### Post date: [July 9, 2024, 2:01pm UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/12 "2024-07-09T14:01:33Z")
Hi,
I am using /var/log/messages now for custom logs integration.
Steps followed:
1. Created an index pipeline:
> PUT \_ingest/pipeline/customlogs
> {
> "processors": [
> {
> "grok": {
> "field": "message",
> "patterns": [
> "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:host} %{WORD:process}: %{GREEDYDATA:message1}"
> ],
> "tag": "failprocessor",
> "ignore\_failure": true
> }
> }
> ],
> "on\_failure": [
> {
> "grok": {
> "field": "message",
> "patterns": [
> "%{SYSLOGTIMESTAMP:timestamp} %{GREEDYDATA:message1}"
> ]
> }
> },
> {
> "set": {
> "field": "message",
> "value": "failed!!!!!!!!!!!!"
> }
> }
> ]
> }
1. Created an index template
> PUT \_index\_template/customlogs
> {
> "template": {
> "settings": {
> "index": {
> "default\_pipeline": "customlogs"
> }
> },
> "mappings": {
> "\_routing": {
> "required": false
> },
> "numeric\_detection": false,
> "dynamic\_date\_formats": [
> "strict\_date\_optional\_time",
> "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
> ],
> "dynamic": true,
> "\_source": {
> "excludes": ,
> "includes": ,
> "enabled": true
> },
> "date\_detection": true
> }
> },
> "index\_patterns": [
> "logs-custom\*"
> ],
> "data\_stream": {
> "hidden": false,
> "allow\_custom\_routing": false
> },
> "composed\_of":
> }
1. Added a custom log integration and added path: /var/log/messages
> PUT kbn:/api/fleet/package\_policies/a417e116-044a-4b80-ba7e-1c98df3882aa
> {
> "package": {
> "name": "log",
> "version": "2.3.1"
> },
> "name": "customlog",
> "namespace": "",
> "description": "",
> "policy\_id": "36dc1784-0bd3-43be-9ac0-9ef8a98c13c5",
> "vars": {},
> "inputs": {
> "logs-logfile": {
> "enabled": true,
> "streams": {
> "log.logs": {
> "enabled": true,
> "vars": {
> "paths": [
> "/var/log/messages"
> ],
> "exclude\_files": ,
> "ignore\_older": "72h",
> "data\_stream.dataset": "custom",
> "tags": ,
> "custom": ""
> }
> }
> }
> }
> }
> }
But when we add integration, we don't have any option to add pipeline. Once the integration is completed, it created a managed pipeline, editing which gives a warning that it can break kibana. I tried adding the same pipeline later. However, the processors are not working.
Please help.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [July 10, 2024, 1:04am UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/13 "2024-07-10T01:04:51Z")
Hi @Ankita_Pachauri
[Here](https://www.elastic.co/guide/en/fleet/current/data-streams.html#data-streams-pipelines) is some background
What I would suggest is to do the following ... clean up the templates etc...
Go to integration and add a custom logs integration and set the dataset name as `custom` and save it.
**BTW setting the name to `custom` can be a bit confusing something like `customapp` might be better**
This will create the templates / ingest pipelines etc and you will inherit all the good logs mappings etc.
It also creates the Ingest pipelines framework... which will be ready to automatically call your custom pipeline if you simply name it correctly
So then just name your pipeline
`PUT _ingest/pipeline/logs-custom@custom`
And it will be automatically called... this is the best way to do this...
OR...... you can also create all this with 1 call....
```auto
POST kbn:/api/fleet/epm/custom_integrations
{
"integrationName": "customapp",
"datasets": [
{
"name": "customapp",
"type": "logs"
}
]
}
GET kbn:/api/fleet/epm/packages/customapp
# if you want to clean up
DELETE kbn:/api/fleet/epm/packages/customapp/1.0.0
```
And then name your ingest pipeline in this case
`PUT _ingest/pipeline/logs-customapp@custom`
Hope this helps
---
### Author:  [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)
#### Post date: [July 10, 2024, 10:59am UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/14 "2024-07-10T10:59:23Z")
Thanks @stephenb it worked!!
---
### Author:  [@osztrovszky](https://discuss.elastic.co/u/osztrovszky)
#### Post date: [September 19, 2024, 8:29am UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/15 "2024-09-19T08:29:22Z")
@stephenb I set up Custom log integration with custom pipeline the same way.
When I add the first processor, everything works:
```auto
{
"set": {
"field": "beforejson",
"value": "true"
}
},
```
When I add the second processor, the log line got lost:
```auto
[
{
"set": {
"field": "beforejson",
"value": "true"
}
},
{
"json": {
"field": "message",
"target_field": "json",
"ignore_failure": true
}
}
]
```
Why is it happening and how can I debug pipelines?
I also set failure processor to debug, but the log line does not arrive even with this settings:
```auto
[
{
"set": {
"field": "pipelinefailed",
"value": "true"
}
}
]
```
Please help.
Thanks.
Regards,
Zsolt
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [September 19, 2024, 4:49pm UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/16 "2024-09-19T16:49:14Z")
Hi @osztrovszky Welcome to the community.
A couple of "house keeping"
First, it is generally not a good Idea to add to a Solved Topic as people tend not to look at them. You should open a new topic and you can refer to it.
Also, please try not to `@` people directly with your questions... it is a community forum, open your topic and see if it will get answered..
All that said / all good....
> [@osztrovszky](#):
>
> ```auto
> {
> "json": {
> "field": "message",
> "target_field": "json",
> "ignore_failure": true
> }
> }
>
> ```
most likely this is failing and since you set `"ignore_failure": true` it will just fail silently... and the failure processor will not get called.. for proper failure handling see [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#handling-pipeline-failures)
I would recommend trying the `_simulate` API with your sample documents and see what is failing
> **[Simulate ingest API | Elasticsearch Guide \[8.15\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-ingest-api.html)**
---
### Author:  [@osztrovszky](https://discuss.elastic.co/u/osztrovszky)
#### Post date: [September 24, 2024, 11:26am UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/17 "2024-09-24T11:26:08Z")
Hello,
Thanks for the info, I'll open a new topic next time.
Regarding this issue, I removed the ignore\_failure flag, but still not working.
```auto
[
{
"json": {
"field": "message",
"target_field": "messagejson"
}
}
]
```
If I test it with a document from index, parse happens succesfully.
However, after applying this pipeline, no more lines are arriving from the log files. If I remove this json processor, the lines are arriving again.
Any idea how can I debug it? Where could I find some log lines about this failing pipeline?
Thanks.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [September 24, 2024, 1:46pm UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/18 "2024-09-24T13:46:32Z")
If you share a full document
The full pipeline
And you're mapping etc. We can take a look but with just snippets it's hard to debug
---
### Author:  [@osztrovszky](https://discuss.elastic.co/u/osztrovszky)
#### Post date: [September 24, 2024, 2:20pm UTC](https://discuss.elastic.co/t/elastic-agent-with-ingest-pipeline/362400/19 "2024-09-24T14:20:24Z")
Sure thing, here are the configs: [elastic-sample-logline.json ยท GitHub](https://gist.github.com/osztrovszkyzs-ahrt/27d8f2b632413469348f882530ae1415)
Tell me if there is any other info needed.
Thanks a lot.