# Filebeat directly to Elastic ingest node
**URL:** https://discuss.elastic.co/t/filebeat-directly-to-elastic-ingest-node/243194
**Category:** Beats
**Tags:** filebeat
**Created:** [July 30, 2020, 9:46am UTC](https://discuss.elastic.co/t/filebeat-directly-to-elastic-ingest-node/243194 "2020-07-30T09:46:43Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@Axellurcher](https://discuss.elastic.co/u/Axellurcher)
#### Post date: [July 30, 2020, 9:46am UTC](https://discuss.elastic.co/t/filebeat-directly-to-elastic-ingest-node/243194/1 "2020-07-30T09:46:43Z")
I am pretty new to ELK so some actions and questions could be simple.
We are sending logs directly from Filebeats to Elasticsearch without Logstash.
Logs can contain JSON in different fields that also need to be parsed. I have created a pipeline to parse logs, tested it in the developer console, and output was as expected. I have set Filebeat to send logs to this pipeline by adding 'pipeline: application\_pipeline' to filebeat.yml. But in Index Management, I see only my docs.
How to check if Filebeat is sending these logs to the pipeline? Or how to set this correct?
log example:
```auto
{"level":"info","message":"Webhook DeletePrice-{\"_headers\":{\"x-forwarded-proto\":[\"https\"],\"x-requested-with\":[\"\"],\"x-client-ip\":[\"93.84.120.32\"],\"user-agent\":[\"1C+Enterprise\\/8.3\"],\"accept\":[\"application\\/json\"],\"host\":[\"host.com\"],\"content-length\":[\"\"],\"content-type\":[\"\"]},\"company_id\":\"10248103\",\"service_id\":\"102.01.02S\",\"service_type\":\"clientApi\"}","service":"servicename","project":"someproject.com","event_id":"255A854BED569B8D4C21B5DE6D8E109C","payload":[],"date_server":"2020-07-24T11:45:48+00:00","date_unix":1595591148.966919}
{"level":"error","message":"NO service integration","service":"servicename","project":"someproject.com","event_id":"D3986456E5A42AF8574230C29D1D474D","payload":{"exception":{"class":"\\Ship\\Exceptions\\IntegrationException","message":"NO service integration","code":0,"file":"/var/www/builds/someproject.com/build.lab.service-public-api.2020_07_22_12_17_45/app/Containers/Price/UI/API/Controllers/Controller.php:406"}},"date_server":"2020-07-24T08:40:34+00:00","date_unix":1595580034.975073}
{"level":"info","message":"No photo in priceId-3696930","service":"service-private-api","project":"someproject.com","event_id":"FBEDA2C9600BFE11523592114B32BAEB","payload":[],"date_server":"2020-07-24T12:16:40+00:00","date_unix":1595593000.97212}
{"level":"error","message":"C404HttpException: 404 \u0421\u0442\u0440\u0430\u043d\u0438\u0446\u0430 \u043d\u0435 \u043d\u0430\u0439\u0434\u0435\u043d\u0430 in \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/modules\/personal\/controllers\/RobotsController.php:65\nStack trace:\n#0 \/var\/www\/builds\/build.artox-lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(4226): RobotsController->actionIndex()\n#1 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(3739): CInlineAction->runWithParams(Array)\n#2 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(3724): CController->runAction(Object(CInlineAction))\n#3 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(3714): CController->runActionWithFilters(Object(CInlineAction), Array)\n#4 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(1799): CController->run('index')\n#5 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(1719): CWebApplication->runController('personal\/robots...')\n#6 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(1236): CWebApplication->processRequest()\n#7 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/index.php(22): CApplication->run()\n#8 {main}\nREQUEST_URI=\/robots.txt\n---","service":"artox-lab\/classified-platform","project":"someproject.com","event_id":"91a10782a3566a74d5abefa9589c926c","payload":"exception.C404HttpException.404","date_server":"2020-07-29T14:25:34+03:00","date_unix":1596021934.218448}
```
pipeline example:
```auto
PUT _ingest/pipeline/application_pipeline
{
"description" : "Pipeline for parsing application.log for services",
"processors" : [
{
"grok" : {
"field" : "message",
"patterns" : [
"%{JSON:json_message_payload}"
],
"pattern_definitions" : {
"JSON" : "{.*$"
},
"ignore_failure" : true,
"ignore_missing" : true
}
},
{
"remove" : {
"field" : "json_message_payload",
"ignore_failure" : true
}
}
]
}
}
```
output:
```auto
{
"_index" : "application_index",
"_type" : "_doc",
"_id" : "6",
"_version" : 1,
"_seq_no" : 3,
"_primary_term" : 1,
"found" : true,
"_source" : {
"date_server" : "2020-07-29T15:16:17+03:00",
"level" : "error",
"project" : "103by",
"message" : """
C404HttpException: 404 Страница не найдена in /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/modules/personal/components/PersonalController.php:140
Stack trace:
#0 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3737): PersonalController->beforeAction(Object(ShowGalleryPhotoAction))
#1 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3724): CController->runAction(Object(ShowGalleryPhotoAction))
#2 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3714): CController->runActionWithFilters(Object(ShowGalleryPhotoAction), Array)
#3 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1799): CController->run('showGalleryPhot...')
#4 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1719): CWebApplication->runController('personal/galler...')
#5 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1236): CWebApplication->processRequest()
#6 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/index.php(22): CApplication->run()
#7 {main}
REQUEST_URI=/gallery/23609/1439643/
HTTP_REFERER=http://rnpcomr.103.by/gallery/23609/1439643/
---
""",
"date_unix" : 1.596024977817727E9,
"event_id" : "b75c7a1ef2f8780986931b038d2f8599",
"payload" : "exception.C404HttpException.404",
"service" : "artox-lab/classified-platform"
}
}
```
Filebeat config:
```auto
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["elk.artoxlab.com:9200"]
pipeline: application_pipeline
```
---
### Author:  [@mtojek](https://discuss.elastic.co/u/mtojek)
#### Post date: [July 30, 2020, 11:34am UTC](https://discuss.elastic.co/t/filebeat-directly-to-elastic-ingest-node/243194/2 "2020-07-30T11:34:00Z")
The simplest way would be to enable debug logging. You can also observe network traffic on port 9200.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 27, 2020, 11:34am UTC](https://discuss.elastic.co/t/filebeat-directly-to-elastic-ingest-node/243194/3 "2020-08-27T11:34:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.