# Problem with pipeline, grok and dashboards **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 23, 2021, 3:01pm UTC](https://discuss.elastic.co/t/problem-with-pipeline-grok-and-dashboards/292818 "2021-12-23T15:01:22Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![roonick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roonick/32/90977_2.png) [@roonick](https://discuss.elastic.co/u/roonick)\ **Post date:** [December 23, 2021, 3:01pm UTC](https://discuss.elastic.co/t/problem-with-pipeline-grok-and-dashboards/292818/1 "2021-12-23T15:01:22Z")
Good day! I had following task. There was necessary to add additional field of client ip address from nginx logs. I composed next pipeline: filebeat -\> logstash (beats pipeline, frok) -\> Elasticsearch. Grok pattern works fine... But after passing through logstash there is no some fields in documents. And some dashboards Filebeat-Nginx don't work. Please, help. How can I add necessary field for dashboards? Sorry, my English is not very good. Best regards, roonick ---
**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\ **Post date:** [December 23, 2021, 3:10pm UTC](https://discuss.elastic.co/t/problem-with-pipeline-grok-and-dashboards/292818/2 "2021-12-23T15:10:58Z")
Are you using Kibana to make your dashboards? Be sure to refresh your index patterns if you are adding new fields and are trying to make dashboards. ---
**Author:** ![roonick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roonick/32/90977_2.png) [@roonick](https://discuss.elastic.co/u/roonick)\ **Post date:** [December 24, 2021, 1:26pm UTC](https://discuss.elastic.co/t/problem-with-pipeline-grok-and-dashboards/292818/3 "2021-12-24T13:26:08Z")
I loaded dashboards from filebeat through "filebeat setup". And now I want these dashboards working after logs pass through logstash. That's wrong way? ---
**Author:** ![FALEN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/falen/32/82754_2.png) [@FALEN](https://discuss.elastic.co/u/FALEN)\ **Post date:** [December 24, 2021, 1:32pm UTC](https://discuss.elastic.co/t/problem-with-pipeline-grok-and-dashboards/292818/4 "2021-12-24T13:32:36Z")
Can you check discover page, are those extra fields visible there? If no, can you please share your log example and grok filter configuration? If you can see those fields in discover, but you cant find them in dashboards. Creating visualizationss is different topic ---
**Author:** ![roonick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roonick/32/90977_2.png) [@roonick](https://discuss.elastic.co/u/roonick)\ **Post date:** [December 24, 2021, 2:05pm UTC](https://discuss.elastic.co/t/problem-with-pipeline-grok-and-dashboards/292818/5 "2021-12-24T14:05:47Z")
I think, it's not a grok problem, but problem between filebeat, logstash and Elasticsearch. I turned off grok filtering and had the same result - without needed fields (~170-180 lines json). If I configure filebeat output directly to Elasticsearch, there is few extra fields in json needed for dashboards (~310-310 lines json). For example, "response": {"status\_code": 200,"body": {"bytes": 1450}} (not only this, it's one of many). Probably, these fields are not parsed from "event {"original"}" field in logstash json. Most likely, I need to make logstash parsing these data. But I don't now how ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 24, 2021, 3:37pm UTC](https://discuss.elastic.co/t/problem-with-pipeline-grok-and-dashboards/292818/6 "2021-12-24T15:37:55Z")
What does logstash conf look like? If you want to use the nginx ingest pipeline in Elasticsearch That was setup by filebeat you will need to set it in the Elasticsearch output section of your logstash conf The code below will determine if there is a pipeline defined in the filebeat like nginx and then makes sure it executed. Obviously you can add more to this conf if you want. ```auto ################################################ # beats->logstash->es default config. ################################################ input { beats { port => 5044 } } output { if [@metadata][pipeline] { elasticsearch { hosts => "http://localhost:9200" manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}" pipeline => "%{[@metadata][pipeline]}" user => "elastic" password => "secret" } } else { elasticsearch { hosts => "http://localhost:9200" manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}" user => "elastic" password => "secret" } } } ``` ---
**Author:** ![roonick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roonick/32/90977_2.png) [@roonick](https://discuss.elastic.co/u/roonick)\ **Post date:** [December 30, 2021, 2:34pm UTC](https://discuss.elastic.co/t/problem-with-pipeline-grok-and-dashboards/292818/7 "2021-12-30T14:34:16Z")
My logstash pipeline conf: > input { > beats { > port =\> 5045 > } > } > > output { > elasticsearch { > hosts =\> ["[http://10.10.10.6:9200](http://10.10.10.6:9200)"] > manage\_template =\> false > index =\> "%{[@metadata][beat]}-%{[@metadata][version]}" > pipeline =\> "%{[@metadata][pipeline]}" > } > } As you can see, I already tried to change conf with "pipeline =\>". After that I saw "right" index name in document. But there was no fields needed for dashboards... ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 30, 2021, 3:04pm UTC](https://discuss.elastic.co/t/problem-with-pipeline-grok-and-dashboards/292818/8 "2021-12-30T15:04:53Z")
Exactly what filebeat setup command did you run? I asked this because a lot of folks just run setup with the `–-dashboards` and don't realize other assets need to be loaded. In fact, the most important ones need to be loaded, You should just run `filebeat setup -e` My suggestion is to always get filebeat directly to Elasticsearch up and running and working first... And ONLY after this works ...then introduce Logstash. So in short Make this architecture work first Filebeat -\> Elasticsearch Then get this architecture to work Filebeat -\> Logstash -\> Elasticsearch Here are the steps I would follow this example happens to be metricbeat but the same pattern applies for filebeat. > [@Metricbeat to Logstash to ElasticSearch - Cannot See Any Hosts Defined, But Data Is Definitely Coming In](https://discuss.elastic.co/t/metricbeat-to-logstash-to-elasticsearch-cannot-see-any-hosts-defined-but-data-is-definitely-coming-in/275715/2): > > Hi @jthart Welcome to the community apologies that you're having some struggles getting this set up Perhaps we can help. Assuming you want to run an architecture like this Metricbeat (1 to Many) -\> Logstash -\> Elasticsearch Basically using Logstash as a collect and pass through Here is my recommendation try to resist the urge to make this more complex. Do not try to manually load index templates dashboards anything else follow the quick start / basic setup. Clean everything up we're star… ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 27, 2022, 5:05pm UTC](https://discuss.elastic.co/t/problem-with-pipeline-grok-and-dashboards/292818/9 "2022-01-27T17:05:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.