# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=445 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 446 --- ## [Why different behavior on @metadata vs normal fields?](https://discuss.elastic.co/t/why-different-behavior-on-metadata-vs-normal-fields/208028)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 2\ **Last updated:** [November 18, 2019, 12:26pm UTC](https://discuss.elastic.co/t/why-different-behavior-on-metadata-vs-normal-fields/208028 "2019-11-18T12:26:57Z")
Hi, I have json data which is harvested by filebeat and shipped to logstash. Filebeat encapsulating the orginal json message in message field by default. I didn't change it. What I want to do (in logstash): parse js… --- ## [Logstash 6.8.3 not exporting its metrics, running on CentOS 7](https://discuss.elastic.co/t/logstash-6-8-3-not-exporting-its-metrics-running-on-centos-7/208311)
**Author:** [@Mohit\_Kunjir](https://discuss.elastic.co/u/Mohit_Kunjir)\ **Replies:** 0\ **Last updated:** [November 18, 2019, 12:24pm UTC](https://discuss.elastic.co/t/logstash-6-8-3-not-exporting-its-metrics-running-on-centos-7/208311 "2019-11-18T12:24:41Z")
I am running the rpm version of Logstash 6.8.3 on full GUI based CentOS 7 running locally on VM Player on my desktop. Downloaded the rpm version from official site. This version does not export its metrics via xpack.mo… --- ## [Automating ES template management using Logstash output](https://discuss.elastic.co/t/automating-es-template-management-using-logstash-output/208282)
**Author:** [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\ **Replies:** 0\ **Last updated:** [November 18, 2019, 9:58am UTC](https://discuss.elastic.co/t/automating-es-template-management-using-logstash-output/208282 "2019-11-18T09:58:27Z")
Hello there. I'm a little bit confused in finding a way to manage ES templates. Now I edit them manually via Cerebro. But I'd like to store it in some storage like git and push them automatically. I know that Logstash… --- ## [Reading from a file and immediately writing to another file](https://discuss.elastic.co/t/reading-from-a-file-and-immediately-writing-to-another-file/208068)
**Author:** [@user112233](https://discuss.elastic.co/u/user112233)\ **Replies:** 3\ **Last updated:** [November 18, 2019, 6:25am UTC](https://discuss.elastic.co/t/reading-from-a-file-and-immediately-writing-to-another-file/208068 "2019-11-18T06:25:30Z")
I'm trying to configure logstash to read data from one file, and write to another file. This simple task is for a test. It's simple (C:\\Bitnami\\elk\\logstash\\conf\\test1.conf): input { file { path =\> "C:\\Upload\\DN… --- ## [Is it a good idea to use MongoDB change stream to send http requests to logstash to use the filter pipeline?](https://discuss.elastic.co/t/is-it-a-good-idea-to-use-mongodb-change-stream-to-send-http-requests-to-logstash-to-use-the-filter-pipeline/208249)
**Author:** [@kychanbp](https://discuss.elastic.co/u/kychanbp)\ **Replies:** 0\ **Last updated:** [November 18, 2019, 3:52am UTC](https://discuss.elastic.co/t/is-it-a-good-idea-to-use-mongodb-change-stream-to-send-http-requests-to-logstash-to-use-the-filter-pipeline/208249 "2019-11-18T03:52:14Z")
I am looking for options to sync and enrich mongodb documents into elasticsearch. I found a plugin logstash-input-mongo. But it is not updated since 2017. I also found a golang package called Monstache which is quite a… --- ## [Date Inconsistency between logstash stout and elastic search](https://discuss.elastic.co/t/date-inconsistency-between-logstash-stout-and-elastic-search/208237)
**Author:** [@Incauto](https://discuss.elastic.co/u/Incauto)\ **Replies:** 2\ **Last updated:** [November 18, 2019, 12:52am UTC](https://discuss.elastic.co/t/date-inconsistency-between-logstash-stout-and-elastic-search/208237 "2019-11-18T00:52:37Z")
Hi, I have a folder with multiple files called disco\_\[date\], when I run the comand bin/logstash -f /etc/logstash/conf.d/disco.conf and output only the stdout the dates are ok, they work and are shown in the timestamp, … --- ## [Need to use Aggregate Filer and custom ruby filter](https://discuss.elastic.co/t/need-to-use-aggregate-filer-and-custom-ruby-filter/208239)
**Author:** [@Avinaba\_Ghosh\_Hajra](https://discuss.elastic.co/u/Avinaba_Ghosh_Hajra)\ **Replies:** 0\ **Last updated:** [November 17, 2019, 11:49pm UTC](https://discuss.elastic.co/t/need-to-use-aggregate-filer-and-custom-ruby-filter/208239 "2019-11-17T23:49:38Z")
DB join query with student and department where a student can be enlisted with multiple departments where each department can have multiple teachers. After query getting the following output student\_id student\_name … --- ## [Any solutions please!](https://discuss.elastic.co/t/any-solutions-please/206343)
**Author:** [@pbona](https://discuss.elastic.co/u/pbona)\ **Replies:** 22\ **Last updated:** [November 17, 2019, 11:30pm UTC](https://discuss.elastic.co/t/any-solutions-please/206343 "2019-11-17T23:30:54Z")
I have followed that link https://www.elastic.co/blog/how-to-setup-tls-for-elasticsearch-kibana-logstash-filebeat-with-offline-install-in-linux As the result, I got the below error \[root@elastic logstash\]# systemctl st… --- ## [About Pipelines.yml](https://discuss.elastic.co/t/about-pipelines-yml/208197)
**Author:** [@Incauto](https://discuss.elastic.co/u/Incauto)\ **Replies:** 1\ **Last updated:** [November 17, 2019, 6:45pm UTC](https://discuss.elastic.co/t/about-pipelines-yml/208197 "2019-11-17T18:45:16Z")
If I use pipelines.yml whenever I start logstash my .conf files will automatically run? there is no need to run te command: bin/logstash -f /etc/logstash/conf.d/foo.conf ? I have a folder with logs that daily receive … --- ## [How to merge documents from logstash output plugin to elasticsearch index while indexing](https://discuss.elastic.co/t/how-to-merge-documents-from-logstash-output-plugin-to-elasticsearch-index-while-indexing/208229)
**Author:** [@Faizan\_Turing](https://discuss.elastic.co/u/Faizan_Turing)\ **Replies:** 0\ **Last updated:** [November 17, 2019, 4:41pm UTC](https://discuss.elastic.co/t/how-to-merge-documents-from-logstash-output-plugin-to-elasticsearch-index-while-indexing/208229 "2019-11-17T16:41:45Z")
I have an elasticsearch index with below structure and sample data like below { studentId:12345, studentName:"abc", age:10, tests:\[ { testId:100, score:70 }, { testId:101, score:60 } \] } and then I have a l… --- ## [Logstash s3 output using role\_arn in GCP](https://discuss.elastic.co/t/logstash-s3-output-using-role-arn-in-gcp/208219)
**Author:** [@Ranika\_Madurawe](https://discuss.elastic.co/u/Ranika_Madurawe)\ **Replies:** 0\ **Last updated:** [November 17, 2019, 12:14pm UTC](https://discuss.elastic.co/t/logstash-s3-output-using-role-arn-in-gcp/208219 "2019-11-17T12:14:57Z")
Related to a question I asked earlier relating to uploading log files from a LogStash Instance running on GCP to an s3 bucket. Is it possible to do using role\_arn? Most use cases of role\_arn involve the running of a LogS… --- ## [Logstash not indexing new files](https://discuss.elastic.co/t/logstash-not-indexing-new-files/208116)
**Author:** [@Incauto](https://discuss.elastic.co/u/Incauto)\ **Replies:** 3\ **Last updated:** [November 17, 2019, 8:19am UTC](https://discuss.elastic.co/t/logstash-not-indexing-new-files/208116 "2019-11-17T08:19:28Z")
Hi, I have a directory that periodically receive new files from another machine, the files start with cpu and the date, ej: cpu\_2019\_11\_03 and are updated daily. yesterday I create the index in elastic search via logst… --- ## [At least one primary shard for the security index is unavailable](https://discuss.elastic.co/t/at-least-one-primary-shard-for-the-security-index-is-unavailable/208170)
**Author:** [@kiran\_kumar1](https://discuss.elastic.co/u/kiran_kumar1)\ **Replies:** 0\ **Last updated:** [November 16, 2019, 2:57pm UTC](https://discuss.elastic.co/t/at-least-one-primary-shard-for-the-security-index-is-unavailable/208170 "2019-11-16T14:57:02Z")
Hi Team, when am deploying logstash , am getting below issue from kibana dashboard, actually am trying to enabling logstash pipeline {"statusCode":503,"error":"Service Unavailable","message":"at least one primary shard… --- ## [How to Disable logstash error logging in syslog](https://discuss.elastic.co/t/how-to-disable-logstash-error-logging-in-syslog/208143)
**Author:** [@arash\_moosavi](https://discuss.elastic.co/u/arash_moosavi)\ **Replies:** 1\ **Last updated:** [November 16, 2019, 2:12pm UTC](https://discuss.elastic.co/t/how-to-disable-logstash-error-logging-in-syslog/208143 "2019-11-16T14:12:04Z")
All Logstash errors logs in /var/log/logstash/\* , but all the error logs of Logstash logs in /var/log/syslog too. Is there any way to disable logging errors of Logstash in /var/log/syslog? --- ## [Logstash Syntax :: Putting the Value of One Field into Another?](https://discuss.elastic.co/t/logstash-syntax-putting-the-value-of-one-field-into-another/208128)
**Author:** [@redapplesonly](https://discuss.elastic.co/u/redapplesonly)\ **Replies:** 2\ **Last updated:** [November 16, 2019, 4:32am UTC](https://discuss.elastic.co/t/logstash-syntax-putting-the-value-of-one-field-into-another/208128 "2019-11-16T04:32:31Z")
Hey Logstash Maestros, I have a syntax question. Suppose I have this as the input data going into Logstash: FieldA FieldB FieldC ============================== 6 "apples" "bananas" 11 "grapes"… --- ## [Unable to create index and store data to es](https://discuss.elastic.co/t/unable-to-create-index-and-store-data-to-es/208084)
**Author:** [@Akshay\_Sadawarte](https://discuss.elastic.co/u/Akshay_Sadawarte)\ **Replies:** 2\ **Last updated:** [November 16, 2019, 3:36am UTC](https://discuss.elastic.co/t/unable-to-create-index-and-store-data-to-es/208084 "2019-11-16T03:36:33Z")
i have this conf file but unable to create index to es .conf file input { file { path =\> "D:/filelog.txt" sincedb\_path =\> "NUL" }} output { elasticsearch { hosts =\> \["localhost:9200"\] index=\>"logstash"} stdout… --- ## [What OIDs should I use when monitoring routers and switches?](https://discuss.elastic.co/t/what-oids-should-i-use-when-monitoring-routers-and-switches/203293)
**Author:** [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Replies:** 5\ **Last updated:** [November 16, 2019, 3:11am UTC](https://discuss.elastic.co/t/what-oids-should-i-use-when-monitoring-routers-and-switches/203293 "2019-11-16T03:11:14Z")
I'm in the middle of testing to see if we can monitor everything we need to monitor via Elastic Stack. Linux servers have been easy enough to deal with due to Beats, but our networking infrastructure has proven a bit mor… --- ## [Gork filter with custom pattern](https://discuss.elastic.co/t/gork-filter-with-custom-pattern/207919)
**Author:** [@rp346](https://discuss.elastic.co/u/rp346)\ **Replies:** 3\ **Last updated:** [November 15, 2019, 10:13pm UTC](https://discuss.elastic.co/t/gork-filter-with-custom-pattern/207919 "2019-11-15T22:13:31Z")
I am trying to aggregate outside logs into ElasticSearch cluster running in Kubernetes. Syslogs get published to logstash instance which forward to elasticsearch instance. Syslogs has pattern Timestamp Serial\_Number:Pro… --- ## [Need help with the Grok filter](https://discuss.elastic.co/t/need-help-with-the-grok-filter/207416)
**Author:** [@o1o1o11o1](https://discuss.elastic.co/u/o1o1o11o1)\ **Replies:** 1\ **Last updated:** [November 15, 2019, 8:28pm UTC](https://discuss.elastic.co/t/need-help-with-the-grok-filter/207416 "2019-11-15T20:28:29Z")
I have the following filter in Logstash that parses AWS ELB access logs: filter { grok { match =\> \[ "message", '%{TIMESTAMP\_ISO8601:timestamp} %{NOTSPACE:loadbalancer} %{IP:client\_ip}:%{NUMBER:client\_port:int} (?:… --- ## [Translate Filters :: How to Check the Dictionary if a Value is Listed?](https://discuss.elastic.co/t/translate-filters-how-to-check-the-dictionary-if-a-value-is-listed/207951)
**Author:** [@redapplesonly](https://discuss.elastic.co/u/redapplesonly)\ **Replies:** 2\ **Last updated:** [November 15, 2019, 7:37pm UTC](https://discuss.elastic.co/t/translate-filters-how-to-check-the-dictionary-if-a-value-is-listed/207951 "2019-11-15T19:37:30Z")
Hi Logstash Ninjas, I have a working Logstash instance that inputs log data about my customers. The raw data contains multiple integer fields, which I’ll call “Code1”, “Code2”, “Code3”, and “Code4”: Name Code1 … --- ## [HTTPD24\_ERRORLOG grok pattern issues](https://discuss.elastic.co/t/httpd24-errorlog-grok-pattern-issues/207946)
**Author:** [@CodeName](https://discuss.elastic.co/u/CodeName)\ **Replies:** 4\ **Last updated:** [November 15, 2019, 6:23pm UTC](https://discuss.elastic.co/t/httpd24-errorlog-grok-pattern-issues/207946 "2019-11-15T18:23:22Z")
I am having trouble getting the HTTPD24\_ERRORLOG to work when the module field is blank it works great in the kibana tester but not in logstash i get a grokparsefailue ls config input { stdin{} } filter{ grok{ m… --- ## [How can I send the body of an http request in logstash](https://discuss.elastic.co/t/how-can-i-send-the-body-of-an-http-request-in-logstash/208096)
**Author:** [@vbryandc](https://discuss.elastic.co/u/vbryandc)\ **Replies:** 0\ **Last updated:** [November 15, 2019, 3:27pm UTC](https://discuss.elastic.co/t/how-can-i-send-the-body-of-an-http-request-in-logstash/208096 "2019-11-15T15:27:44Z")
I need to retrieve the response of a POST REST API, for which I have to send the body of this service in json format, for example: { "customerID": "123456", "context": "001" } I have seen that the documentation uses… --- ## [Parse log with different lines in logstash](https://discuss.elastic.co/t/parse-log-with-different-lines-in-logstash/208045)
**Author:** [@ahmedrao](https://discuss.elastic.co/u/ahmedrao)\ **Replies:** 1\ **Last updated:** [November 15, 2019, 3:22pm UTC](https://discuss.elastic.co/t/parse-log-with-different-lines-in-logstash/208045 "2019-11-15T15:22:38Z")
Hi, I am trying to parse following logs: \<0\> 2019-11-08 16:26:03.420 10.5.4.2 \[logsource="" logid="50"\] SystemTrace TID:1 Tracer started. \<1\> 2019-11-08 16:26:03.418 10.5.4.2 \[logsource="" logid="50"\] SystemTrace TID:… --- ## [Percentage symbol](https://discuss.elastic.co/t/percentage-symbol/208022)
**Author:** [@KrisKarrots](https://discuss.elastic.co/u/KrisKarrots)\ **Replies:** 1\ **Last updated:** [November 15, 2019, 3:09pm UTC](https://discuss.elastic.co/t/percentage-symbol/208022 "2019-11-15T15:09:43Z")
Hi, I could not find information on what is the purpose of % symbol when using a field? For example "%{\[Temp1\]\[Temp2\]\[Temp3\]}" Thanks, --- ## [Logash ip address in ELastic search](https://discuss.elastic.co/t/logash-ip-address-in-elastic-search/207987)
**Author:** [@Tejesh](https://discuss.elastic.co/u/Tejesh)\ **Replies:** 1\ **Last updated:** [November 15, 2019, 2:43pm UTC](https://discuss.elastic.co/t/logash-ip-address-in-elastic-search/207987 "2019-11-15T14:43:19Z")
i have 6 logstash instances processing events to elastic search .Is there a way that i can get the Logstash ip from which the event is processed to elastic search as a field --- ## [Unable to parse in grok](https://discuss.elastic.co/t/unable-to-parse-in-grok/208076)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 1\ **Last updated:** [November 15, 2019, 1:26pm UTC](https://discuss.elastic.co/t/unable-to-parse-in-grok/208076 "2019-11-15T13:26:00Z")
Hi Folks, Somehow I am not able to parse the below Text in grok. grokdebug is consistently showing no match. {"org\_name": "google.com", "org\_email": "abuse-support@google.com", } Here are my parser which are pretty b… --- ## [Parsing Dynatrace dashboard reports in XML format](https://discuss.elastic.co/t/parsing-dynatrace-dashboard-reports-in-xml-format/206179)
**Author:** [@Volkan\_E](https://discuss.elastic.co/u/Volkan_E)\ **Replies:** 6\ **Last updated:** [November 15, 2019, 12:52pm UTC](https://discuss.elastic.co/t/parsing-dynatrace-dashboard-reports-in-xml-format/206179 "2019-11-15T12:52:29Z")
I'm trying to use Logstash 7.4.1 to parse XML files that are being generated each minute upon request to the Dynatrace server via REST api. The output will be sent to ElasticSearch. Dynatrace Dashboard Report My goal i… --- ## [Logstash input JDBC dupplica dati](https://discuss.elastic.co/t/logstash-input-jdbc-dupplica-dati/208025)
**Author:** [@daniele.saccon](https://discuss.elastic.co/u/daniele.saccon)\ **Replies:** 0\ **Last updated:** [November 15, 2019, 9:29am UTC](https://discuss.elastic.co/t/logstash-input-jdbc-dupplica-dati/208025 "2019-11-15T09:29:32Z")
Hi everybody I am using logstash 6.8.1 to import data from a DB and in the file set in last\_run\_metadata\_path saves a wrong ID. This results in a duplication of the rows since the ID saved in the file is less than the … --- ## [Problem with fetching data from Kafka logs to Logstash (my producers are consumed back in Kafka)](https://discuss.elastic.co/t/problem-with-fetching-data-from-kafka-logs-to-logstash-my-producers-are-consumed-back-in-kafka/208020)
**Author:** [@Lisa1](https://discuss.elastic.co/u/Lisa1)\ **Replies:** 0\ **Last updated:** [November 15, 2019, 9:04am UTC](https://discuss.elastic.co/t/problem-with-fetching-data-from-kafka-logs-to-logstash-my-producers-are-consumed-back-in-kafka/208020 "2019-11-15T09:04:06Z")
I have trouble with fetching data from Kafka logs to Logstash. Actually I can vizualise my logs correctly in Kibana after using logstash to get them there but when I return back to my Kafka console I realize that my pro… --- ## [Elasticsearch Unreachable error](https://discuss.elastic.co/t/elasticsearch-unreachable-error/205216)
**Author:** [@Leks](https://discuss.elastic.co/u/Leks)\ **Replies:** 1\ **Last updated:** [November 15, 2019, 7:54am UTC](https://discuss.elastic.co/t/elasticsearch-unreachable-error/205216 "2019-11-15T07:54:41Z")
Hi all! I spent the whole night yesterday and the night before yesterday to solve this problem but I couldn't find any solution. What's important I configured all environment on the docker container. I want to send some… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=444) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=446)