# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=192 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 193 --- ## [Logstash Pipeline events emits and received](https://discuss.elastic.co/t/logstash-pipeline-events-emits-and-received/285465)
**Author:** [@vikramdayma](https://discuss.elastic.co/u/vikramdayma)\ **Replies:** 0\ **Last updated:** [September 29, 2021, 11:20am UTC](https://discuss.elastic.co/t/logstash-pipeline-events-emits-and-received/285465 "2021-09-29T11:20:43Z")
Hi, I am setting up a logstash pipeline to get data from sql db. I executed it and monitored it after pipeline completion on kibana server. I saw Events Received and Events Emitted. Both values are different. I did… --- ## [Logstash dropping a lot of logs](https://discuss.elastic.co/t/logstash-dropping-a-lot-of-logs/285429)
**Author:** [@Parthib\_Dutta](https://discuss.elastic.co/u/Parthib_Dutta)\ **Replies:** 2\ **Last updated:** [September 29, 2021, 11:12am UTC](https://discuss.elastic.co/t/logstash-dropping-a-lot-of-logs/285429 "2021-09-29T11:12:44Z")
I have this current setup where Filebeat is reading Lumen logs and forwarding them to Logstash . And Logstash is parsing one of those field in log lines as JSON . I am only seeing partial outputs in my Kibana dashboard a… --- ## [Error with JDBC statement on MaxDB](https://discuss.elastic.co/t/error-with-jdbc-statement-on-maxdb/285392)
**Author:** [@franckfct](https://discuss.elastic.co/u/franckfct)\ **Replies:** 2\ **Last updated:** [September 28, 2021, 7:43pm UTC](https://discuss.elastic.co/t/error-with-jdbc-statement-on-maxdb/285392 "2021-09-28T19:43:09Z")
Hi, I would like to synchronize a MaxDB table to ES via logstash/JDBC, so in my pipeline i have =\> input { jdbc { jdbc\_driver\_library =\> "\/sapdbc.jar" jdbc\_driver\_class =\> "com.sap.dbtech.jdbc.DriverSa… --- ## [Logstash MetricNotFound](https://discuss.elastic.co/t/logstash-metricnotfound/285390)
**Author:** [@rachelyang](https://discuss.elastic.co/u/rachelyang)\ **Replies:** 1\ **Last updated:** [September 28, 2021, 4:05pm UTC](https://discuss.elastic.co/t/logstash-metricnotfound/285390 "2021-09-28T16:05:01Z")
Please have a look at the error in the logstash-plain.log file: \[2021-09-28T15:51:47,918\]\[DEBUG\]\[logstash.agent \] 2021-09-28 15:51:47 - LogStash::Instrument::MetricStore::MetricNotFound - For path: events. Map… --- ## [Insert records for parent/child in same index of elastic search](https://discuss.elastic.co/t/insert-records-for-parent-child-in-same-index-of-elastic-search/285385)
**Author:** [@manjurgani](https://discuss.elastic.co/u/manjurgani)\ **Replies:** 0\ **Last updated:** [September 28, 2021, 3:40pm UTC](https://discuss.elastic.co/t/insert-records-for-parent-child-in-same-index-of-elastic-search/285385 "2021-09-28T15:40:28Z")
I have created a mapping in Elasticsearch like below POST /INDEX { "mappings": { "properties": { "id": { "type": "keyword" }, "join\_field": { "typ… --- ## [Logstash tuning w/ 35 pipelines](https://discuss.elastic.co/t/logstash-tuning-w-35-pipelines/285345)
**Author:** [@pkward](https://discuss.elastic.co/u/pkward)\ **Replies:** 0\ **Last updated:** [September 28, 2021, 11:12am UTC](https://discuss.elastic.co/t/logstash-tuning-w-35-pipelines/285345 "2021-09-28T11:12:39Z")
Hi, I have an Elasticsearch cluster with 24 data nodes. I have about 35 logstash pipelines on a single server that has 4 CPUs and 16 GB RAM. Each Logstash pipeline has 18 filters w/ two of them being Elasticsearch filte… --- ## [Logstash crashes at startup when custom codec is used in the config](https://discuss.elastic.co/t/logstash-crashes-at-startup-when-custom-codec-is-used-in-the-config/285078)
**Author:** [@Rara](https://discuss.elastic.co/u/Rara)\ **Replies:** 1\ **Last updated:** [September 28, 2021, 2:23pm UTC](https://discuss.elastic.co/t/logstash-crashes-at-startup-when-custom-codec-is-used-in-the-config/285078 "2021-09-28T14:23:45Z")
Hi Team I'm participating in the migration of a project from Logstash 6.4.3 to Logstash 7.10.2. The problem is that the new Logstash crashes at startup when one of our custom codecs is used in the configuration. The er… --- ## [After Split again merge the json object](https://discuss.elastic.co/t/after-split-again-merge-the-json-object/285333)
**Author:** [@ansgoya](https://discuss.elastic.co/u/ansgoya)\ **Replies:** 0\ **Last updated:** [September 28, 2021, 9:25am UTC](https://discuss.elastic.co/t/after-split-again-merge-the-json-object/285333 "2021-09-28T09:25:33Z")
Hi Team, I am splitting the data and mutate the fields but i want to again merge the objects so filter { if \[data\] { split { field =\> "data" target =\> "indexData" } if !("" in \[indexData\]\[bcd\])… --- ## [Translate plugin, iterate\_on and item id on fallback](https://discuss.elastic.co/t/translate-plugin-iterate-on-and-item-id-on-fallback/284205)
**Author:** [@elastic\_user\_forum](https://discuss.elastic.co/u/elastic_user_forum)\ **Replies:** 3\ **Last updated:** [September 28, 2021, 8:23am UTC](https://discuss.elastic.co/t/translate-plugin-iterate-on-and-item-id-on-fallback/284205 "2021-09-28T08:23:20Z")
Hi everyone, I'm using ELK 7.13 and I have a question about the translate filter plugin. I have the following dictionary.json { "TUTU": {"identifiant": "TRANSLATE VALUE FOR TUTU"}, "TOTO": {"identifiant": "TRANSLATE… --- ## [Error : Logstash shut down](https://discuss.elastic.co/t/error-logstash-shut-down/285307)
**Author:** [@Busra\_Duygu](https://discuss.elastic.co/u/Busra_Duygu)\ **Replies:** 9\ **Last updated:** [September 28, 2021, 7:59am UTC](https://discuss.elastic.co/t/error-logstash-shut-down/285307 "2021-09-28T07:59:03Z")
Hello friends, I got this error while trying to transfer data to Elasticsearch with logstash. Actually, the transfer started, some data was transferred, but then it gave this error. Does anyone have an idea what could be… --- ## [Join two index ( metricbeat & business )](https://discuss.elastic.co/t/join-two-index-metricbeat-business/285318)
**Author:** [@rodri.gz](https://discuss.elastic.co/u/rodri.gz)\ **Replies:** 2\ **Last updated:** [September 28, 2021, 7:40am UTC](https://discuss.elastic.co/t/join-two-index-metricbeat-business/285318 "2021-09-28T07:40:31Z")
Hello! I will explain my case with an example. I have 2 indexes, one of them is the created by metricbeat and the other is an index that contains business data of my company. I would like to create a canvas. This canv… --- ## [Enrich fields in new index from other index present in kibana using logstash](https://discuss.elastic.co/t/enrich-fields-in-new-index-from-other-index-present-in-kibana-using-logstash/284397)
**Author:** [@Divyank\_Mahalle](https://discuss.elastic.co/u/Divyank_Mahalle)\ **Replies:** 6\ **Last updated:** [September 27, 2021, 1:16pm UTC](https://discuss.elastic.co/t/enrich-fields-in-new-index-from-other-index-present-in-kibana-using-logstash/284397 "2021-09-27T13:16:48Z")
Hi , I am using logstash to create a new index from csv file which is having the index name-"therapy" Another index is "participation" already present in kibana with other fields. Both participation and therapy index … --- ## [GrokPraseFailure - JSON log - Success on Grokdebugger](https://discuss.elastic.co/t/grokprasefailure-json-log-success-on-grokdebugger/285282)
**Author:** [@Athul\_Devkar](https://discuss.elastic.co/u/Athul_Devkar)\ **Replies:** 3\ **Last updated:** [September 27, 2021, 10:48pm UTC](https://discuss.elastic.co/t/grokprasefailure-json-log-success-on-grokdebugger/285282 "2021-09-27T22:48:15Z")
I am facing issues while trying to parse this log. I have tried multiple options but nothing seems to work. No error message in the logs, except for this \_grokprasefailure tag in the records. Can you please help with wha… --- ## [Separate Indexes Per Log File From Two Different Servers (Filebeat -\> Logstash - \> Elastic -\> Kibana)](https://discuss.elastic.co/t/separate-indexes-per-log-file-from-two-different-servers-filebeat-logstash-elastic-kibana/285262)
**Author:** [@kingmilo](https://discuss.elastic.co/u/kingmilo)\ **Replies:** 5\ **Last updated:** [September 27, 2021, 8:50pm UTC](https://discuss.elastic.co/t/separate-indexes-per-log-file-from-two-different-servers-filebeat-logstash-elastic-kibana/285262 "2021-09-27T20:50:28Z")
I have two servers configured with Filebeat to send logs to Logstash which is working properly. I would now like to view each of those logs separately under their respective indexes within Kibana Discover but I am strugg… --- ## [Logstash S3 Output plugin](https://discuss.elastic.co/t/logstash-s3-output-plugin/285276)
**Author:** [@John\_snow](https://discuss.elastic.co/u/John_snow)\ **Replies:** 0\ **Last updated:** [September 27, 2021, 7:58pm UTC](https://discuss.elastic.co/t/logstash-s3-output-plugin/285276 "2021-09-27T19:58:25Z")
Hey all, Trying to send messages to S3 USING Logstash SQS as inout, but getting message in this format %{host} %{message} Also not able to see added filed in the logs. --- ## [Logstash not processing syslog data (logs not arriving in Elastic)](https://discuss.elastic.co/t/logstash-not-processing-syslog-data-logs-not-arriving-in-elastic/285177)
**Author:** [@hazenvs](https://discuss.elastic.co/u/hazenvs)\ **Replies:** 0\ **Last updated:** [September 26, 2021, 8:53pm UTC](https://discuss.elastic.co/t/logstash-not-processing-syslog-data-logs-not-arriving-in-elastic/285177 "2021-09-26T20:53:48Z")
Logstash is up and running: \[root@node1 logstash\]# systemctl status logstash.service ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled) Active: ac… --- ## [How many hosts I can list in logstash output](https://discuss.elastic.co/t/how-many-hosts-i-can-list-in-logstash-output/285093)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 6\ **Last updated:** [September 27, 2021, 1:00pm UTC](https://discuss.elastic.co/t/how-many-hosts-i-can-list-in-logstash-output/285093 "2021-09-27T13:00:04Z")
how many hosts I can list in output section of host part? can I use wildcard? something like this? hosts =\> \["host\_123\*"\] --- ## [Logstash DLQ Metrics](https://discuss.elastic.co/t/logstash-dlq-metrics/285207)
**Author:** [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Replies:** 0\ **Last updated:** [September 27, 2021, 8:54am UTC](https://discuss.elastic.co/t/logstash-dlq-metrics/285207 "2021-09-27T08:54:05Z")
Hi, Are there dead-letter-queue metrics collected by xpack? I don't see any but would have expected to find something. Thx D --- ## [Issue after install Logstash on Mac OS](https://discuss.elastic.co/t/issue-after-install-logstash-on-mac-os/284028)
**Author:** [@INS](https://discuss.elastic.co/u/INS)\ **Replies:** 3\ **Last updated:** [September 27, 2021, 9:09am UTC](https://discuss.elastic.co/t/issue-after-install-logstash-on-mac-os/284028 "2021-09-27T09:09:08Z")
Hi I've met some problem within run Logstash on Mac also I've have followed by all commands and at least try to up Logstash instance by (logstash -f /usr/local/etc/logstash/logstash.conf --path.settings=/usr/local/etc/lo… --- ## [Logstash fails on startup](https://discuss.elastic.co/t/logstash-fails-on-startup/285170)
**Author:** [@hazenvs](https://discuss.elastic.co/u/hazenvs)\ **Replies:** 2\ **Last updated:** [September 26, 2021, 7:36pm UTC](https://discuss.elastic.co/t/logstash-fails-on-startup/285170 "2021-09-26T19:36:16Z")
So I've narrowed this issue down to this line in my example.conf (/etc/logstash/conf.d/example.conf) input { tcp { port =\> 5002 type =\> syslog } udp { port =\> 5002 type =\> syslog } beats { … --- ## [Elasticsearch setup did not complete normally, please review previously logged errors {:message=\>"No Available connections", :exception=\>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}](https://discuss.elastic.co/t/elasticsearch-setup-did-not-complete-normally-please-review-previously-logged-errors-message-no-available-connections-exception-logstash-noconnectionavailableerror/285167)
**Author:** [@Bzpo](https://discuss.elastic.co/u/Bzpo)\ **Replies:** 3\ **Last updated:** [September 26, 2021, 6:45pm UTC](https://discuss.elastic.co/t/elasticsearch-setup-did-not-complete-normally-please-review-previously-logged-errors-message-no-available-connections-exception-logstash-noconnectionavailableerror/285167 "2021-09-26T18:45:50Z")
Hello, I am running Logstash and Elasticsearch both in docker. Elasticsearch takes some time to set up after very first build. Logstash at some moment starts to log errors like the one in topic title and even if Elastics… --- ## [How to Ingest data into Elasticsearch with URL Callback?](https://discuss.elastic.co/t/how-to-ingest-data-into-elasticsearch-with-url-callback/284003)
**Author:** [@e-fo](https://discuss.elastic.co/u/e-fo)\ **Replies:** 4\ **Last updated:** [September 25, 2021, 12:10pm UTC](https://discuss.elastic.co/t/how-to-ingest-data-into-elasticsearch-with-url-callback/284003 "2021-09-25T12:10:33Z")
Hello, thanks for your great community I want to logging some data from some mobile app attribution service (such as Adjust or Appsflyer) into Elasticsearch the admin panel of attribution service allow me to set an url… --- ## [Issue parsing db2diag log using grok filter](https://discuss.elastic.co/t/issue-parsing-db2diag-log-using-grok-filter/281957)
**Author:** [@Atta](https://discuss.elastic.co/u/Atta)\ **Replies:** 3\ **Last updated:** [August 28, 2021, 2:13pm UTC](https://discuss.elastic.co/t/issue-parsing-db2diag-log-using-grok-filter/281957 "2021-08-28T14:13:09Z")
Hi all, I am new in ELK solution and currently I am working on Logstash -\> Elasticsearch -\> Kibana. I need to parse db2diag.log. I am using grok filter for this. The problem is when I use grok debugger website everythi… --- ## [Creating separate elasticsearch docs with batched log messages](https://discuss.elastic.co/t/creating-separate-elasticsearch-docs-with-batched-log-messages/284902)
**Author:** [@sdndude](https://discuss.elastic.co/u/sdndude)\ **Replies:** 2\ **Last updated:** [September 24, 2021, 3:44pm UTC](https://discuss.elastic.co/t/creating-separate-elasticsearch-docs-with-batched-log-messages/284902 "2021-09-24T15:44:22Z")
I have searched for this and have found a few forum entries that can help me figure out how to get the batch msgs (below) separated but I have no idea what to do with them after they are separated. As an example, here i… --- ## [Using multiple hosts in env var / Elasticsearch filter](https://discuss.elastic.co/t/using-multiple-hosts-in-env-var-elasticsearch-filter/285090)
**Author:** [@whatgeorgemade](https://discuss.elastic.co/u/whatgeorgemade)\ **Replies:** 1\ **Last updated:** [September 24, 2021, 2:56pm UTC](https://discuss.elastic.co/t/using-multiple-hosts-in-env-var-elasticsearch-filter/285090 "2021-09-24T14:56:07Z")
Hi all, I have a three node Elasticsearch cluster. I'm using Logstash to do some enrichment by pulling un-enriched documents from an index, using the elasticsearch filter to look up values, then output again to Elastics… --- ## [Logstash-output-syslog plugin fix message and structured data](https://discuss.elastic.co/t/logstash-output-syslog-plugin-fix-message-and-structured-data/285071)
**Author:** [@ZuperZero](https://discuss.elastic.co/u/ZuperZero)\ **Replies:** 0\ **Last updated:** [September 24, 2021, 10:58am UTC](https://discuss.elastic.co/t/logstash-output-syslog-plugin-fix-message-and-structured-data/285071 "2021-09-24T10:58:53Z")
So I tried to use the logstash-output-syslog plugin to send information to a syslog server. I had multiple issues with messages containing all sorts of garbage (duplicate timestamp, duplicate message ID and host field??… --- ## [Unable to parse multiline json data into logstash](https://discuss.elastic.co/t/unable-to-parse-multiline-json-data-into-logstash/284114)
**Author:** [@Magesh\_02](https://discuss.elastic.co/u/Magesh_02)\ **Replies:** 11\ **Last updated:** [September 24, 2021, 6:17am UTC](https://discuss.elastic.co/t/unable-to-parse-multiline-json-data-into-logstash/284114 "2021-09-24T06:17:09Z")
I have tried to parse my json data into logstash data with separate fields. But unable to parse my data, logstash console struck on pipeline starts line input : { "id": 1, "first\_name": "Frank", "las… --- ## [Connect to elasticsearch using logstash](https://discuss.elastic.co/t/connect-to-elasticsearch-using-logstash/285021)
**Author:** [@Nikparab](https://discuss.elastic.co/u/Nikparab)\ **Replies:** 2\ **Last updated:** [September 24, 2021, 12:40am UTC](https://discuss.elastic.co/t/connect-to-elasticsearch-using-logstash/285021 "2021-09-24T00:40:06Z")
I have opensearch cluster in aws us-east region and I have logstash installed on aws ubuntu server i us-west region. Both vpc are peered. Still I am not able to load logs to Elasticsearch using logstash Below is the err… --- ## [Substring with grokPattern](https://discuss.elastic.co/t/substring-with-grokpattern/285026)
**Author:** [@mvasqueznr](https://discuss.elastic.co/u/mvasqueznr)\ **Replies:** 1\ **Last updated:** [September 23, 2021, 8:48pm UTC](https://discuss.elastic.co/t/substring-with-grokpattern/285026 "2021-09-23T20:48:31Z")
Hi. With this example data GigabitEthernet102/0/0/28 = TLU-46356\_CAR\_ONE\_RIVERO\_AUTO\_CENTER\_PRINCIPAL Traffic (SNMP Traffic) Down (The interface is disconnected: ifOperStatus=down (2) (code: PE058)) Im try to get a s… --- ## [Multiple query and aggregation filter](https://discuss.elastic.co/t/multiple-query-and-aggregation-filter/285020)
**Author:** [@Roberto\_B](https://discuss.elastic.co/u/Roberto_B)\ **Replies:** 0\ **Last updated:** [September 23, 2021, 5:42pm UTC](https://discuss.elastic.co/t/multiple-query-and-aggregation-filter/285020 "2021-09-23T17:42:07Z")
Let's assume that i have in input 3 different query that fetch data from 3 different tables. from the first i take for example 4 fields one of them is "id\_test" and it is common and present in the other 2 input query r… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=191) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=193)