# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=469 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 470 --- ## [Add an array using Mutate's add\_field](https://discuss.elastic.co/t/add-an-array-using-mutates-add-field/200946)
**Author:** [@buch11](https://discuss.elastic.co/u/buch11)\ **Replies:** 4\ **Last updated:** [October 1, 2019, 1:06am UTC](https://discuss.elastic.co/t/add-an-array-using-mutates-add-field/200946 "2019-10-01T01:06:25Z")
Hi Guys, I have a logstash pipeline where I am receiving a JSON file as HTTP input and forwarding it to output plugin. I want to introduce below structure to input JSON : "parentField": { "field0": "value0", "arrayN… --- ## [Using a slice/partial var in a logstash config?](https://discuss.elastic.co/t/using-a-slice-partial-var-in-a-logstash-config/201559)
**Author:** [@paul\_m](https://discuss.elastic.co/u/paul_m)\ **Replies:** 2\ **Last updated:** [October 1, 2019, 12:33am UTC](https://discuss.elastic.co/t/using-a-slice-partial-var-in-a-logstash-config/201559 "2019-10-01T00:33:38Z")
Hi there, We are using %{env} as a way to set what Elasticsearch indices Logstash should post to. We combine this with the application type %{component} to make a namespace The problem is that some of our log sources s… --- ## [Fields matched but still Grokparsefaliure?](https://discuss.elastic.co/t/fields-matched-but-still-grokparsefaliure/201642)
**Author:** [@Dan\_Kennedy](https://discuss.elastic.co/u/Dan_Kennedy)\ **Replies:** 3\ **Last updated:** [September 30, 2019, 10:57pm UTC](https://discuss.elastic.co/t/fields-matched-but-still-grokparsefaliure/201642 "2019-09-30T22:57:07Z")
Hi. I am using logstash to collect infrastructure logs but seem to be having an issue. I used a groktest site to check the pattern which appears fine. But in logstash I can see I'm getting a \_grokparsefailure. Can some… --- ## [Error while creating index template in elastisearch](https://discuss.elastic.co/t/error-while-creating-index-template-in-elastisearch/201698)
**Author:** [@dje](https://discuss.elastic.co/u/dje)\ **Replies:** 0\ **Last updated:** [September 30, 2019, 8:35pm UTC](https://discuss.elastic.co/t/error-while-creating-index-template-in-elastisearch/201698 "2019-09-30T20:35:35Z")
Hi everyone, This is my setup : Elasticsearch 7.3.2 + Kibana 7.3.2 on Ubuntu (x-pack/ssl on both) logstash 7.3.2 on Ubuntu I get the following error while trying to run logstash against ES : $ sudo /usr/share/logs… --- ## [Conditional Index Value in output](https://discuss.elastic.co/t/conditional-index-value-in-output/201448)
**Author:** [@ccaviris89](https://discuss.elastic.co/u/ccaviris89)\ **Replies:** 2\ **Last updated:** [September 30, 2019, 4:08pm UTC](https://discuss.elastic.co/t/conditional-index-value-in-output/201448 "2019-09-30T16:08:28Z")
I was looking at this and trying to work in conditional logic on my pipeline.conf output so that if a given field exists, the index with have one values, and otherwise it will default to another value. I currently have … --- ## [Logstash dynamic fields](https://discuss.elastic.co/t/logstash-dynamic-fields/201487)
**Author:** [@Ankit\_Sawant](https://discuss.elastic.co/u/Ankit_Sawant)\ **Replies:** 3\ **Last updated:** [September 30, 2019, 3:53pm UTC](https://discuss.elastic.co/t/logstash-dynamic-fields/201487 "2019-09-30T15:53:53Z")
I have a field "msg\_string" in my logs which has multiple fields saparated by tilda "~" e.g. msg\_string =firstname~lasname~city~hobby~marks ... position of fields firstname,lastname are at index 0 and 1 the position o… --- ## [Cannot install Salesforce input plugin](https://discuss.elastic.co/t/cannot-install-salesforce-input-plugin/201666)
**Author:** [@MARCO\_RAMBALDI](https://discuss.elastic.co/u/MARCO_RAMBALDI)\ **Replies:** 0\ **Last updated:** [September 30, 2019, 3:13pm UTC](https://discuss.elastic.co/t/cannot-install-salesforce-input-plugin/201666 "2019-09-30T15:13:01Z")
Hi, I tried to install the Salesforce input plugin with the following command: sudo bin/logstash-plugin install logstash-input-salesforce The following error appears: ERROR: Installation Aborted, message: Bundler cou… --- ## [Read files and once daily read all the files again](https://discuss.elastic.co/t/read-files-and-once-daily-read-all-the-files-again/201620)
**Author:** [@ansamHox](https://discuss.elastic.co/u/ansamHox)\ **Replies:** 2\ **Last updated:** [September 30, 2019, 2:54pm UTC](https://discuss.elastic.co/t/read-files-and-once-daily-read-all-the-files-again/201620 "2019-09-30T14:54:37Z")
Hi, I have situation where our database team export around 100 csv files during a day which I need to import to Elasticsearch. It is being done by NFS, and I have logstash constantly running so when new file is created h… --- ## [Filter KV](https://discuss.elastic.co/t/filter-kv/201623)
**Author:** [@franco.federico](https://discuss.elastic.co/u/franco.federico)\ **Replies:** 1\ **Last updated:** [September 30, 2019, 2:16pm UTC](https://discuss.elastic.co/t/filter-kv/201623 "2019-09-30T14:16:46Z")
I'm using KV to filter the data from the raw data from a firewall. I have a field that sometimes has two values, and I'd like to keep only the first one. How could I do to filter only the first one. I attach my filter.… --- ## [Logstash Remove Nested Dynamic-named fields](https://discuss.elastic.co/t/logstash-remove-nested-dynamic-named-fields/201619)
**Author:** [@Jsoini](https://discuss.elastic.co/u/Jsoini)\ **Replies:** 1\ **Last updated:** [September 30, 2019, 2:07pm UTC](https://discuss.elastic.co/t/logstash-remove-nested-dynamic-named-fields/201619 "2019-09-30T14:07:41Z")
Hello. I have a problem with a Logstash pipeline, which is getting events from Filebeat containers running on Kubernetes. The problem is, that some of the events will have fields that have dynamically generated names o… --- ## [How to remove multiple lines and how to pick specifics information from log files](https://discuss.elastic.co/t/how-to-remove-multiple-lines-and-how-to-pick-specifics-information-from-log-files/201645)
**Author:** [@rkcharlie](https://discuss.elastic.co/u/rkcharlie)\ **Replies:** 0\ **Last updated:** [September 30, 2019, 1:11pm UTC](https://discuss.elastic.co/t/how-to-remove-multiple-lines-and-how-to-pick-specifics-information-from-log-files/201645 "2019-09-30T13:11:46Z")
Oracle xxx Capture for xxx Version 12.2.0.1.160823 OGGCORE\_12.2.0.1.0OGGBP\_PLATFORMS\_160805.1058\_FBO Linux, x64, 64bit (optimized), xxx 12c on Aug 5 2016 23:48:26 Copyright (C) 1995, 2016, Oracle and/or… --- ## [Configuring output to remote machine](https://discuss.elastic.co/t/configuring-output-to-remote-machine/201637)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 3\ **Last updated:** [September 30, 2019, 12:55pm UTC](https://discuss.elastic.co/t/configuring-output-to-remote-machine/201637 "2019-09-30T12:55:10Z")
Hi Is it possible to configure the logstash output to a elasticsearch installed on remote ip??? --- ## [Defenxor DSIEM for Event Correlation](https://discuss.elastic.co/t/defenxor-dsiem-for-event-correlation/201638)
**Author:** [@gargantua](https://discuss.elastic.co/u/gargantua)\ **Replies:** 0\ **Last updated:** [September 30, 2019, 12:33pm UTC](https://discuss.elastic.co/t/defenxor-dsiem-for-event-correlation/201638 "2019-09-30T12:33:34Z")
Hello, I want to correlate my events with defenxor/dsiem. Is there anyone who uses dsiem for event correlation? Is it enough for correlation? What are advantages and disadvantages of dsiem? --- ## [Logstash stopped processing because of an error: system Exit](https://discuss.elastic.co/t/logstash-stopped-processing-because-of-an-error-system-exit/201602)
**Author:** [@satyam2593](https://discuss.elastic.co/u/satyam2593)\ **Replies:** 0\ **Last updated:** [September 30, 2019, 8:52am UTC](https://discuss.elastic.co/t/logstash-stopped-processing-because-of-an-error-system-exit/201602 "2019-09-30T08:52:33Z")
I am getting the Logstash error as below. Error: java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit An unexpected error occurred! {:error=\>#\ **Author:** [@namali](https://discuss.elastic.co/u/namali)\ **Replies:** 5\ **Last updated:** [September 30, 2019, 8:29am UTC](https://discuss.elastic.co/t/not-giving-correct-column-from-translate-pluging/201386 "2019-09-30T08:29:57Z") two csv files are hotel.csv hotelcode,hotelname,citycode CountryCityLive.csv citycode,CityName,CountryRegionCode,CountryCode taking all columns from hotel.csv and from CountryCityLive.csv only take CountryRegionCode… --- ## [Hide the credentials in the .conf file](https://discuss.elastic.co/t/hide-the-credentials-in-the-conf-file/201586)
**Author:** [@ErikWarming](https://discuss.elastic.co/u/ErikWarming)\ **Replies:** 2\ **Last updated:** [September 30, 2019, 8:02am UTC](https://discuss.elastic.co/t/hide-the-credentials-in-the-conf-file/201586 "2019-09-30T08:02:54Z")
I am using jdbc input to pull data from a Microsoft Azure SQL server. In my .conf file USERNAME and PASSWORD is in cleartext. Is there any way to hide those, encrypt them or another way to secure them? This is an examp… --- ## [Dynamic field fetching](https://discuss.elastic.co/t/dynamic-field-fetching/201584)
**Author:** [@amanpradhan](https://discuss.elastic.co/u/amanpradhan)\ **Replies:** 1\ **Last updated:** [September 30, 2019, 7:19am UTC](https://discuss.elastic.co/t/dynamic-field-fetching/201584 "2019-09-30T07:19:09Z")
I have a field "msg\_string" in my logs which has multiple fields saparated by tilda "~" e.g. msg\_string =firstname~lasname~city~hobby~marks ... position of fields firstname,lastname are at index 0 and 1 the position o… --- ## [Environment variable for setting logstash CA file for xpack monitoring](https://discuss.elastic.co/t/environment-variable-for-setting-logstash-ca-file-for-xpack-monitoring/200879)
**Author:** [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Replies:** 3\ **Last updated:** [September 30, 2019, 2:38am UTC](https://discuss.elastic.co/t/environment-variable-for-setting-logstash-ca-file-for-xpack-monitoring/200879 "2019-09-30T02:38:28Z")
I am trying to setup Docker Logstash with xpack monitoring, however I am not able to get the environment variable for specifying the CA file location. xpack.monitoring.elasticsearch.ssl.certificate\_authority Here is th… --- ## [Geoip.ip field not translating to coordinates on Region Map](https://discuss.elastic.co/t/geoip-ip-field-not-translating-to-coordinates-on-region-map/201543)
**Author:** [@hackdefendr](https://discuss.elastic.co/u/hackdefendr)\ **Replies:** 0\ **Last updated:** [September 29, 2019, 7:42pm UTC](https://discuss.elastic.co/t/geoip-ip-field-not-translating-to-coordinates-on-region-map/201543 "2019-09-29T19:42:09Z")
Once upon a time I was able to create a Region map that showed circles for location hits based on the geoip.ip field. For some reason now only the geoip.countrycode2 field works. Here is my logstash config: input { … --- ## [Merging fields of documents based on some field](https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413)
**Author:** [@vasek](https://discuss.elastic.co/u/vasek)\ **Replies:** 9\ **Last updated:** [September 29, 2019, 1:28pm UTC](https://discuss.elastic.co/t/merging-fields-of-documents-based-on-some-field/201413 "2019-09-29T13:28:44Z")
Hi Elasticians, is it possible using Logstash to do this: #fields of document 1 connection-id: 1000 ip: 10.88.88.201 #fileds of document 2 connection-id: 1000 uid: user #fields of document 3 connection-id: 1000 messa… --- ## [Logstash permission issue and need help on this](https://discuss.elastic.co/t/logstash-permission-issue-and-need-help-on-this/201515)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 1\ **Last updated:** [September 29, 2019, 12:48pm UTC](https://discuss.elastic.co/t/logstash-permission-issue-and-need-help-on-this/201515 "2019-09-29T12:48:22Z")
Hi Folks, I have directory /var/log/data where my files are being written and those are the docker volumes. drwxrwxr-x 25 pbdocker pbdocker 4096 Sep 29 02:49 data When I am trying to run the logstash t… --- ## [Compatibility with Java 12](https://discuss.elastic.co/t/compatibility-with-java-12/201533)
**Author:** [@rad](https://discuss.elastic.co/u/rad)\ **Replies:** 0\ **Last updated:** [September 29, 2019, 12:41pm UTC](https://discuss.elastic.co/t/compatibility-with-java-12/201533 "2019-09-29T12:41:56Z")
Is logstash 7.3.x compatible with java 12? there's no java 12 column in the compatibility matrix for logstash, while there is for elasticsearch --- ## [Do i need license for logstash JDBC driver?](https://discuss.elastic.co/t/do-i-need-license-for-logstash-jdbc-driver/201524)
**Author:** [@ranjeet.tiwari](https://discuss.elastic.co/u/ranjeet.tiwari)\ **Replies:** 1\ **Last updated:** [September 29, 2019, 11:39am UTC](https://discuss.elastic.co/t/do-i-need-license-for-logstash-jdbc-driver/201524 "2019-09-29T11:39:31Z")
I want to push data in elastic search from sql server using JDBC driver. Am I allowed to use JDBC driver with logstash under Open source license? Do I need license of elasticsearch to use JDBC driver for logstash ? --- ## [Outputting into different indices based on source](https://discuss.elastic.co/t/outputting-into-different-indices-based-on-source/201457)
**Author:** [@d112](https://discuss.elastic.co/u/d112)\ **Replies:** 0\ **Last updated:** [September 27, 2019, 6:54pm UTC](https://discuss.elastic.co/t/outputting-into-different-indices-based-on-source/201457 "2019-09-27T18:54:47Z")
I want to create different indices based on the source of the data. I currently have: input { beats { port =\> 5044 }} filter { if (\[source\] =~ "examiner.log") { json { source =\> "message" … --- ## [Resolved: Exampleconfiguration how to send logs from Logstash Kafka output plugin (8.1.0) to Azure Event Hubs with Kafka enabled](https://discuss.elastic.co/t/resolved-exampleconfiguration-how-to-send-logs-from-logstash-kafka-output-plugin-8-1-0-to-azure-event-hubs-with-kafka-enabled/201485)
**Author:** [@ssddgreg](https://discuss.elastic.co/u/ssddgreg)\ **Replies:** 0\ **Last updated:** [September 28, 2019, 9:45am UTC](https://discuss.elastic.co/t/resolved-exampleconfiguration-how-to-send-logs-from-logstash-kafka-output-plugin-8-1-0-to-azure-event-hubs-with-kafka-enabled/201485 "2019-09-28T09:45:12Z")
This post is for information sharing as there is so little complete documentation on how to connect Kafka clients (2.x) to Azure Event Hubs with Kafka enabled. Feel free to comment and use my examples: --- ## [Elasticstack and Logstash](https://discuss.elastic.co/t/elasticstack-and-logstash/199761)
**Author:** [@bcochofel](https://discuss.elastic.co/u/bcochofel)\ **Replies:** 2\ **Last updated:** [September 28, 2019, 9:43am UTC](https://discuss.elastic.co/t/elasticstack-and-logstash/199761 "2019-09-28T09:43:32Z")
Hi, I'm creating a ELK cluster with 3 master/data nodes, 1 kibana with o coordination-only node and want to separate the logstash nodes. Is it best practice to install logstash with coordination-only node and configure… --- ## [Jdbc input plugin full and incremental updates](https://discuss.elastic.co/t/jdbc-input-plugin-full-and-incremental-updates/201464)
**Author:** [@ma3](https://discuss.elastic.co/u/ma3)\ **Replies:** 2\ **Last updated:** [September 27, 2019, 9:00pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-full-and-incremental-updates/201464 "2019-09-27T21:00:37Z")
I'm using the jdbc input plugin to feed MySQL data into elasticsearch. I'm using sql\_last\_value to send incremental updates to elastic. Is there an automated way to send full updates to elasticsearch say every 24 hours a… --- ## [Netflow setup giving me error](https://discuss.elastic.co/t/netflow-setup-giving-me-error/200761)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 15\ **Last updated:** [September 27, 2019, 7:55pm UTC](https://discuss.elastic.co/t/netflow-setup-giving-me-error/200761 "2019-09-27T19:55:26Z")
Trying to setup simple netflow for test. What am I missing? install new elasticsearch/kibana and all working fine install metricbeat and it works fine as well my kibana and elasticsearch is setup with hostname:9200 a… --- ## [How to prettyprint logstash conf file](https://discuss.elastic.co/t/how-to-prettyprint-logstash-conf-file/201458)
**Author:** [@kmiklas](https://discuss.elastic.co/u/kmiklas)\ **Replies:** 0\ **Last updated:** [September 27, 2019, 6:56pm UTC](https://discuss.elastic.co/t/how-to-prettyprint-logstash-conf-file/201458 "2019-09-27T18:56:44Z")
Anyone know how to prettyprint these files? I've tried a bunch of the online prettyprinters, using JSON, Ruby, C++, Python, and a few others... but they do weird things with comments. How can I prettyprint these files?… --- ## [How to grok then send to kafka as JSON text?](https://discuss.elastic.co/t/how-to-grok-then-send-to-kafka-as-json-text/201456)
**Author:** [@DeliciousPickle](https://discuss.elastic.co/u/DeliciousPickle)\ **Replies:** 0\ **Last updated:** [September 27, 2019, 6:48pm UTC](https://discuss.elastic.co/t/how-to-grok-then-send-to-kafka-as-json-text/201456 "2019-09-27T18:48:23Z")
In my Logstash config I want to read in a syslog, parse the message text to get some field values, then assemble those fields/values into a JSON string which I then output to a kafka topic. I must be missing something, … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=468) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=470)