# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=520 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 521 --- ## [Need to define different index name](https://discuss.elastic.co/t/need-to-define-different-index-name/186451)
**Author:** [@Nishant\_Hurnekar](https://discuss.elastic.co/u/Nishant_Hurnekar)\ **Replies:** 12\ **Last updated:** [June 27, 2019, 1:59pm UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451 "2019-06-27T13:59:40Z")
I am running with ELK cluster 5.6.16. My system as below Filebeat --\> Logstash --\> Elasticsearch (3 server in cluster) --\> KIbana I have few servers where many web vhosts are configured. From that server, I have con… --- ## [Parsing with skipping characters](https://discuss.elastic.co/t/parsing-with-skipping-characters/187840)
**Author:** [@taku](https://discuss.elastic.co/u/taku)\ **Replies:** 0\ **Last updated:** [June 27, 2019, 1:23pm UTC](https://discuss.elastic.co/t/parsing-with-skipping-characters/187840 "2019-06-27T13:23:39Z")
Hello! who can help me? I can not understand there are many files with different structures, but there is a common for all this MQ\_SERVER\_ALIVE, how can I pull out MQ\_SERVER\_ALIVE and MQ\_SERVER\_DOWN from all files? lo… --- ## [Logstash error using logstash -f](https://discuss.elastic.co/t/logstash-error-using-logstash-f/187801)
**Author:** [@kumar4](https://discuss.elastic.co/u/kumar4)\ **Replies:** 5\ **Last updated:** [June 27, 2019, 10:50am UTC](https://discuss.elastic.co/t/logstash-error-using-logstash-f/187801 "2019-06-27T10:50:24Z")
i am using this tutorial all things work fine , but when i created a new index and launch logstash by creating a conf file in /etc/logstash/conf.d the i restarted logstash, so i try to collect data unig "logstash -f… --- ## [Kafka consumers seem to stop working after a period](https://discuss.elastic.co/t/kafka-consumers-seem-to-stop-working-after-a-period/187786)
**Author:** [@eblindeman](https://discuss.elastic.co/u/eblindeman)\ **Replies:** 0\ **Last updated:** [June 27, 2019, 8:53am UTC](https://discuss.elastic.co/t/kafka-consumers-seem-to-stop-working-after-a-period/187786 "2019-06-27T08:53:18Z")
on our production system we use the kafka import to export data to elasticsearch. we read the data from about 10 topics and insert them in indeces with the same name + date in elastic search. We run logstash and elastic… --- ## [Elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=\>"http://199.192.23.237:9200/", :error\_type=\>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :Connection refused (Connection refused)"](https://discuss.elastic.co/t/elasticsearch-attempted-to-resurrect-connection-to-dead-es-instance-but-got-an-error-url-http-199-192-23-237-9200-error-type-logstash-hostunreachableerror-connection-refused-connection-refused/187720)
**Author:** [@nere\_Rasolo](https://discuss.elastic.co/u/nere_Rasolo)\ **Replies:** 0\ **Last updated:** [June 27, 2019, 5:44am UTC](https://discuss.elastic.co/t/elasticsearch-attempted-to-resurrect-connection-to-dead-es-instance-but-got-an-error-url-http-199-192-23-237-9200-error-type-logstash-hostunreachableerror-connection-refused-connection-refused/187720 "2019-06-27T05:44:36Z")
\-I use openJdk 1.8 \*-Elasticsearch 6.8.1+Logstash 6.8.1+mysql \* \*-My file elasticsearch/elasticsearch.yml : \* cluster.name: mamaison node.name: server1.bonjour path.data: /var/lib/elasticsearch path.logs: /var/lo… --- ## [Logstash is trying to connect to xpack in logstash-oss 7.0.1 version](https://discuss.elastic.co/t/logstash-is-trying-to-connect-to-xpack-in-logstash-oss-7-0-1-version/186640)
**Author:** [@akshatsharma](https://discuss.elastic.co/u/akshatsharma)\ **Replies:** 3\ **Last updated:** [June 27, 2019, 6:06am UTC](https://discuss.elastic.co/t/logstash-is-trying-to-connect-to-xpack-in-logstash-oss-7-0-1-version/186640 "2019-06-27T06:06:04Z")
Hi, I am using elasticseach-oss 7.0.1 and logstash-oss 7.0.1 rpms in my docker image, both with searchguard. Elasticsearch is getting deployed and searchguard index is getting created. But when I deploy logstash, it kee… --- ## [Logstash output field order](https://discuss.elastic.co/t/logstash-output-field-order/187723)
**Author:** [@Hengheng1](https://discuss.elastic.co/u/Hengheng1)\ **Replies:** 0\ **Last updated:** [June 27, 2019, 5:48am UTC](https://discuss.elastic.co/t/logstash-output-field-order/187723 "2019-06-27T05:48:02Z")
first of all. I run a kfk.conf which I defined to read json file and output to kafka topic. when I run the kfk.conf , the output order is \[ "name" : "lili", "sex":"w", "age":"20"\]. then I kill the process. Rerun… --- ## [Csv file upload in logstash but data is not showing in kibana and elasticsearch](https://discuss.elastic.co/t/csv-file-upload-in-logstash-but-data-is-not-showing-in-kibana-and-elasticsearch/187508)
**Author:** [@Syed.Ubaid](https://discuss.elastic.co/u/Syed.Ubaid)\ **Replies:** 7\ **Last updated:** [June 27, 2019, 4:32am UTC](https://discuss.elastic.co/t/csv-file-upload-in-logstash-but-data-is-not-showing-in-kibana-and-elasticsearch/187508 "2019-06-27T04:32:14Z")
I am trying to upload countrties csv file on logstash and follow output I get on windows 10. Can some one help me on this so I will upload csv file in logstash D:\\ELK\\Logstash\\logstash-7.1.1\\bin\>logstash --config.test\_… --- ## [Regarding big dictionary (2GB) in filter translate plug-in](https://discuss.elastic.co/t/regarding-big-dictionary-2gb-in-filter-translate-plug-in/187423)
**Author:** [@leeyu](https://discuss.elastic.co/u/leeyu)\ **Replies:** 5\ **Last updated:** [June 27, 2019, 12:51am UTC](https://discuss.elastic.co/t/regarding-big-dictionary-2gb-in-filter-translate-plug-in/187423 "2019-06-27T00:51:55Z")
Hi all, Is a 2GB dictionary considered too big for the filter translate plug-in ? Is there any alternative option (such as access the dictionary from elastic search)? Thanks. --- ## [Logstash elasticsearch input plugin error](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-error/187178)
**Author:** [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Replies:** 4\ **Last updated:** [June 26, 2019, 11:28pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-error/187178 "2019-06-26T23:28:08Z")
While re-indexing data from an index using logstash, at times, it gives the following error: \[2019-06-24T13:45:08,442\]\[INFO \]\[logstash.pipeline \] Starting pipeline {"id"=\>"main", "pipeline.workers"=\>4, "pipeline… --- ## [Unable to start logstash using docker compose](https://discuss.elastic.co/t/unable-to-start-logstash-using-docker-compose/187651)
**Author:** [@nila](https://discuss.elastic.co/u/nila)\ **Replies:** 2\ **Last updated:** [June 26, 2019, 11:10pm UTC](https://discuss.elastic.co/t/unable-to-start-logstash-using-docker-compose/187651 "2019-06-26T23:10:37Z")
Hello, I am trying to configure logstash with docker compose. Here is my configuration. From the logs, it shows it is bringing up my pipeline but crashes after that. can someone please help what went wrong? docker-comp… --- ## [Logstash cloudwatch\_logs input plugin has sincedb\_path issues](https://discuss.elastic.co/t/logstash-cloudwatch-logs-input-plugin-has-sincedb-path-issues/187679)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 4\ **Last updated:** [June 26, 2019, 9:34pm UTC](https://discuss.elastic.co/t/logstash-cloudwatch-logs-input-plugin-has-sincedb-path-issues/187679 "2019-06-26T21:34:00Z")
According to lukewaite: Cloudwatch\_logs input plugin the default sincedb\_path for cloudwatch\_logs is $HOME/.sincedb\*". However, when I use this, I get some logs filtered, and I can see them being processed using the ruby… --- ## [What logstash input plugins can you set sincedb\_path for?](https://discuss.elastic.co/t/what-logstash-input-plugins-can-you-set-sincedb-path-for/187443)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 4\ **Last updated:** [June 26, 2019, 9:25pm UTC](https://discuss.elastic.co/t/what-logstash-input-plugins-can-you-set-sincedb-path-for/187443 "2019-06-26T21:25:24Z")
I have seen some Logstash configuration online like this: input { file { start\_position =\> "beginning" sincedb\_path =\> "/pathToSincedb/sincedb" path =\> "/pathToYourData/\*" } } As we can see, there is th… --- ## [Logstash Date Parsing Error - How to Address](https://discuss.elastic.co/t/logstash-date-parsing-error-how-to-address/187660)
**Author:** [@thadc](https://discuss.elastic.co/u/thadc)\ **Replies:** 1\ **Last updated:** [June 26, 2019, 8:19pm UTC](https://discuss.elastic.co/t/logstash-date-parsing-error-how-to-address/187660 "2019-06-26T20:19:39Z")
Hello, I had a situation where some of our indexes were not being discovered in kibana. The problem turned out to be that grok parser was using %{TIMESTAMP\_ISO8601:logDateTime} for log timestamp field, and kibana requir… --- ## [Logstash 7 timeout between input file modification and output file write control](https://discuss.elastic.co/t/logstash-7-timeout-between-input-file-modification-and-output-file-write-control/187664)
**Author:** [@Alexl](https://discuss.elastic.co/u/Alexl)\ **Replies:** 1\ **Last updated:** [June 26, 2019, 8:16pm UTC](https://discuss.elastic.co/t/logstash-7-timeout-between-input-file-modification-and-output-file-write-control/187664 "2019-06-26T20:16:12Z")
Input logstash configuration file was switched to this content: input { file{ path =\> "C:/l/example.txt" start\_position =\> "beginning" } } output { file { path =\> "C:/l/output.txt" } } Logstash logs contain… --- ## [ELK 7.1.1 with defaults - Logstash doesn't fill index](https://discuss.elastic.co/t/elk-7-1-1-with-defaults-logstash-doesnt-fill-index/186377)
**Author:** [@Alexl](https://discuss.elastic.co/u/Alexl)\ **Replies:** 3\ **Last updated:** [June 26, 2019, 7:19pm UTC](https://discuss.elastic.co/t/elk-7-1-1-with-defaults-logstash-doesnt-fill-index/186377 "2019-06-26T19:19:45Z")
Hello all! I'm using Windows 10 x64. First, I'm running bin/elasticsearch from cmd. http://localhost:9200/\_cat/indices says: green open .kibana\_task\_manager LSRA1gP5TCevDafKx51ecQ 1 0 2 0 53.6kb 53.6kb green open .k… --- ## [Export to CSV with all fields - Logstash 7.0.1](https://discuss.elastic.co/t/export-to-csv-with-all-fields-logstash-7-0-1/187420)
**Author:** [@etsai7](https://discuss.elastic.co/u/etsai7)\ **Replies:** 4\ **Last updated:** [June 26, 2019, 7:25pm UTC](https://discuss.elastic.co/t/export-to-csv-with-all-fields-logstash-7-0-1/187420 "2019-06-26T19:25:03Z")
Hello, Is there a way to configure the output to csv file to include all fields in the index by default so that each one does not have to be manually typed out? Thanks, Eric --- ## [Unable to send data from udp port inlogstash to telegraf's socket listener(udp), neither able to see any error in debug logs](https://discuss.elastic.co/t/unable-to-send-data-from-udp-port-inlogstash-to-telegrafs-socket-listener-udp-neither-able-to-see-any-error-in-debug-logs/187622)
**Author:** [@sgorai](https://discuss.elastic.co/u/sgorai)\ **Replies:** 0\ **Last updated:** [June 26, 2019, 4:17pm UTC](https://discuss.elastic.co/t/unable-to-send-data-from-udp-port-inlogstash-to-telegrafs-socket-listener-udp-neither-able-to-see-any-error-in-debug-logs/187622 "2019-06-26T16:17:29Z")
I am working on a requirement to send data from logstash through udp to telegraf and then to influxdb. I am not seeing any errors in the logs but neither my data is reaching there, i tried sending to both ipv6 and ipv4 u… --- ## [Logstash output conditional not working](https://discuss.elastic.co/t/logstash-output-conditional-not-working/187617)
**Author:** [@pcantea](https://discuss.elastic.co/u/pcantea)\ **Replies:** 1\ **Last updated:** [June 26, 2019, 4:24pm UTC](https://discuss.elastic.co/t/logstash-output-conditional-not-working/187617 "2019-06-26T16:24:35Z")
Sample logstash message { "@version": "1", "agent": { "type": "filebeat", "hostname": "83b529353ae1", "ephemeral\_id": "1df074c7-acb2-4c57-897d-143403cd5af9", "version": "7.1.1", "id": "61831d51-b5a3-47bf-96e8-99e… --- ## [Use older beats version with 7.X Elasticsearch](https://discuss.elastic.co/t/use-older-beats-version-with-7-x-elasticsearch/187624)
**Author:** [@Simon18](https://discuss.elastic.co/u/Simon18)\ **Replies:** 0\ **Last updated:** [June 26, 2019, 4:18pm UTC](https://discuss.elastic.co/t/use-older-beats-version-with-7-x-elasticsearch/187624 "2019-06-26T16:18:36Z")
Hi, Im quiet new to elasticsearch. I have a little (may very simple) Question about the compatibility betwen beats - logstash and elasticsearch. If I want to upgrade my elasticsearch to 7.2 i can see that its compatib… --- ## [Can logstash take in a Json Object straight from a java application](https://discuss.elastic.co/t/can-logstash-take-in-a-json-object-straight-from-a-java-application/187610)
**Author:** [@nhbala](https://discuss.elastic.co/u/nhbala)\ **Replies:** 2\ **Last updated:** [June 26, 2019, 3:38pm UTC](https://discuss.elastic.co/t/can-logstash-take-in-a-json-object-straight-from-a-java-application/187610 "2019-06-26T15:38:55Z")
Is there a way to configure Logstash to take data straight from a Java application without the need to write it to a file? --- ## [Logstash sending json written by Java](https://discuss.elastic.co/t/logstash-sending-json-written-by-java/187583)
**Author:** [@nhbala](https://discuss.elastic.co/u/nhbala)\ **Replies:** 6\ **Last updated:** [June 26, 2019, 3:35pm UTC](https://discuss.elastic.co/t/logstash-sending-json-written-by-java/187583 "2019-06-26T15:35:33Z")
I currently have code that writes a json array to a file. However, logstash only seems to attempt to parse it if I manually go in and touch the file myself, rather than take it in after the initial writing to the file. H… --- ## [Reading logs from DB - Avoiding of duplicate records](https://discuss.elastic.co/t/reading-logs-from-db-avoiding-of-duplicate-records/186996)
**Author:** [@vasek](https://discuss.elastic.co/u/vasek)\ **Replies:** 10\ **Last updated:** [June 26, 2019, 3:13pm UTC](https://discuss.elastic.co/t/reading-logs-from-db-avoiding-of-duplicate-records/186996 "2019-06-26T15:13:53Z")
Hello, I would like to read LOGS from database from table logs. These logs I would like to send using logstash to syslog server. I would to ensure sending events only once: I mark events that are going to be sent I s… --- ## [Logstash gives: Unknown setting '"bucket"' for s3 error](https://discuss.elastic.co/t/logstash-gives-unknown-setting-bucket-for-s3-error/187559)
**Author:** [@K\_B1](https://discuss.elastic.co/u/K_B1)\ **Replies:** 3\ **Last updated:** [June 26, 2019, 2:12pm UTC](https://discuss.elastic.co/t/logstash-gives-unknown-setting-bucket-for-s3-error/187559 "2019-06-26T14:12:32Z")
Referring to: Security for Elasticsearch is now free | Elastic Blog In order to get role-based access control, I upgraded my ELK stack from 6.8.1 to 7.1.1, though it should run on 6.8.1 also. I followed Secure Elas… --- ## [Logstash not starting after upgrade](https://discuss.elastic.co/t/logstash-not-starting-after-upgrade/187573)
**Author:** [@hriegger](https://discuss.elastic.co/u/hriegger)\ **Replies:** 0\ **Last updated:** [June 26, 2019, 1:02pm UTC](https://discuss.elastic.co/t/logstash-not-starting-after-upgrade/187573 "2019-06-26T13:02:30Z")
After upgrading logstash to 7.1.1 and activating xpack security logstash does not start Log: 03-output.conf role logstash\_writer --- ## [Filter if condition in Logstash](https://discuss.elastic.co/t/filter-if-condition-in-logstash/187484)
**Author:** [@Laurent\_Beretti](https://discuss.elastic.co/u/Laurent_Beretti)\ **Replies:** 7\ **Last updated:** [June 26, 2019, 1:01pm UTC](https://discuss.elastic.co/t/filter-if-condition-in-logstash/187484 "2019-06-26T13:01:29Z")
Hello, I'm new in Elastic and now I'm trying to modify fields with if condition, but it doesn't work. Note that I've no error after restarting Logstash. I just want to create a new field (this part is ok) and then I w… --- ## [Problem with Grok of message](https://discuss.elastic.co/t/problem-with-grok-of-message/187394)
**Author:** [@robymemo](https://discuss.elastic.co/u/robymemo)\ **Replies:** 5\ **Last updated:** [June 26, 2019, 12:41pm UTC](https://discuss.elastic.co/t/problem-with-grok-of-message/187394 "2019-06-26T12:41:38Z")
Hi, i'm trying to grok a message (\*) where each field is separated by "|" . I used this sintax %{WORD:project\_name}\\|(?\\[^\\|\]\*)\\|(?\\[^\\|\]\*)\\|(?\\[^\\|\]\*)\\|(?\\[^\\|\]\*)\\|(?\ **Author:** [@deepanshu\_goel](https://discuss.elastic.co/u/deepanshu_goel)\ **Replies:** 9\ **Last updated:** [June 26, 2019, 12:09pm UTC](https://discuss.elastic.co/t/can-anyone-please-help-me-how-can-i-parse-this-firewall-log/186599 "2019-06-26T12:09:50Z") date stamp timestamp(data)private IP \[public IP\]-\>\[public IP(client side)(protocol) -\>i am not able to understand the type of log and i am new in the field so kindly please help me to parse the data of this kind of log --- ## [Ruby exception occurred: undefined method \`split' for nil:NilClass](https://discuss.elastic.co/t/ruby-exception-occurred-undefined-method-split-for-nil-nilclass/187352)
**Author:** [@monika](https://discuss.elastic.co/u/monika)\ **Replies:** 9\ **Last updated:** [June 26, 2019, 11:16am UTC](https://discuss.elastic.co/t/ruby-exception-occurred-undefined-method-split-for-nil-nilclass/187352 "2019-06-26T11:16:16Z")
This is my config for logstash (filter file ) filter { if "ETL\_log\_enriched" in \[tags\] { grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:log\_timestamp} (%{LOGLEVEL:loglevel}|%{WORD:loglevel}) (?\[^\]\]+)- %{GREEDY… --- ## [Can't (yet) decode flowset id 256 from source id 0, because no template to decode it with has been received](https://discuss.elastic.co/t/cant-yet-decode-flowset-id-256-from-source-id-0-because-no-template-to-decode-it-with-has-been-received/186465)
**Author:** [@thirty2](https://discuss.elastic.co/u/thirty2)\ **Replies:** 4\ **Last updated:** [June 26, 2019, 10:34am UTC](https://discuss.elastic.co/t/cant-yet-decode-flowset-id-256-from-source-id-0-because-no-template-to-decode-it-with-has-been-received/186465 "2019-06-26T10:34:53Z")
Hi, my conf file for logstash looks like: input { udp { port =\> 9995 codec =\> netflow { versions =\> \[5, 9\] } type =\> netflow tags =\> "port\_9995" } } output { if "port\_9995" in \[tags\] {… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=519) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=521)