# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=938 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 939 --- ## [How to grok using json filter](https://discuss.elastic.co/t/how-to-grok-using-json-filter/94261)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 4\ **Last updated:** [July 27, 2017, 1:52am UTC](https://discuss.elastic.co/t/how-to-grok-using-json-filter/94261 "2017-07-27T01:52:12Z")
Hi, I have json text (errorCode, errorMessage, etc...) in json text ("raw") and I need to grok this text. I assume I need to use the json filter for that. Any chance to get help how to do it? { "timestamp" : 1500… --- ## [Logstash.pipeline - A plugin had an unrecoverable error when path is long](https://discuss.elastic.co/t/logstash-pipeline-a-plugin-had-an-unrecoverable-error-when-path-is-long/93893)
**Author:** [@alex257](https://discuss.elastic.co/u/alex257)\ **Replies:** 1\ **Last updated:** [July 27, 2017, 1:46am UTC](https://discuss.elastic.co/t/logstash-pipeline-a-plugin-had-an-unrecoverable-error-when-path-is-long/93893 "2017-07-27T01:46:40Z")
Hello , I am using file input and the pipeline crashes when the path to the file is too long or the folder name is long 2:03:16.001 \[\[main\]\ **Author:** [@burnzoire](https://discuss.elastic.co/u/burnzoire)\ **Replies:** 1\ **Last updated:** [July 27, 2017, 12:20am UTC](https://discuss.elastic.co/t/there-are-no-external-requests-known-to-support-wildcards-that-dont-support-replacing-their-indices/94574 "2017-07-27T00:20:33Z") I'm trying to log to an elastic cloud cluster but my logstash server keeps throwing this error whenever it receives logs: \[2017-07-26T06:42:27,934\]\[INFO \]\[logstash.outputs.elasticsearch\] retrying failed action with re… --- ## [Logstash docker file input and file output](https://discuss.elastic.co/t/logstash-docker-file-input-and-file-output/94666)
**Author:** [@sslv](https://discuss.elastic.co/u/sslv)\ **Replies:** 2\ **Last updated:** [July 26, 2017, 11:36pm UTC](https://discuss.elastic.co/t/logstash-docker-file-input-and-file-output/94666 "2017-07-26T23:36:54Z")
Hi, I am new to Logstash, trying my hand with docker image. I have followed instructions from official documentation and was able to pull the image successfully. Through this command: docker run --rm -it -v C:\\Users\\xy… --- ## [Creating separate documents from log](https://discuss.elastic.co/t/creating-separate-documents-from-log/94521)
**Author:** [@jukeboxhero85](https://discuss.elastic.co/u/jukeboxhero85)\ **Replies:** 3\ **Last updated:** [July 26, 2017, 11:03pm UTC](https://discuss.elastic.co/t/creating-separate-documents-from-log/94521 "2017-07-26T23:03:43Z")
Hi, is it possible to create separate documents from logstash? what I mean is, I have this log that's one single string, I want to get the first part of the string since that's where the identifier is and the rest are pa… --- ## [S3 Input read from multiple folders?](https://discuss.elastic.co/t/s3-input-read-from-multiple-folders/94708)
**Author:** [@infamous](https://discuss.elastic.co/u/infamous)\ **Replies:** 0\ **Last updated:** [July 26, 2017, 9:49pm UTC](https://discuss.elastic.co/t/s3-input-read-from-multiple-folders/94708 "2017-07-26T21:49:00Z")
Hi all, I want to be able to read in S3 logs from multiple folders within my bucket. For example, my prefix will be something like prefix =\> "logs/2000/\*/\*.log". I heard that this feature is implemented in file inputs,… --- ## [A java application log](https://discuss.elastic.co/t/a-java-application-log/94625)
**Author:** [@Bouali\_Inass](https://discuss.elastic.co/u/Bouali_Inass)\ **Replies:** 1\ **Last updated:** [July 26, 2017, 8:58pm UTC](https://discuss.elastic.co/t/a-java-application-log/94625 "2017-07-26T20:58:29Z")
Hey everyone i have a java application and i wanna visualise her logs with kibana but i dont wanna read those files with filebeat and send them to logstash i want to send them directly to logstash !!! Is there anyway i c… --- ## [Logstash ignore the port defined in pipeline](https://discuss.elastic.co/t/logstash-ignore-the-port-defined-in-pipeline/94505)
**Author:** [@camarar](https://discuss.elastic.co/u/camarar)\ **Replies:** 3\ **Last updated:** [July 26, 2017, 8:37pm UTC](https://discuss.elastic.co/t/logstash-ignore-the-port-defined-in-pipeline/94505 "2017-07-26T20:37:26Z")
Dear, I would like receive help to understand a behavior... My infrastructure is Filebeat -\> Logstash -\> Elasticsearch -\> Kibana Filebeat: We have many server send documents to Logstash using differents pipelines a… --- ## [Logstash - add new field under array of json objects](https://discuss.elastic.co/t/logstash-add-new-field-under-array-of-json-objects/94704)
**Author:** [@prasperl](https://discuss.elastic.co/u/prasperl)\ **Replies:** 0\ **Last updated:** [July 26, 2017, 8:27pm UTC](https://discuss.elastic.co/t/logstash-add-new-field-under-array-of-json-objects/94704 "2017-07-26T20:27:14Z")
I have a JSON input as given below { "id":4, "locale":"en", "restaurants": \[ { "addressLine" : "123 Road Dr", "city" : "City ST", "phone" : "555-555-5555", "distance" : "0" } , { "addressLine" : "456 Fake St… --- ## [Kafka and Logstash](https://discuss.elastic.co/t/kafka-and-logstash/94310)
**Author:** [@armsepehr](https://discuss.elastic.co/u/armsepehr)\ **Replies:** 1\ **Last updated:** [July 26, 2017, 8:23pm UTC](https://discuss.elastic.co/t/kafka-and-logstash/94310 "2017-07-26T20:23:21Z")
I read many articles around Logstash and Kafka and their integration. I know that kafka is scalable producer/consumer message queue, and I can gather, normalize, enrich and index different source with Logstash. In ot… --- ## [Unable to figure out how to get the 'scripts' output for a port into ES with the JSON template](https://discuss.elastic.co/t/unable-to-figure-out-how-to-get-the-scripts-output-for-a-port-into-es-with-the-json-template/94682)
**Author:** [@TheZeke](https://discuss.elastic.co/u/TheZeke)\ **Replies:** 0\ **Last updated:** [July 26, 2017, 6:10pm UTC](https://discuss.elastic.co/t/unable-to-figure-out-how-to-get-the-scripts-output-for-a-port-into-es-with-the-json-template/94682 "2017-07-26T18:10:12Z")
I'm trying to get the script output added to elasticsearch when using --script.. on an nmap scan. I think I'm doing it in the right place but it's not adding anything whatsoever (or even generating an error in logstash)… --- ## [Logstash Not Running - No errors in the logs](https://discuss.elastic.co/t/logstash-not-running-no-errors-in-the-logs/94669)
**Author:** [@httpex](https://discuss.elastic.co/u/httpex)\ **Replies:** 0\ **Last updated:** [July 26, 2017, 5:04pm UTC](https://discuss.elastic.co/t/logstash-not-running-no-errors-in-the-logs/94669 "2017-07-26T17:04:10Z")
When running logstash with the xml filter below I get the following output every second: \[2017-07-26T12:56:16,546\]\[DEBUG\]\[logstash.inputs.file \] globbedfiles: C:/U sers/gpadmin/Downloads/icd10cm\_index\_2017.xml: glo… --- ## [No events in DLQ](https://discuss.elastic.co/t/no-events-in-dlq/94309)
**Author:** [@AndreAga](https://discuss.elastic.co/u/AndreAga)\ **Replies:** 6\ **Last updated:** [July 26, 2017, 2:52pm UTC](https://discuss.elastic.co/t/no-events-in-dlq/94309 "2017-07-26T14:52:19Z")
Hi all, I'm working with Logstash 5.5 and the new Feature DQL. I've installed the plugin version 1.0.5 for LS 5.5: /usr/share/logstash/bin/logstash-plugin list --verbose ... logstash-input-dead\_letter\_queue (1.0.5) l… --- ## [My log has the timestamp splited i need to join various fields into a timestamp unique field](https://discuss.elastic.co/t/my-log-has-the-timestamp-splited-i-need-to-join-various-fields-into-a-timestamp-unique-field/94247)
**Author:** [@marcelo](https://discuss.elastic.co/u/marcelo)\ **Replies:** 4\ **Last updated:** [July 26, 2017, 12:45pm UTC](https://discuss.elastic.co/t/my-log-has-the-timestamp-splited-i-need-to-join-various-fields-into-a-timestamp-unique-field/94247 "2017-07-26T12:45:43Z")
Hi guys, In my application log the fields separated by character ";" ab929caa-1c44-4a1d-bf76-c43e7a5fa4ca;rf;POST;https://my.domain.com/system/list.seam;2017;07;20;11;56;25;586;4;328;172.18.98.33;ajp-internalserver.t… --- ## [GeoIP error: "The database provided is invalid or corrupted"](https://discuss.elastic.co/t/geoip-error-the-database-provided-is-invalid-or-corrupted/92650)
**Author:** [@Zeineb](https://discuss.elastic.co/u/Zeineb)\ **Replies:** 1\ **Last updated:** [July 26, 2017, 12:30pm UTC](https://discuss.elastic.co/t/geoip-error-the-database-provided-is-invalid-or-corrupted/92650 "2017-07-26T12:30:20Z")
Hi, I need your help, I use ELK stack. But I have a probleme with filter geoIP when I add it in my file confirugation I lose everything in visualisation kibana !! and this error in my file logstash-palin.log … --- ## [Could not find gem 'logstash-devutils' in any of the gem sources listed in your Gemfile](https://discuss.elastic.co/t/could-not-find-gem-logstash-devutils-in-any-of-the-gem-sources-listed-in-your-gemfile/94638)
**Author:** [@Shalini\_Goel](https://discuss.elastic.co/u/Shalini_Goel)\ **Replies:** 0\ **Last updated:** [July 26, 2017, 12:23pm UTC](https://discuss.elastic.co/t/could-not-find-gem-logstash-devutils-in-any-of-the-gem-sources-listed-in-your-gemfile/94638 "2017-07-26T12:23:24Z")
Hi All, I am new to Ruby and I was trying to run "bundle install" for logstash-filter-example plugin(git clone https://github.com/logstash-plugins/logstash-filter-example.git) using logstash 2.3 . I am getting below … --- ## [Macros available for identifying the logstash host](https://discuss.elastic.co/t/macros-available-for-identifying-the-logstash-host/94520)
**Author:** [@jclose](https://discuss.elastic.co/u/jclose)\ **Replies:** 2\ **Last updated:** [July 26, 2017, 12:13pm UTC](https://discuss.elastic.co/t/macros-available-for-identifying-the-logstash-host/94520 "2017-07-26T12:13:03Z")
We have several logstash hosts that have the same config, and we like to add a field upon ingestion via the input to help us show which logstash host ingested each line. It helps us to do some graphing, mapping, and che… --- ## [Logstash filter elasticsearch : "Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool"](https://discuss.elastic.co/t/logstash-filter-elasticsearch-elasticsearch-transport-transport-error-cannot-get-new-connection-from-pool/94635)
**Author:** [@Marlon](https://discuss.elastic.co/u/Marlon)\ **Replies:** 0\ **Last updated:** [July 26, 2017, 12:11pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-elasticsearch-transport-transport-error-cannot-get-new-connection-from-pool/94635 "2017-07-26T12:11:29Z")
Hello everyone, i've create this topic because i got a problem with the utilisation of the filter plugin elasticsearch. Here is my situation : I use elasticsearch, logstash and kibana in version 5.5.0 on a Windows Serv… --- ## [Flush Logstash-output-email Queue](https://discuss.elastic.co/t/flush-logstash-output-email-queue/94517)
**Author:** [@Nishant\_Kumar1](https://discuss.elastic.co/u/Nishant_Kumar1)\ **Replies:** 4\ **Last updated:** [July 26, 2017, 11:23am UTC](https://discuss.elastic.co/t/flush-logstash-output-email-queue/94517 "2017-07-26T11:23:07Z")
I configured Filebeats to add new Logfiles in the harvestors' path and restarted the filebeat service on those instances from where the Logs were to be harvested. On the Logstash side, I configured the Logstash-output-… --- ## [\_dateparsefailure error despite appearing correctly](https://discuss.elastic.co/t/-dateparsefailure-error-despite-appearing-correctly/94260)
**Author:** [@JohnWayne](https://discuss.elastic.co/u/JohnWayne)\ **Replies:** 4\ **Last updated:** [July 26, 2017, 11:22am UTC](https://discuss.elastic.co/t/-dateparsefailure-error-despite-appearing-correctly/94260 "2017-07-26T11:22:52Z")
Hi All, I've been testing out the very basics of ELK and I was able to get an index going by reading a simple CSV. As a test, I tried to do ANOTHER index with the same timestamp format, but different columns and it gave… --- ## [Dynamically build the conditional statement from field value](https://discuss.elastic.co/t/dynamically-build-the-conditional-statement-from-field-value/94305)
**Author:** [@staodd](https://discuss.elastic.co/u/staodd)\ **Replies:** 7\ **Last updated:** [July 26, 2017, 10:00am UTC](https://discuss.elastic.co/t/dynamically-build-the-conditional-statement-from-field-value/94305 "2017-07-26T10:00:23Z")
Is it possible to use the value of one field in a json document as an argument to a conditional statement? f.ex: { value1: "test" value2: "This is a string with the value test included" } if value2 =~ /value1/ then… --- ## [Elasticsearch plugin query wilcard](https://discuss.elastic.co/t/elasticsearch-plugin-query-wilcard/94346)
**Author:** [@RickT](https://discuss.elastic.co/u/RickT)\ **Replies:** 2\ **Last updated:** [July 26, 2017, 9:38am UTC](https://discuss.elastic.co/t/elasticsearch-plugin-query-wilcard/94346 "2017-07-26T09:38:22Z")
Hi i've a simple question : Is it possible to use wildcards in query of Elastic input plugin ? Like this, for example : input { elasticsearch { hosts =\> \["myhost"\] index =\> "myindex" query =\> '{ "query… --- ## [One of my nginx access log report "\_jsonparsefailure" after add to elk?](https://discuss.elastic.co/t/one-of-my-nginx-access-log-report--jsonparsefailure-after-add-to-elk/94196)
**Author:** [@KeithTt](https://discuss.elastic.co/u/KeithTt)\ **Replies:** 6\ **Last updated:** [July 26, 2017, 9:00am UTC](https://discuss.elastic.co/t/one-of-my-nginx-access-log-report--jsonparsefailure-after-add-to-elk/94196 "2017-07-26T09:00:46Z")
Here is one piece of my nginx access log: {"@timestamp":"2017-07-22T17:14:23+08:00","host":"117.119.33.237","clientip":"182.246.61.241","remote\_user":"-","request":"GET /s?slot=-1933190001&cb=jsonpCallback\_37×ta… --- ## [SNMPTrap not get any snmp message through logstash](https://discuss.elastic.co/t/snmptrap-not-get-any-snmp-message-through-logstash/94595)
**Author:** [@Michael2](https://discuss.elastic.co/u/Michael2)\ **Replies:** 0\ **Last updated:** [July 26, 2017, 8:30am UTC](https://discuss.elastic.co/t/snmptrap-not-get-any-snmp-message-through-logstash/94595 "2017-07-26T08:30:56Z")
I'm trying to use the snmptrap input plugin. But after I running the logstash, there is no any data in the output. Is anyone meet the same issue? I want to know how to debug it and make it works. Anyone can help me? Than… --- ## [Logstash Output to Elasticsearch Timezone](https://discuss.elastic.co/t/logstash-output-to-elasticsearch-timezone/94296)
**Author:** [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\ **Replies:** 4\ **Last updated:** [July 26, 2017, 8:17am UTC](https://discuss.elastic.co/t/logstash-output-to-elasticsearch-timezone/94296 "2017-07-26T08:17:08Z")
This is my output snippet. output { elasticsearch { hosts =\> "localhost" index =\> "logstash-%{+YYYY.MM.dd}" } stdout { codec =\> rubydebug } } On my index, I got the index of the next day. So i don have 2 in… --- ## [Read csv file and insert/update the index based on condition in logstash](https://discuss.elastic.co/t/read-csv-file-and-insert-update-the-index-based-on-condition-in-logstash/94497)
**Author:** [@skodari](https://discuss.elastic.co/u/skodari)\ **Replies:** 1\ **Last updated:** [July 26, 2017, 8:04am UTC](https://discuss.elastic.co/t/read-csv-file-and-insert-update-the-index-based-on-condition-in-logstash/94497 "2017-07-26T08:04:37Z")
Hi folks, I have quick qusetion. My scenario is read csv file and insert/update the index based on condition here my query, I need to filter 'age' field from csv file and check if the age \> 18, then only I should update… --- ## [Private IP Address](https://discuss.elastic.co/t/private-ip-address/94589)
**Author:** [@Bondo](https://discuss.elastic.co/u/Bondo)\ **Replies:** 1\ **Last updated:** [July 26, 2017, 7:57am UTC](https://discuss.elastic.co/t/private-ip-address/94589 "2017-07-26T07:57:59Z")
I am trying to enter in an IF statement to preset the geo information based on some internal IPs, but I'm not sure what the syntax needs to be when listing the range. The example below is for any IP with a 10.x.x.x addre… --- ## [Upgrade to 5.5.1 = gsub fails?](https://discuss.elastic.co/t/upgrade-to-5-5-1-gsub-fails/94534)
**Author:** [@Callahan](https://discuss.elastic.co/u/Callahan)\ **Replies:** 1\ **Last updated:** [July 26, 2017, 7:56am UTC](https://discuss.elastic.co/t/upgrade-to-5-5-1-gsub-fails/94534 "2017-07-26T07:56:13Z")
Hi, While I'm not seeing any errors when doing a config check, my gsub that did work to strip down usernames is not no longer firing. Can't figure out why. It worked fine in 5.0.0. So I use the following to simply st… --- ## [Logstash problem accessing symbolic link config-file during start-up](https://discuss.elastic.co/t/logstash-problem-accessing-symbolic-link-config-file-during-start-up/94129)
**Author:** [@bterhorst](https://discuss.elastic.co/u/bterhorst)\ **Replies:** 2\ **Last updated:** [July 26, 2017, 7:44am UTC](https://discuss.elastic.co/t/logstash-problem-accessing-symbolic-link-config-file-during-start-up/94129 "2017-07-26T07:44:21Z")
Hi, I have a file : c:\\elk\\checkout\\logstash\\bin\\logstash.conf I created a hard symbolic link so that this file is linked to the location where logstash is running. mklink /D C:\\dev\\elk\\logstash-5.4.0\\bin\\logstash.co… --- ## [Multiple logstash nodes use file-output plugin to output messages to one file in a shared file system?](https://discuss.elastic.co/t/multiple-logstash-nodes-use-file-output-plugin-to-output-messages-to-one-file-in-a-shared-file-system/94275)
**Author:** [@KeithTt](https://discuss.elastic.co/u/KeithTt)\ **Replies:** 8\ **Last updated:** [July 26, 2017, 7:41am UTC](https://discuss.elastic.co/t/multiple-logstash-nodes-use-file-output-plugin-to-output-messages-to-one-file-in-a-shared-file-system/94275 "2017-07-26T07:41:31Z")
ELK version: 5.4.1 There are 3 logstash nodes in my BELK arch, and I configure load-balancing in filebeat. if I use output-plugin to output the messages from 3 logstash nodes to a same file in a shared filesystem, wi… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=937) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=939)