# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=345 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 346 --- ## [I have configured logstash to send logs to diffrent indexes](https://discuss.elastic.co/t/i-have-configured-logstash-to-send-logs-to-diffrent-indexes/236291)
**Author:** [@altaf123](https://discuss.elastic.co/u/altaf123)\ **Replies:** 0\ **Last updated:** [June 9, 2020, 9:01am UTC](https://discuss.elastic.co/t/i-have-configured-logstash-to-send-logs-to-diffrent-indexes/236291 "2020-06-09T09:01:38Z")
i tried something like below, it is creating saparate indexes but i want to redirect only one machine logs to one index, but here in my case logs are clubbing if \[host.hostname\] == "fth-qa" { elasticsearch { hosts … --- ## [Xml PARSING IN LOGSTASH & filebeat](https://discuss.elastic.co/t/xml-parsing-in-logstash-filebeat/233072)
**Author:** [@Puneeth\_S\_B\_Gowda1](https://discuss.elastic.co/u/Puneeth_S_B_Gowda1)\ **Replies:** 1\ **Last updated:** [June 9, 2020, 9:00am UTC](https://discuss.elastic.co/t/xml-parsing-in-logstash-filebeat/233072 "2020-06-09T09:00:23Z")
Hi I got new task in ELK and i never worked on ELK My task is take xml data from mysql and store it in notepad and forward it through filebeat and pars it from logstash my xml i need BedType as saparate value in the c… --- ## [\[2020-06-03T17:14:02,261\]\[WARN \]\[org.logstash.instrument.metrics.gauge.LazyDelegatingGauge\]\[main\] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster\_uuids. This may result in invalid seri](https://discuss.elastic.co/t/2020-06-03t1702-261-warn-org-logstash-instrument-metrics-gauge-lazydelegatinggauge-main-a-gauge-metric-of-an-unknown-type-org-jruby-specialized-rubyarrayoneobject-has-been-created-for-key-cluster-uuids-this-may-result-in-invalid-seri/235540)
**Author:** [@Puneeth\_S\_B\_Gowda1](https://discuss.elastic.co/u/Puneeth_S_B_Gowda1)\ **Replies:** 3\ **Last updated:** [June 9, 2020, 8:55am UTC](https://discuss.elastic.co/t/2020-06-03t1702-261-warn-org-logstash-instrument-metrics-gauge-lazydelegatinggauge-main-a-gauge-metric-of-an-unknown-type-org-jruby-specialized-rubyarrayoneobject-has-been-created-for-key-cluster-uuids-this-may-result-in-invalid-seri/235540 "2020-06-09T08:55:57Z")
\[2020-06-03T17:14:02,261\]\[WARN \]\[org.logstash.instrument.metrics.gauge.LazyDelegatingGauge\]\[main\] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster\_uuids. This… --- ## [How to make Ruby selectively remove the top level field from a specific nested field pattern](https://discuss.elastic.co/t/how-to-make-ruby-selectively-remove-the-top-level-field-from-a-specific-nested-field-pattern/232733)
**Author:** [@kevin\_branch](https://discuss.elastic.co/u/kevin_branch)\ **Replies:** 4\ **Last updated:** [June 9, 2020, 8:53am UTC](https://discuss.elastic.co/t/how-to-make-ruby-selectively-remove-the-top-level-field-from-a-specific-nested-field-pattern/232733 "2020-06-09T08:53:10Z")
My Logstash instance will be receiving JSON records containing a wide variety of nested fields whose names start with \[data\]\[ecs\].... and I need to trim the \[data\] prefix off of all of those field names before indexing. … --- ## [Csv output: add new line](https://discuss.elastic.co/t/csv-output-add-new-line/236265)
**Author:** [@cdra](https://discuss.elastic.co/u/cdra)\ **Replies:** 0\ **Last updated:** [June 9, 2020, 5:31am UTC](https://discuss.elastic.co/t/csv-output-add-new-line/236265 "2020-06-09T05:31:17Z")
My config file is: if \[type\] == "syslog" { csv { fields =\> \["Dia","Hora","Evento","Usuario","Politica","SesionID","Web","Enviado\_Bytes","Recibido\_Bytes","Origen","Destino"\] path =\> "D:\\NavegacionUsuarios\\Navegacion-… --- ## [Mongodb input plugin for logstash](https://discuss.elastic.co/t/mongodb-input-plugin-for-logstash/57254)
**Author:** [@sivapuvvada](https://discuss.elastic.co/u/sivapuvvada)\ **Replies:** 8\ **Last updated:** [June 9, 2020, 6:02am UTC](https://discuss.elastic.co/t/mongodb-input-plugin-for-logstash/57254 "2020-06-09T06:02:00Z")
Anyone could you please provide me the link to download mongoDB input plugin for logstash --- ## [Json filter message filed](https://discuss.elastic.co/t/json-filter-message-filed/236232)
**Author:** [@iramki90](https://discuss.elastic.co/u/iramki90)\ **Replies:** 3\ **Last updated:** [June 9, 2020, 3:41am UTC](https://discuss.elastic.co/t/json-filter-message-filed/236232 "2020-06-09T03:41:53Z")
I currently have 2 different type of log formats coming in Json format. message{ id:scenario1 ip:0.0.0. } message { id:scenario2 message: "This is a message field coming from application" } json { source =\> "message" … --- ## [Convert value of nested date field to UTC with date filter](https://discuss.elastic.co/t/convert-value-of-nested-date-field-to-utc-with-date-filter/236250)
**Author:** [@alxsss](https://discuss.elastic.co/u/alxsss)\ **Replies:** 1\ **Last updated:** [June 9, 2020, 2:41am UTC](https://discuss.elastic.co/t/convert-value-of-nested-date-field-to-utc-with-date-filter/236250 "2020-06-09T02:41:01Z")
Hello, I am trying to convert values of all date fields to UTC. I use date filter, which works fine for top level fields. However, looks like it does not work for nested fields like "abc" : \[ { "dateField" : "2020-0… --- ## [How to handle nil or null values within mutate add field](https://discuss.elastic.co/t/how-to-handle-nil-or-null-values-within-mutate-add-field/236160)
**Author:** [@genehunter29009](https://discuss.elastic.co/u/genehunter29009)\ **Replies:** 6\ **Last updated:** [June 8, 2020, 11:38pm UTC](https://discuss.elastic.co/t/how-to-handle-nil-or-null-values-within-mutate-add-field/236160 "2020-06-08T23:38:09Z")
So I have my data being parsed , transformed correctly however I have this issue where null or nil values cause me problems. "userActions.cdnBusyTime" =\> "%{\[userActions\]\[cdnBusyTime\]}" output ends up looking like this … --- ## [Can't run Logstash on FreeBSD](https://discuss.elastic.co/t/cant-run-logstash-on-freebsd/236240)
**Author:** [@bherczyk](https://discuss.elastic.co/u/bherczyk)\ **Replies:** 0\ **Last updated:** [June 8, 2020, 10:25pm UTC](https://discuss.elastic.co/t/cant-run-logstash-on-freebsd/236240 "2020-06-08T22:25:52Z")
Hello, I try running elastisearch, kibana and logstash on FreeBSD 11.3. Elasticseach and Kibana works very well, but I can't running logstash. When I try to do this, an error occurs: \[ERROR\]\[org.logstash.Logstash \]… --- ## [Aggregate filter for mixed data lines](https://discuss.elastic.co/t/aggregate-filter-for-mixed-data-lines/236110)
**Author:** [@mskadu](https://discuss.elastic.co/u/mskadu)\ **Replies:** 4\ **Last updated:** [June 8, 2020, 7:54pm UTC](https://discuss.elastic.co/t/aggregate-filter-for-mixed-data-lines/236110 "2020-06-08T19:54:03Z")
Hello all! I am trying to use the Aggregate filter plugin to correlate and combine data from two different CSV file inputs which represent API data calls. The idea is to produce a record showing a combined picture. As y… --- ## [Syslog - logs contain dynamic fields(parameters) based on event type, how to index?](https://discuss.elastic.co/t/syslog-logs-contain-dynamic-fields-parameters-based-on-event-type-how-to-index/236144)
**Author:** [@Ilya\_Geller](https://discuss.elastic.co/u/Ilya_Geller)\ **Replies:** 6\ **Last updated:** [June 8, 2020, 7:42pm UTC](https://discuss.elastic.co/t/syslog-logs-contain-dynamic-fields-parameters-based-on-event-type-how-to-index/236144 "2020-06-08T19:42:10Z")
Hi, I am trying to write a syslog collector for to store logs from a Cisco CUCM into elasticsearch. Cisco CUCM sends audit logs and alarms as syslog to my logstash server for different system and application events. T… --- ## [Logstash should not use the "tags" field](https://discuss.elastic.co/t/logstash-should-not-use-the-tags-field/236209)
**Author:** [@elastictake](https://discuss.elastic.co/u/elastictake)\ **Replies:** 0\ **Last updated:** [June 8, 2020, 5:10pm UTC](https://discuss.elastic.co/t/logstash-should-not-use-the-tags-field/236209 "2020-06-08T17:10:33Z")
Logstash should not use the "tags" field The name tags is easy to use in applications And when Elasticsearch mapping is set to strict, it becomes a problem. mapping set to strict, dynamic introduction of \[tags\] withi… --- ## [Calcule elapsed time of a VPN user](https://discuss.elastic.co/t/calcule-elapsed-time-of-a-vpn-user/236199)
**Author:** [@Leonardo\_Posadas](https://discuss.elastic.co/u/Leonardo_Posadas)\ **Replies:** 1\ **Last updated:** [June 8, 2020, 3:39pm UTC](https://discuss.elastic.co/t/calcule-elapsed-time-of-a-vpn-user/236199 "2020-06-08T15:39:36Z")
Hi, What is the best way to calculate the duration or elapsed time of a VPN user trough the logs?. I had thought of it as a timestamp difference. Format Message: Jun 7, 2020 @ 18:11:22.165 message: pptp,ppp,info \ **Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 10\ **Last updated:** [June 8, 2020, 3:28pm UTC](https://discuss.elastic.co/t/insert-in-to-existing-index-using-some-condition/235848 "2020-06-08T15:28:24Z") Here is what I am trying to do I have a existing index with { index= my\_index id=1 fname= xyz lname = abc emp\_num=123 loction=usa } now i am reading new data from new source A which have fnam… --- ## [Ruby Filter URI Request](https://discuss.elastic.co/t/ruby-filter-uri-request/235556)
**Author:** [@aannee](https://discuss.elastic.co/u/aannee)\ **Replies:** 1\ **Last updated:** [June 8, 2020, 3:09pm UTC](https://discuss.elastic.co/t/ruby-filter-uri-request/235556 "2020-06-08T15:09:19Z")
Hi Guys, I have a little problem here and I just really don't know what I'm missing because it's my first time using ruby filter. No result is coming from the output stdout. Hope you can help me. Here's my only data. {… --- ## [Convert UTC datetime to specified timezone](https://discuss.elastic.co/t/convert-utc-datetime-to-specified-timezone/235970)
**Author:** [@not\_found](https://discuss.elastic.co/u/not_found)\ **Replies:** 4\ **Last updated:** [June 8, 2020, 2:09pm UTC](https://discuss.elastic.co/t/convert-utc-datetime-to-specified-timezone/235970 "2020-06-08T14:09:58Z")
Hello, I have this following date : 2020-02-05 15:31:46 UTC I would like to convert it with ruby into Europe/Brussels. I tried : puts Time.parse(mydate2) // 2020-02-05 15:31:46 UTC puts Time.parse(mydate2).zone // UT… --- ## [Handling events after using split](https://discuss.elastic.co/t/handling-events-after-using-split/235929)
**Author:** [@pasno](https://discuss.elastic.co/u/pasno)\ **Replies:** 2\ **Last updated:** [June 8, 2020, 1:16pm UTC](https://discuss.elastic.co/t/handling-events-after-using-split/235929 "2020-06-08T13:16:33Z")
Hi, I'm trying to figure out how to use the split-filter in Logstash correctly. I've read forum-posts and the doc, but don't get to the bottom of this. I'm using the Azure Eventhub plugin to ingest stream of logs. The … --- ## [Logstash jdbc plugin internals](https://discuss.elastic.co/t/logstash-jdbc-plugin-internals/235743)
**Author:** [@luis\_g](https://discuss.elastic.co/u/luis_g)\ **Replies:** 4\ **Last updated:** [June 8, 2020, 1:07pm UTC](https://discuss.elastic.co/t/logstash-jdbc-plugin-internals/235743 "2020-06-08T13:07:30Z")
Hi, I have some questions regarding the internals of logstash when using the jdbc input plugin. I would like to mention that my current logstash configuration is working properly, but I want to know how the jdbc\_page\_s… --- ## [One doc for each array item](https://discuss.elastic.co/t/one-doc-for-each-array-item/235936)
**Author:** [@genehunter29009](https://discuss.elastic.co/u/genehunter29009)\ **Replies:** 9\ **Last updated:** [June 8, 2020, 10:24am UTC](https://discuss.elastic.co/t/one-doc-for-each-array-item/235936 "2020-06-08T10:24:58Z")
I have documents with several arrays. one array I have gotten to work with help here. Thank you very much Badger. I have a 3 arrays I want to be able to visualize , but one is most important. It is useractions array. It… --- ## [Logstash pipeline blocks every write if one index isn't available](https://discuss.elastic.co/t/logstash-pipeline-blocks-every-write-if-one-index-isnt-available/236151)
**Author:** [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Replies:** 0\ **Last updated:** [June 8, 2020, 10:02am UTC](https://discuss.elastic.co/t/logstash-pipeline-blocks-every-write-if-one-index-isnt-available/236151 "2020-06-08T10:02:13Z")
Hi, My issue is that when I receive an event with a timestamp that's in the past, which would be written to the previous date-based index, it'll be blocked due to frozen state. However, that prevents every other event (… --- ## [How to get JSON data from AWS s3 bucket](https://discuss.elastic.co/t/how-to-get-json-data-from-aws-s3-bucket/235386)
**Author:** [@Apollyon](https://discuss.elastic.co/u/Apollyon)\ **Replies:** 4\ **Last updated:** [June 8, 2020, 8:35am UTC](https://discuss.elastic.co/t/how-to-get-json-data-from-aws-s3-bucket/235386 "2020-06-08T08:35:22Z")
Hello, ELK 7.7.0 Logstash 7.7.0 I've been trying to do the below: I have a set up in which the source data is placed in to a s3 bucket. Due to the bucket config the files are saved as .txt. However, the data inside t… --- ## [Elastic not creating index daily based on date](https://discuss.elastic.co/t/elastic-not-creating-index-daily-based-on-date/235753)
**Author:** [@Youssef\_SBAI](https://discuss.elastic.co/u/Youssef_SBAI)\ **Replies:** 12\ **Last updated:** [June 8, 2020, 6:17am UTC](https://discuss.elastic.co/t/elastic-not-creating-index-daily-based-on-date/235753 "2020-06-08T06:17:02Z")
I use Logstash for ingestion in elastic but my index not creating index daily based on date my config logstsh input { beats { port =\> "5044" } } filter { csv { separator =\> "," skip\_header =\> "true" columns =\> \[… --- ## [In Logstash output elastisearch plugin is there a way to get response code from elastic after insert?](https://discuss.elastic.co/t/in-logstash-output-elastisearch-plugin-is-there-a-way-to-get-response-code-from-elastic-after-insert/236049)
**Author:** [@Ullas7](https://discuss.elastic.co/u/Ullas7)\ **Replies:** 2\ **Last updated:** [June 8, 2020, 5:50am UTC](https://discuss.elastic.co/t/in-logstash-output-elastisearch-plugin-is-there-a-way-to-get-response-code-from-elastic-after-insert/236049 "2020-06-08T05:50:48Z")
I trying to insert a document in elasticDB using the following logstash output config elasticsearch{ hosts =\> "localhost:9200" index =\> "indexname" document\_id =\> "%{conversationId}" # action =\> "create" … --- ## [Specifying Logstash filter workers?](https://discuss.elastic.co/t/specifying-logstash-filter-workers/236089)
**Author:** [@mskadu](https://discuss.elastic.co/u/mskadu)\ **Replies:** 4\ **Last updated:** [June 7, 2020, 6:20pm UTC](https://discuss.elastic.co/t/specifying-logstash-filter-workers/236089 "2020-06-07T18:20:29Z")
I am currently trialling the use of the Aggregated Filter plugin within Logstash v7.7 for a specific use case. I noticed that the description the specific Plugin (here) says: You should be very careful to set Logstash … --- ## [Value change in Kibana](https://discuss.elastic.co/t/value-change-in-kibana/236071)
**Author:** [@apurvakhatri](https://discuss.elastic.co/u/apurvakhatri)\ **Replies:** 1\ **Last updated:** [June 7, 2020, 11:58am UTC](https://discuss.elastic.co/t/value-change-in-kibana/236071 "2020-06-07T11:58:25Z")
I have defined my index pattern somewhat to be: %{TIMESTAMP\_ISO8601:record\_time} %{IP:app-ip} %{NUMBER:time-taken}" } I get the app-ip as IP. Now, I know that each app-IP also has an app-Name. I have the app-IP, app-N… --- ## [Want to remove filed who change his position from json response](https://discuss.elastic.co/t/want-to-remove-filed-who-change-his-position-from-json-response/235794)
**Author:** [@Ay\_Sa](https://discuss.elastic.co/u/Ay_Sa)\ **Replies:** 3\ **Last updated:** [June 7, 2020, 8:08am UTC](https://discuss.elastic.co/t/want-to-remove-filed-who-change-his-position-from-json-response/235794 "2020-06-07T08:08:10Z")
hi i need help i got this error on logstash \\], :response=\>{"index"=\>{"\_index"=\>"shodan\_data\_ip\_history", "\_type"=\>"\_doc", "\_id"=\>"sy7ef3IBfJwC4yBc10gf", "status"=\>400, "error"=\>{"type"=\>"map… --- ## [Configuration in Logstash don't match logging of docker container](https://discuss.elastic.co/t/configuration-in-logstash-dont-match-logging-of-docker-container/234205)
**Author:** [@hhwvg7](https://discuss.elastic.co/u/hhwvg7)\ **Replies:** 5\ **Last updated:** [June 6, 2020, 6:28pm UTC](https://discuss.elastic.co/t/configuration-in-logstash-dont-match-logging-of-docker-container/234205 "2020-06-06T18:28:28Z")
I run the next container: docker run -d --log-driver=syslog --log-opt syslog-address=tcp://:5000 --log-opt syslog-facility=daemon --name piet -p 8080:80 httpd But In Kibana I see a \_grokparsefailure. Please help: How… --- ## [Index management aws](https://discuss.elastic.co/t/index-management-aws/236045)
**Author:** [@ranjini](https://discuss.elastic.co/u/ranjini)\ **Replies:** 0\ **Last updated:** [June 6, 2020, 4:57pm UTC](https://discuss.elastic.co/t/index-management-aws/236045 "2020-06-06T16:57:56Z")
Elastic search cluster is managed by AWS. My requirement is to delete the index older than 5 days. Below are the things have done. Through kibana , I have created policy. I applied policy for the index pattern thro… --- ## [Logstash stops streaming/uploading csv files to elastic search after some time](https://discuss.elastic.co/t/logstash-stops-streaming-uploading-csv-files-to-elastic-search-after-some-time/235726)
**Author:** [@pran1](https://discuss.elastic.co/u/pran1)\ **Replies:** 2\ **Last updated:** [June 6, 2020, 8:58am UTC](https://discuss.elastic.co/t/logstash-stops-streaming-uploading-csv-files-to-elastic-search-after-some-time/235726 "2020-06-06T08:58:28Z")
ElasticSearch Configuration 1 zone - 4gb ram - 120GB storage Logstash Configuration 4vcpu - 16gb ram - t2.xlarge instance - logstash 7.7.0 Workflow One tool gets data from different sources and creates csv files i… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=344) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=346)