# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=409 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 410 --- ## [Logstash configuration file should be change in running pod (without restarting container) after modifying configmap](https://discuss.elastic.co/t/logstash-configuration-file-should-be-change-in-running-pod-without-restarting-container-after-modifying-configmap/218109)
**Author:** [@hits.elastic](https://discuss.elastic.co/u/hits.elastic)\ **Replies:** 0\ **Last updated:** [February 6, 2020, 8:05am UTC](https://discuss.elastic.co/t/logstash-configuration-file-should-be-change-in-running-pod-without-restarting-container-after-modifying-configmap/218109 "2020-02-06T08:05:38Z")
Hello, Suppose I have a logstash's pod running in Kubernetes, which is using a configuration file mounted from ConfigMap. When I modify configMap in Kubernetes. I am expecting to changes done in the configuration file … --- ## [Recommended Workflow for creating Logstash configs](https://discuss.elastic.co/t/recommended-workflow-for-creating-logstash-configs/218081)
**Author:** [@BlueTeamNinja](https://discuss.elastic.co/u/BlueTeamNinja)\ **Replies:** 0\ **Last updated:** [February 6, 2020, 4:25am UTC](https://discuss.elastic.co/t/recommended-workflow-for-creating-logstash-configs/218081 "2020-02-06T04:25:17Z")
Pretty much what I said in the subject. I can't seem to find any rock solid resources - what is your workflow for building new configs? I have a relatively small instance and I'm still learning the ropes of index manag… --- ## [Pipeline multiple but no data coming in](https://discuss.elastic.co/t/pipeline-multiple-but-no-data-coming-in/217936)
**Author:** [@skyluke.1987](https://discuss.elastic.co/u/skyluke.1987)\ **Replies:** 2\ **Last updated:** [February 6, 2020, 1:43am UTC](https://discuss.elastic.co/t/pipeline-multiple-but-no-data-coming-in/217936 "2020-02-06T01:43:47Z")
Hi below are the description on the setup. Under /usr/share/logstash/config, located pipelines.yml. 5 - pipeline.id: logstash\_main 6 path.config: "/usr/share/logstash/logstash.conf" 7 8 - pipeline.id: Inpu… --- ## [External Ruby script](https://discuss.elastic.co/t/external-ruby-script/218049)
**Author:** [@Incauto](https://discuss.elastic.co/u/Incauto)\ **Replies:** 1\ **Last updated:** [February 5, 2020, 8:47pm UTC](https://discuss.elastic.co/t/external-ruby-script/218049 "2020-02-05T20:47:25Z")
Hi im trying to use a ruby script outside the logstash conf, the ruby secction in the conf is outside the filter, hostgroup on grok is the field of interest, it can be variable, somethimes has one hostgroup, sometimes 3,… --- ## [Logstash conf error - NoSuchMethodError for PipelineAction::Create](https://discuss.elastic.co/t/logstash-conf-error-nosuchmethoderror-for-pipelineaction-create/217553)
**Author:** [@c95mbq](https://discuss.elastic.co/u/c95mbq)\ **Replies:** 23\ **Last updated:** [February 5, 2020, 7:53pm UTC](https://discuss.elastic.co/t/logstash-conf-error-nosuchmethoderror-for-pipelineaction-create/217553 "2020-02-05T19:53:36Z")
Hi, I've setup elastic-7.1.1 and enabled http.ssl and transport.ssl and this seems to be working fine (I've also installed Kibana). I next lifted an old conf file from a previous version of Logstash that I'm running on… --- ## [Can't index from logstash to Elasticsearch using ILM](https://discuss.elastic.co/t/cant-index-from-logstash-to-elasticsearch-using-ilm/215536)
**Author:** [@anirudhra10](https://discuss.elastic.co/u/anirudhra10)\ **Replies:** 5\ **Last updated:** [February 5, 2020, 6:55pm UTC](https://discuss.elastic.co/t/cant-index-from-logstash-to-elasticsearch-using-ilm/215536 "2020-02-05T18:55:08Z")
Hi Logstash support team, I am trying to configure the output section of logstash to send logs to the right index based on the type of log by using ILM. Here is the relevant logstash conf section: if \[type\] == "usprod… --- ## [Order by class invalid in views, inline functions Logstash JDBC](https://discuss.elastic.co/t/order-by-class-invalid-in-views-inline-functions-logstash-jdbc/217994)
**Author:** [@Mohan\_vel](https://discuss.elastic.co/u/Mohan_vel)\ **Replies:** 1\ **Last updated:** [February 5, 2020, 5:40pm UTC](https://discuss.elastic.co/t/order-by-class-invalid-in-views-inline-functions-logstash-jdbc/217994 "2020-02-05T17:40:17Z")
Hi Team, I am using below config file to index document from MSSQL server to logstash. Here i am using the order by class in SQL statement, but it's throwing error. Attached screenshot for reference. How to order the r… --- ## [Substring a field from Logstash](https://discuss.elastic.co/t/substring-a-field-from-logstash/217990)
**Author:** [@Haalanam](https://discuss.elastic.co/u/Haalanam)\ **Replies:** 2\ **Last updated:** [February 5, 2020, 5:31pm UTC](https://discuss.elastic.co/t/substring-a-field-from-logstash/217990 "2020-02-05T17:31:40Z")
I want create a new field based on an field from an JDBC input i want to use the filter option of Logstash but not sure the how to. My goal is to have something like this: new\_field = substr(old\_field,1,300) i want t… --- ## [How to groking and nested output](https://discuss.elastic.co/t/how-to-groking-and-nested-output/217930)
**Author:** [@Muhammad\_Ariful\_Isla](https://discuss.elastic.co/u/Muhammad_Ariful_Isla)\ **Replies:** 2\ **Last updated:** [February 5, 2020, 5:25pm UTC](https://discuss.elastic.co/t/how-to-groking-and-nested-output/217930 "2020-02-05T17:25:38Z")
Hello, I am having a log from a router and I have parsed with grok. But the thing I am struggling with is that I cannot create nested json object. For Example, proto TCP (ACK,FLAG) To perse above log, I have applied … --- ## [Comparing field values with value from external file without dictionary k/v?](https://discuss.elastic.co/t/comparing-field-values-with-value-from-external-file-without-dictionary-k-v/217876)
**Author:** [@hueyg](https://discuss.elastic.co/u/hueyg)\ **Replies:** 2\ **Last updated:** [February 5, 2020, 3:24pm UTC](https://discuss.elastic.co/t/comparing-field-values-with-value-from-external-file-without-dictionary-k-v/217876 "2020-02-05T15:24:03Z")
I need to compare a field against a simple text file that contains nothing but a column of values. If the field and any value in the text file match then I need to set a field value to true, otherwise false. It appears… --- ## [Logstash not making indices in ES, but everything is fine](https://discuss.elastic.co/t/logstash-not-making-indices-in-es-but-everything-is-fine/217915)
**Author:** [@Sagar\_Mandal](https://discuss.elastic.co/u/Sagar_Mandal)\ **Replies:** 3\ **Last updated:** [February 5, 2020, 1:47pm UTC](https://discuss.elastic.co/t/logstash-not-making-indices-in-es-but-everything-is-fine/217915 "2020-02-05T13:47:38Z")
Hi Team, I'm having this issue where I'm fetching data from filebeat to logstash, but I'm unable to make indices in ES. here is my config of logstash input{ beats{ port =\> "5044" } } filter{ grok { match =\> { … --- ## [Logstash stops processing after forbidden index write](https://discuss.elastic.co/t/logstash-stops-processing-after-forbidden-index-write/217639)
**Author:** [@hlamsc](https://discuss.elastic.co/u/hlamsc)\ **Replies:** 1\ **Last updated:** [February 5, 2020, 1:18pm UTC](https://discuss.elastic.co/t/logstash-stops-processing-after-forbidden-index-write/217639 "2020-02-05T13:18:07Z")
Hello, our logstash stops processing all logs after a few FORBIDDEN/8/index write errors. I guess those are documents that have an old timestamp and the index for that timerange is already frozen. I believe we had a si… --- ## [Cipher is not available](https://discuss.elastic.co/t/cipher-is-not-available/217983)
**Author:** [@sai\_kiran1](https://discuss.elastic.co/u/sai_kiran1)\ **Replies:** 0\ **Last updated:** [February 5, 2020, 11:50am UTC](https://discuss.elastic.co/t/cipher-is-not-available/217983 "2020-02-05T11:50:43Z")
Hello, We see below error when the configuration is run by adding it in pipelines.yml . I can run my configuration manually without any issue Error registering plugin {:pipeline\_id=\>"linux\_shipper\_xxx", :plugin=\>"\ **Author:** [@Ram\_29](https://discuss.elastic.co/u/Ram_29)\ **Replies:** 0\ **Last updated:** [February 5, 2020, 10:33am UTC](https://discuss.elastic.co/t/logstash-unit-testing/217961 "2020-02-05T10:33:04Z") Hi Can I write unit test cases for Logstash Configuration file?? Is there any official tool from ELK for that??? Regards Ram --- ## [Final mapping would have more than 1 type After Ugrade 5.x -\> 6.x](https://discuss.elastic.co/t/final-mapping-would-have-more-than-1-type-after-ugrade-5-x-6-x/217833)
**Author:** [@Fin4l](https://discuss.elastic.co/u/Fin4l)\ **Replies:** 10\ **Last updated:** [February 5, 2020, 10:10am UTC](https://discuss.elastic.co/t/final-mapping-would-have-more-than-1-type-after-ugrade-5-x-6-x/217833 "2020-02-05T10:10:31Z")
Hi, i have problem with mappings after upgrade 5.6 to 6.8. Actually priority is how i can set new template for 6.8 version with multiple types Error "reason"=\>"Rejecting mapping update to \[logstash-2020.02.04\] as the… --- ## [Http\_poller: how to use an environment variable for the url?](https://discuss.elastic.co/t/http-poller-how-to-use-an-environment-variable-for-the-url/217935)
**Author:** [@Michel99\_7](https://discuss.elastic.co/u/Michel99_7)\ **Replies:** 1\ **Last updated:** [February 5, 2020, 9:48am UTC](https://discuss.elastic.co/t/http-poller-how-to-use-an-environment-variable-for-the-url/217935 "2020-02-05T09:48:11Z")
I'm trying to use an env variable to define the url when using the http\_poller input plugin. But I have an error about the url encoding which is not good. How to encode it? http\_poller { urls =\> { test =\> { url … --- ## [Sql\_Last\_Value date format seems to be irrelevant](https://discuss.elastic.co/t/sql-last-value-date-format-seems-to-be-irrelevant/217790)
**Author:** [@Mohan\_vel](https://discuss.elastic.co/u/Mohan_vel)\ **Replies:** 4\ **Last updated:** [February 5, 2020, 9:08am UTC](https://discuss.elastic.co/t/sql-last-value-date-format-seems-to-be-irrelevant/217790 "2020-02-05T09:08:14Z")
Hi All i am using logstash to index document from MSSQL server to elasticsearch, i am using below config file in order to do an incremental indexing, for that i am using using column called modified\_date but having probl… --- ## [Logstash-SNMP Trap: doesnt read yamlmibdir when it run as service](https://discuss.elastic.co/t/logstash-snmp-trap-doesnt-read-yamlmibdir-when-it-run-as-service/217897)
**Author:** [@Vian\_Handika](https://discuss.elastic.co/u/Vian_Handika)\ **Replies:** 1\ **Last updated:** [February 5, 2020, 6:20am UTC](https://discuss.elastic.co/t/logstash-snmp-trap-doesnt-read-yamlmibdir-when-it-run-as-service/217897 "2020-02-05T06:20:49Z")
Hello, i am sorry before i am a newbie here... i have stuck at this things, i have a problem that my logstash doesnt read snmptrap input for external yaml mib directory when its run as a service (systemctl start logstas… --- ## [Creating pipeline for windows using logstash](https://discuss.elastic.co/t/creating-pipeline-for-windows-using-logstash/217726)
**Author:** [@Ananya\_Mohiley](https://discuss.elastic.co/u/Ananya_Mohiley)\ **Replies:** 2\ **Last updated:** [February 5, 2020, 4:02am UTC](https://discuss.elastic.co/t/creating-pipeline-for-windows-using-logstash/217726 "2020-02-05T04:02:20Z")
i have written the following code for ingesting the files into elastic search /\*input{ file{ path=\>"D:\\SOATE-41\\project\\DOCUMENTATION\\interim\\datacsv.csv" start\_position=\>"beginning" sincedb\_path=\>"/dev/null" } } … --- ## [Grok, variable lenght array beside normal fields](https://discuss.elastic.co/t/grok-variable-lenght-array-beside-normal-fields/217864)
**Author:** [@Incauto](https://discuss.elastic.co/u/Incauto)\ **Replies:** 5\ **Last updated:** [February 5, 2020, 2:42am UTC](https://discuss.elastic.co/t/grok-variable-lenght-array-beside-normal-fields/217864 "2020-02-05T02:42:49Z")
Hi, I have a log tat contains an array and normal fields, separated by a space field1 field2 \['27', '32'\] field4 my grok will be %{DATA:field1} %{DATA:field2} ??????? %{DATA:field4} How can I grok this variable leng… --- ## [What is the grok pattern for this jenkins log?](https://discuss.elastic.co/t/what-is-the-grok-pattern-for-this-jenkins-log/216357)
**Author:** [@souravatta](https://discuss.elastic.co/u/souravatta)\ **Replies:** 1\ **Last updated:** [February 5, 2020, 2:17am UTC](https://discuss.elastic.co/t/what-is-the-grok-pattern-for-this-jenkins-log/216357 "2020-02-05T02:17:15Z")
Hi, Can you please help me with the grok pattern for this jenkins sample data or log. The log is only a single line. hudson.slaves.CommandLauncher launch\\nSEVERE: Unable to launch the agent for dot-dewsttlas403-ci\\njav… --- ## [Apply elastic plugin on array items](https://discuss.elastic.co/t/apply-elastic-plugin-on-array-items/216445)
**Author:** [@parosio](https://discuss.elastic.co/u/parosio)\ **Replies:** 1\ **Last updated:** [February 5, 2020, 2:15am UTC](https://discuss.elastic.co/t/apply-elastic-plugin-on-array-items/216445 "2020-02-05T02:15:18Z")
Hello, I need to apply a search/translate for a field which optionally contains multiple values: event\_id;ext\_ref; ... other fields "#123";"ref3"; . . . "#244";"ref2|ref7"; . . . the second field should be s… --- ## [How can I remove logstash process?](https://discuss.elastic.co/t/how-can-i-remove-logstash-process/216474)
**Author:** [@lilyyy](https://discuss.elastic.co/u/lilyyy)\ **Replies:** 1\ **Last updated:** [February 5, 2020, 2:14am UTC](https://discuss.elastic.co/t/how-can-i-remove-logstash-process/216474 "2020-02-05T02:14:24Z")
I killed logstash process, but this process still remain. so I tried to kill the process with 'kill -9 \[PID\]', but I got error message like bash: kill: (36888) - No such process even if it is existing and then chaning … --- ## [While Filebeat talking to Logstash, logstash throws this error](https://discuss.elastic.co/t/while-filebeat-talking-to-logstash-logstash-throws-this-error/216495)
**Author:** [@praveenmak](https://discuss.elastic.co/u/praveenmak)\ **Replies:** 1\ **Last updated:** [February 5, 2020, 2:11am UTC](https://discuss.elastic.co/t/while-filebeat-talking-to-logstash-logstash-throws-this-error/216495 "2020-02-05T02:11:45Z")
I see this error in Logstash logs (earlier I started with DEBUG mode but seems its all the time). The data is not getting ingested at all. Any idea how to resolve this ? please help \[2020-01-24T17:30:22,282\]\[INFO \]\[or… --- ## [Updating the dictionary file used in translate filter while logs are being processed](https://discuss.elastic.co/t/updating-the-dictionary-file-used-in-translate-filter-while-logs-are-being-processed/214494)
**Author:** [@dmalhan](https://discuss.elastic.co/u/dmalhan)\ **Replies:** 8\ **Last updated:** [February 4, 2020, 10:00pm UTC](https://discuss.elastic.co/t/updating-the-dictionary-file-used-in-translate-filter-while-logs-are-being-processed/214494 "2020-02-04T22:00:53Z")
Currently the Logstash instance that we're running basically runs 24 hours a day (processes millions of logs everyday). It's using a file as a mapping dictionary through a Translate filter with the default refresh interv… --- ## [Problem indexing the year](https://discuss.elastic.co/t/problem-indexing-the-year/217870)
**Author:** [@Ay00b\_B](https://discuss.elastic.co/u/Ay00b_B)\ **Replies:** 1\ **Last updated:** [February 4, 2020, 8:55pm UTC](https://discuss.elastic.co/t/problem-indexing-the-year/217870 "2020-02-04T20:55:31Z")
Hello, I am new to this whole ELK thing. Im an intern working on indexing a bunch of syslogs into elastic using logstash. The only problem is that the logs do not contain the year which I also want to index for each mapp… --- ## [Mapping Timeout](https://discuss.elastic.co/t/mapping-timeout/217787)
**Author:** [@NewToThis](https://discuss.elastic.co/u/NewToThis)\ **Replies:** 2\ **Last updated:** [February 4, 2020, 5:04pm UTC](https://discuss.elastic.co/t/mapping-timeout/217787 "2020-02-04T17:04:51Z")
Hey I am using a very simple aggregation to count entries in an SQL Database. If a specific event is being monitored the counter should be lowered by one otherwise it should count up. Everything is running under a uniq… --- ## [Remove double quotes from the field in Logstash message](https://discuss.elastic.co/t/remove-double-quotes-from-the-field-in-logstash-message/217745)
**Author:** [@Kamesh\_M](https://discuss.elastic.co/u/Kamesh_M)\ **Replies:** 1\ **Last updated:** [February 4, 2020, 5:03pm UTC](https://discuss.elastic.co/t/remove-double-quotes-from-the-field-in-logstash-message/217745 "2020-02-04T17:03:46Z")
raw=\>"{ "device" : "TIR1-SF2.abc.om - Gignet1/2/4 · "f1-sf2:\[7\]" test abc " }" How to remove doublequotes for "f1-sf2:\[7\]" for the field "device". Im getting error while parsing this as json :exception=\>#\ **Author:** [@preetish\_P](https://discuss.elastic.co/u/preetish_P)\ **Replies:** 0\ **Last updated:** [February 4, 2020, 1:52pm UTC](https://discuss.elastic.co/t/sort-date-time-fields-within-a-document-in-elastic/217825 "2020-02-04T13:52:37Z") Hi All, I have an index in elasticsearch to track a user's journey through a web portal. Each document records the page URL and timestamp which user visited as part of their journey. E.g: if user visited 10 pages of th… --- ## [Logstash UTF-16 codepoint \[0x0\] at position 0 is a reserved character](https://discuss.elastic.co/t/logstash-utf-16-codepoint-0x0-at-position-0-is-a-reserved-character/217810)
**Author:** [@devopselk](https://discuss.elastic.co/u/devopselk)\ **Replies:** 0\ **Last updated:** [February 4, 2020, 1:24pm UTC](https://discuss.elastic.co/t/logstash-utf-16-codepoint-0x0-at-position-0-is-a-reserved-character/217810 "2020-02-04T13:24:21Z")
Hello everyone, I transfer data from an elasticsearch to another elasticsearch with logstash. I am using codec but could not solve the UTF problem. My Config elasticsearch { codec =\> json { charset =\> "UTF-16" }… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=408) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=410)