# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=515 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 516 --- ## [Logstash elasticsearch input plugin check point](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-check-point/189237)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 0\ **Last updated:** [July 7, 2019, 9:51am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-check-point/189237 "2019-07-07T09:51:58Z")
hi all, I want to use elasticsearch input plugin in logstash in order to make a query on elasticsearch and extract desire data; then do some filtering and finally index them. Is it possible to define a check point for t… --- ## [Data.path problem](https://discuss.elastic.co/t/data-path-problem/188902)
**Author:** [@Rafa\_M](https://discuss.elastic.co/u/Rafa_M)\ **Replies:** 1\ **Last updated:** [July 6, 2019, 9:49pm UTC](https://discuss.elastic.co/t/data-path-problem/188902 "2019-07-06T21:49:41Z")
Hello, I installed logstash in a different path of default path. When I'm trying to launch more than 1 pipeline I'm getting the following error: Logstash could not be started because there is already another instance u… --- ## [Can you denormalize kafka topics in logstash before indexing to ES](https://discuss.elastic.co/t/can-you-denormalize-kafka-topics-in-logstash-before-indexing-to-es/186856)
**Author:** [@Aymen\_Ben\_Moussa](https://discuss.elastic.co/u/Aymen_Ben_Moussa)\ **Replies:** 12\ **Last updated:** [July 6, 2019, 8:03pm UTC](https://discuss.elastic.co/t/can-you-denormalize-kafka-topics-in-logstash-before-indexing-to-es/186856 "2019-07-06T20:03:54Z")
My question is if logstash is capable of denormalizing kafka topics --- ## [Grok Pattern works in Grok debugger but fails in the parsing](https://discuss.elastic.co/t/grok-pattern-works-in-grok-debugger-but-fails-in-the-parsing/189115)
**Author:** [@hyder](https://discuss.elastic.co/u/hyder)\ **Replies:** 2\ **Last updated:** [July 6, 2019, 6:55pm UTC](https://discuss.elastic.co/t/grok-pattern-works-in-grok-debugger-but-fails-in-the-parsing/189115 "2019-07-06T18:55:05Z")
The Grok Patter works in Grok debugger but fails to parse in logstash. I need help not sure whats wrong, I need to sort out ERROR logs separate Index and regular Debug into different index, My trying to get parsing done,… --- ## [Move file after its been indexed](https://discuss.elastic.co/t/move-file-after-its-been-indexed/189210)
**Author:** [@OpSecMonkey](https://discuss.elastic.co/u/OpSecMonkey)\ **Replies:** 1\ **Last updated:** [July 6, 2019, 3:58pm UTC](https://discuss.elastic.co/t/move-file-after-its-been-indexed/189210 "2019-07-06T15:58:17Z")
I was wondering in the conf if there is something I can add that auto moves a file to a different folder after its been read. --- ## [Multiple sql table to one elastic index](https://discuss.elastic.co/t/multiple-sql-table-to-one-elastic-index/189188)
**Author:** [@manish\_kaushik](https://discuss.elastic.co/u/manish_kaushik)\ **Replies:** 3\ **Last updated:** [July 6, 2019, 1:58pm UTC](https://discuss.elastic.co/t/multiple-sql-table-to-one-elastic-index/189188 "2019-07-06T13:58:54Z")
I have been using logstash to index data in elastic from mysql via jdbc plugin. I was using join query to pull the data but my usecase increased to check for insert and updates. So, i used tracking columns and :sql\_last\_… --- ## [Filter section is showing error](https://discuss.elastic.co/t/filter-section-is-showing-error/189119)
**Author:** [@pathri](https://discuss.elastic.co/u/pathri)\ **Replies:** 2\ **Last updated:** [July 6, 2019, 5:37am UTC](https://discuss.elastic.co/t/filter-section-is-showing-error/189119 "2019-07-06T05:37:02Z")
I have a csv with 184 columns. when i trying to index the data using logstash, it's config file is looking like the attached screenshot everything was working fine but as i crossed the character number 3000, logstash… --- ## [How to enable ILM when I i have custom index name with hostname via logstash](https://discuss.elastic.co/t/how-to-enable-ilm-when-i-i-have-custom-index-name-with-hostname-via-logstash/189171)
**Author:** [@Thanura\_Kannangara](https://discuss.elastic.co/u/Thanura_Kannangara)\ **Replies:** 0\ **Last updated:** [July 5, 2019, 9:38pm UTC](https://discuss.elastic.co/t/how-to-enable-ilm-when-i-i-have-custom-index-name-with-hostname-via-logstash/189171 "2019-07-05T21:38:11Z")
I'm sending data from filebeat to elasticsearch via Logstash I managed to send data to ES with my custom index name even with this. index =\> "%{\[@metadata\]\[beat\]}-%{\[host\]\[hostname\]}-%{\[@metadata\]\[version\]}-% But I'm … --- ## [Need help with Grok](https://discuss.elastic.co/t/need-help-with-grok/189146)
**Author:** [@EvanG](https://discuss.elastic.co/u/EvanG)\ **Replies:** 4\ **Last updated:** [July 5, 2019, 6:39pm UTC](https://discuss.elastic.co/t/need-help-with-grok/189146 "2019-07-05T18:39:14Z")
Hey Guys, New to Logstash, and I tried to read the documentation as much as possible but still confused.. Can anyone help me grok 'sessionId' and 'user' out of this message? Trying to get the values into new fields. I… --- ## [Logstash elasticsearch filter, query with nested field](https://discuss.elastic.co/t/logstash-elasticsearch-filter-query-with-nested-field/189093)
**Author:** [@gyterpena](https://discuss.elastic.co/u/gyterpena)\ **Replies:** 1\ **Last updated:** [July 5, 2019, 3:24pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-filter-query-with-nested-field/189093 "2019-07-05T15:24:29Z")
Hello I have issue running query with nested field. Sample input data: {"url\_details":{"\_id":"1c928bb630eaf248"},"usernameID":"test1"} Filter config: if \[url\_details\]\[\_id\] { elasticsearch { … --- ## [JDBC Logstash - Json datatype is unknown](https://discuss.elastic.co/t/jdbc-logstash-json-datatype-is-unknown/189019)
**Author:** [@selimseker](https://discuss.elastic.co/u/selimseker)\ **Replies:** 2\ **Last updated:** [July 5, 2019, 3:14pm UTC](https://discuss.elastic.co/t/jdbc-logstash-json-datatype-is-unknown/189019 "2019-07-05T15:14:06Z")
Hello everyone, I'm collecting the data from Postgresql table via Logstash. Postgre table contains text typed Json field. In the config file I filtered the json and I can see in the Kibana json keys separately but data … --- ## [Jdbc\_static: Getting a sql syntax error I did'nt write](https://discuss.elastic.co/t/jdbc-static-getting-a-sql-syntax-error-i-didnt-write/187844)
**Author:** [@ErikWarming](https://discuss.elastic.co/u/ErikWarming)\ **Replies:** 6\ **Last updated:** [July 5, 2019, 2:53pm UTC](https://discuss.elastic.co/t/jdbc-static-getting-a-sql-syntax-error-i-didnt-write/187844 "2019-07-05T14:53:38Z")
With logstash I am trying to Extract some tables, Transform them locally on the logstash mashine, and then Load the result to Elasticsearch. The reason for this solution is due to very limited computing power on the sour… --- ## [Using a boolean setting in Java plugin crashes logstash](https://discuss.elastic.co/t/using-a-boolean-setting-in-java-plugin-crashes-logstash/189120)
**Author:** [@SavioAlp](https://discuss.elastic.co/u/SavioAlp)\ **Replies:** 1\ **Last updated:** [July 5, 2019, 2:52pm UTC](https://discuss.elastic.co/t/using-a-boolean-setting-in-java-plugin-crashes-logstash/189120 "2019-07-05T14:52:01Z")
I am building my own java filter plugin for logstash and trying to make use of the booleanSetting available in PluginConfigSpec, but when I attempt to use it in a logstash instance it throws a IllegalStateException I've … --- ## [Logstash New Field is not added](https://discuss.elastic.co/t/logstash-new-field-is-not-added/188912)
**Author:** [@Syed.Ubaid](https://discuss.elastic.co/u/Syed.Ubaid)\ **Replies:** 3\ **Last updated:** [July 5, 2019, 2:43pm UTC](https://discuss.elastic.co/t/logstash-new-field-is-not-added/188912 "2019-07-05T14:43:41Z")
I am trying to add new field in my config file there is already a field message in my csv file i am trying to add test field . In this add field i want to copy of message data into test field for testing file my config … --- ## [Combine several fields using ruby/merge/aggregate](https://discuss.elastic.co/t/combine-several-fields-using-ruby-merge-aggregate/188747)
**Author:** [@akhilsharma.in](https://discuss.elastic.co/u/akhilsharma.in)\ **Replies:** 3\ **Last updated:** [July 5, 2019, 2:39pm UTC](https://discuss.elastic.co/t/combine-several-fields-using-ruby-merge-aggregate/188747 "2019-07-05T14:39:18Z")
Below is the sample log which I get as an output: { "field1": { --content-- {}, , } "field2": { --content-- {}, , } "field3": { --content-- {}, , } } I want to combine all the fields so that I see outpu… --- ## [WORD is not working in grok filter for Apache logs in Logstash](https://discuss.elastic.co/t/word-is-not-working-in-grok-filter-for-apache-logs-in-logstash/188914)
**Author:** [@Dibyakanta](https://discuss.elastic.co/u/Dibyakanta)\ **Replies:** 5\ **Last updated:** [July 5, 2019, 12:23pm UTC](https://discuss.elastic.co/t/word-is-not-working-in-grok-filter-for-apache-logs-in-logstash/188914 "2019-07-05T12:23:35Z")
Hi, I am using the below grok filter to parse the Apache logs pattern. The grok filter is : (%{IPORHOST:clientip})?|%{QUOTEDSTRING:user}|(%{WORD:cust})?| The first field clientip is optional so have added ? in it. Th… --- ## [Index Json file data into Elasticsearch](https://discuss.elastic.co/t/index-json-file-data-into-elasticsearch/189067)
**Author:** [@umeshmishra099](https://discuss.elastic.co/u/umeshmishra099)\ **Replies:** 3\ **Last updated:** [July 5, 2019, 10:41am UTC](https://discuss.elastic.co/t/index-json-file-data-into-elasticsearch/189067 "2019-07-05T10:41:36Z")
I have a josn file with below data: \[ { "City": "0miya-ku", "Company": "3COM", "Country": "ARUBA", "CountryCodeIso2": "AF", "CountryCodeIso3": "AFG", "LastName": "AABERG", "Gender": "Male" }, { "City": "0saki… --- ## [Issue with message field while parsing JSON](https://discuss.elastic.co/t/issue-with-message-field-while-parsing-json/188894)
**Author:** [@ankamraok](https://discuss.elastic.co/u/ankamraok)\ **Replies:** 2\ **Last updated:** [July 5, 2019, 9:57am UTC](https://discuss.elastic.co/t/issue-with-message-field-while-parsing-json/188894 "2019-07-05T09:57:25Z")
Hi, I am parsing jboss fuse container logs ,which are JSON messags,following a sample JSON message. {"@timestamp":"2019-06-25T08:34:36.517Z","source\_host":"sitfabr02.local","level":"INFO","thread\_name":"qtp1821745282-2… --- ## [Filebeat to logstash error](https://discuss.elastic.co/t/filebeat-to-logstash-error/188525)
**Author:** [@Jitendra\_Kumhar](https://discuss.elastic.co/u/Jitendra_Kumhar)\ **Replies:** 1\ **Last updated:** [July 5, 2019, 9:27am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-error/188525 "2019-07-05T09:27:25Z")
Hello, I am sending logs from filebeat to logstash to elasticsearch I am getting these error on logstash after running for some time \[2019-07-02T12:45:46,076\]\[WARN \]\[io.netty.channel.DefaultChannelPipeline\] An excepti… --- ## [Mutate join filter](https://discuss.elastic.co/t/mutate-join-filter/186884)
**Author:** [@Andreasky](https://discuss.elastic.co/u/Andreasky)\ **Replies:** 17\ **Last updated:** [July 5, 2019, 9:22am UTC](https://discuss.elastic.co/t/mutate-join-filter/186884 "2019-07-05T09:22:06Z")
Is it possible to use the mutate join filter to make it possible to make queries across indexes in Kibana? Like joining tables in SQL? --- ## [Increase shards and replicas in new Index](https://discuss.elastic.co/t/increase-shards-and-replicas-in-new-index/188900)
**Author:** [@mrashid](https://discuss.elastic.co/u/mrashid)\ **Replies:** 4\ **Last updated:** [July 5, 2019, 8:17am UTC](https://discuss.elastic.co/t/increase-shards-and-replicas-in-new-index/188900 "2019-07-05T08:17:35Z")
In my logstash configuration I am creating Index daily with date using the below configuration : elasticsearch { hosts =\> \["elkhost:9200"\] manage\_template =\> false index =\> "%{\[@metadata\]\[beat\]}-%{+YYYY.MM.dd}-failurel… --- ## [Duration Time Calculation: missing started time](https://discuss.elastic.co/t/duration-time-calculation-missing-started-time/188999)
**Author:** [@yanly011](https://discuss.elastic.co/u/yanly011)\ **Replies:** 1\ **Last updated:** [July 5, 2019, 5:40am UTC](https://discuss.elastic.co/t/duration-time-calculation-missing-started-time/188999 "2019-07-05T05:40:08Z")
Hi, I just created a pipeline and transfer .csv log file to index. I want to find the started time and calculate the duration time. I used the code from official document: filter { csv { separator =\> "," columns =\> \[… --- ## [Command to run multiple instances of logstash in windows](https://discuss.elastic.co/t/command-to-run-multiple-instances-of-logstash-in-windows/188845)
**Author:** [@Uchiha\_Kakashi](https://discuss.elastic.co/u/Uchiha_Kakashi)\ **Replies:** 2\ **Last updated:** [July 5, 2019, 5:33am UTC](https://discuss.elastic.co/t/command-to-run-multiple-instances-of-logstash-in-windows/188845 "2019-07-05T05:33:34Z")
I want to run multiple instances of logstash on my windows system. In elasticsearch to run multiple nodes at once I used below command. bin/elasticsearch -Epath.data=data1 -Epath.logs=log1 -Ecluster.name=elasticsearch -… --- ## [.CSV input with JSON in it :(](https://discuss.elastic.co/t/csv-input-with-json-in-it/188952)
**Author:** [@EvanG](https://discuss.elastic.co/u/EvanG)\ **Replies:** 5\ **Last updated:** [July 4, 2019, 7:48pm UTC](https://discuss.elastic.co/t/csv-input-with-json-in-it/188952 "2019-07-04T19:48:50Z")
Hey Team, New here! And new to ELK stack. Me and my colleague have spent days trying to figure out a solution to our problem, and yet have not come up with a resolution. Background: Our company has tasked us to ingest … --- ## [Correct Grok Pattern for an optional field including spaces](https://discuss.elastic.co/t/correct-grok-pattern-for-an-optional-field-including-spaces/188571)
**Author:** [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Replies:** 5\ **Last updated:** [July 4, 2019, 3:18pm UTC](https://discuss.elastic.co/t/correct-grok-pattern-for-an-optional-field-including-spaces/188571 "2019-07-04T15:18:11Z")
In some of the log messages, an optional field \[opt\_field=xyz\] or \[opt\_field=\] is present. I arrived at the following pattern to handle it: (?m)%{TIMESTAMP\_ISO8601:Src\_CreateDate} %{LOGLEVEL:Src\_Severity} \\\[(?\ **Author:** [@stcdarrell](https://discuss.elastic.co/u/stcdarrell)\ **Replies:** 9\ **Last updated:** [July 4, 2019, 3:17pm UTC](https://discuss.elastic.co/t/simple-mutate-copy-ip-to-string-doesnt-work/188405 "2019-07-04T15:17:35Z") what am i missing here.. suricata logs store source and destination fields in type IP.. which is fine.. but i also need that information in string format for another purpose.. so i just want to copy it to another field.… --- ## [Logstash US-ASCII](https://discuss.elastic.co/t/logstash-us-ascii/188317)
**Author:** [@selimseker](https://discuss.elastic.co/u/selimseker)\ **Replies:** 1\ **Last updated:** [July 4, 2019, 3:00pm UTC](https://discuss.elastic.co/t/logstash-us-ascii/188317 "2019-07-04T15:00:12Z")
Hi everyone I'm a newbie on ELK. I just download the logstash and when I run it it gives this error: \[main\] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (ArgumentError) in… --- ## [How to parse the below scenario?](https://discuss.elastic.co/t/how-to-parse-the-below-scenario/188928)
**Author:** [@Debashis\_Adak](https://discuss.elastic.co/u/Debashis_Adak)\ **Replies:** 1\ **Last updated:** [July 4, 2019, 2:32pm UTC](https://discuss.elastic.co/t/how-to-parse-the-below-scenario/188928 "2019-07-04T14:32:02Z")
Hi All, Please find the below mentioned inputs /abcd/efgh/yj/kl/mn/op abcd/efgh/yj/kl/mn/op abcd /abcd abcd/ Output Required: servicename = abcd I have tried two grok parser (/)?%{DATA:servicename}/ ==\> Making … --- ## [Curl result of a time range query is 8 hours earlier which is not my timezone](https://discuss.elastic.co/t/curl-result-of-a-time-range-query-is-8-hours-earlier-which-is-not-my-timezone/188688)
**Author:** [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Replies:** 4\ **Last updated:** [July 4, 2019, 2:04pm UTC](https://discuss.elastic.co/t/curl-result-of-a-time-range-query-is-8-hours-earlier-which-is-not-my-timezone/188688 "2019-07-04T14:04:48Z")
Hello all, I am trying with the below simple search query in Kibana: POST /esd-dhcplog\*/\_search { "query": { "bool": { "must": \[ { "range": { "rece… --- ## [Unable to load data from PostgreSQL to Elasticsearch via Logstash](https://discuss.elastic.co/t/unable-to-load-data-from-postgresql-to-elasticsearch-via-logstash/188697)
**Author:** [@Himanshu\_Jatale](https://discuss.elastic.co/u/Himanshu_Jatale)\ **Replies:** 3\ **Last updated:** [July 4, 2019, 1:22pm UTC](https://discuss.elastic.co/t/unable-to-load-data-from-postgresql-to-elasticsearch-via-logstash/188697 "2019-07-04T13:22:43Z")
Hello, I'm trying to load data from PostgreSQL Server to Elasticsearch so as to use it in Kibana for visualization purpose. But while connecting it with Logstash, I'm not getting any index. I'm attaching my logstash con… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=514) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=516)