# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=733 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 734 --- ## [Mixed columns in logstash output](https://discuss.elastic.co/t/mixed-columns-in-logstash-output/137395)
**Author:** [@slobodan.hyperoptic](https://discuss.elastic.co/u/slobodan.hyperoptic)\ **Replies:** 1\ **Last updated:** [June 27, 2018, 8:20am UTC](https://discuss.elastic.co/t/mixed-columns-in-logstash-output/137395 "2018-06-27T08:20:44Z")
Hi all, I have setup of logstash that is processing some logs (text files), and then puts output to elasticsearch, s3 and file. The problem is that output is not consistent, eg. columns are not always in the same order … --- ## [Index wont create](https://discuss.elastic.co/t/index-wont-create/137436)
**Author:** [@MaikLinnemann](https://discuss.elastic.co/u/MaikLinnemann)\ **Replies:** 5\ **Last updated:** [June 27, 2018, 7:45am UTC](https://discuss.elastic.co/t/index-wont-create/137436 "2018-06-27T07:45:54Z")
Hi all, i know its discussed a lot of times but i cant find my solution. A index is not created from one of my logstash config files and i cant find why. this is the config: input { file { path =\> "/var/log/named/… --- ## [Logstash synchronization with postgresql](https://discuss.elastic.co/t/logstash-synchronization-with-postgresql/137192)
**Author:** [@Iremsu](https://discuss.elastic.co/u/Iremsu)\ **Replies:** 2\ **Last updated:** [June 27, 2018, 7:43am UTC](https://discuss.elastic.co/t/logstash-synchronization-with-postgresql/137192 "2018-06-27T07:43:07Z")
Hi all, I am new on elk stack. I am using postgresql and the thing that i am trying to do it: i have a table called testtable with two columns key and value; whenever i add something in to these columns in my database i… --- ## [ES stack 6.3.0 stopped working =\>Can't get text on a START\_OBJECT](https://discuss.elastic.co/t/es-stack-6-3-0-stopped-working-cant-get-text-on-a-start-object/137424)
**Author:** [@Mojster](https://discuss.elastic.co/u/Mojster)\ **Replies:** 3\ **Last updated:** [June 27, 2018, 7:16am UTC](https://discuss.elastic.co/t/es-stack-6-3-0-stopped-working-cant-get-text-on-a-start-object/137424 "2018-06-27T07:16:30Z")
After upgrading ES stack from 6.2.4 =\> 6.3.0 Logstash isn't producing any data. In LS log I've found: \[2018-06-26T13:44:05,968\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>4… --- ## [How to check if pipeline execution completed in Logstash](https://discuss.elastic.co/t/how-to-check-if-pipeline-execution-completed-in-logstash/136870)
**Author:** [@Somnath\_Pal](https://discuss.elastic.co/u/Somnath_Pal)\ **Replies:** 1\ **Last updated:** [June 27, 2018, 6:16am UTC](https://discuss.elastic.co/t/how-to-check-if-pipeline-execution-completed-in-logstash/136870 "2018-06-27T06:16:33Z")
We are using logstash 6.0 and in log it shows scheduled pipeline running, but how do I know if current run - input - filter - output has completed? Does following input JDBC INFO message with timespent is conclusion for … --- ## [Logstash error after integrating X Pack](https://discuss.elastic.co/t/logstash-error-after-integrating-x-pack/137185)
**Author:** [@vigneshr35](https://discuss.elastic.co/u/vigneshr35)\ **Replies:** 1\ **Last updated:** [June 26, 2018, 11:19pm UTC](https://discuss.elastic.co/t/logstash-error-after-integrating-x-pack/137185 "2018-06-26T23:19:09Z")
Hi All, Logstash has stopped working after integrating X Pack. Logstash version used - 6.2.4 X Pack version - 6.2.4 Is there any issue with the integration on these versions? I see the following error message when I… --- ## [Indexing Problem in Kibana](https://discuss.elastic.co/t/indexing-problem-in-kibana/137493)
**Author:** [@rahulsaini](https://discuss.elastic.co/u/rahulsaini)\ **Replies:** 6\ **Last updated:** [June 26, 2018, 8:17pm UTC](https://discuss.elastic.co/t/indexing-problem-in-kibana/137493 "2018-06-26T20:17:58Z")
I'm getting data from cloud watch but my date field have values in different formats how can i get those values without getting the field type error ? --- ## [Logstash resending data after log file finishes](https://discuss.elastic.co/t/logstash-resending-data-after-log-file-finishes/137502)
**Author:** [@magdieldominguez](https://discuss.elastic.co/u/magdieldominguez)\ **Replies:** 0\ **Last updated:** [June 26, 2018, 7:22pm UTC](https://discuss.elastic.co/t/logstash-resending-data-after-log-file-finishes/137502 "2018-06-26T19:22:31Z")
Hi I am monitoring a set of log files with Logstash and sending them to Elasticsearch. But it seems like after the logs are done logstash resends the logs from the files again, leaving me with duplicate data in elastics… --- ## [Elasticsearch query sort order index](https://discuss.elastic.co/t/elasticsearch-query-sort-order-index/137419)
**Author:** [@espogian](https://discuss.elastic.co/u/espogian)\ **Replies:** 3\ **Last updated:** [June 26, 2018, 7:33pm UTC](https://discuss.elastic.co/t/elasticsearch-query-sort-order-index/137419 "2018-06-26T19:33:34Z")
Hi, if - in a Elasticsearch filter plugin - I do the following: elasticsearch { hosts =\> \["elasticsearch:9200"\] index =\> \["logstash-\*"\] ... And let's assume I have several Logstash indexes like e.g. logstash-2018.0… --- ## [Grok Filter Pattern Clarification](https://discuss.elastic.co/t/grok-filter-pattern-clarification/137478)
**Author:** [@JackTiber](https://discuss.elastic.co/u/JackTiber)\ **Replies:** 1\ **Last updated:** [June 26, 2018, 5:29pm UTC](https://discuss.elastic.co/t/grok-filter-pattern-clarification/137478 "2018-06-26T17:29:28Z")
Hey all, I am setting up centralized logging for a network with 30 - 40 nodes running a bunch of different services. I have divided things up into namespaces to make logging easier and better for visualizing down the ro… --- ## [Logstash action "Update"](https://discuss.elastic.co/t/logstash-action-update/137406)
**Author:** [@safhw91](https://discuss.elastic.co/u/safhw91)\ **Replies:** 1\ **Last updated:** [June 26, 2018, 5:20pm UTC](https://discuss.elastic.co/t/logstash-action-update/137406 "2018-06-26T17:20:04Z")
I have several huge files (hundreds of GBs) that I want to "double parse" with logstash. I have one file which maps user IDs to user emails, and then another huge file which maps user IDs to their actions. So in my fir… --- ## [Wsdl in logstash input file](https://discuss.elastic.co/t/wsdl-in-logstash-input-file/137248)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 9\ **Last updated:** [June 26, 2018, 5:01pm UTC](https://discuss.elastic.co/t/wsdl-in-logstash-input-file/137248 "2018-06-26T17:01:10Z")
Hey Guys, I have been trying to fetch data from BMC Remedy using logstash http poller method, as i was not able to get the restapi's but managed to get a wsdl link, the link was working properly when i test using soap U… --- ## [Replaying Dead Letter messages via RabbitMQ](https://discuss.elastic.co/t/replaying-dead-letter-messages-via-rabbitmq/137475)
**Author:** [@davidbugeja](https://discuss.elastic.co/u/davidbugeja)\ **Replies:** 0\ **Last updated:** [June 26, 2018, 4:54pm UTC](https://discuss.elastic.co/t/replaying-dead-letter-messages-via-rabbitmq/137475 "2018-06-26T16:54:21Z")
Logstash and Elasticion version: 5.6.9 Setting up multiple pipelines so that any messages that end up in Logstash's Dead Letter get sent back to RabbitMQ so that we can re-process them after sorting out any potential ma… --- ## [Logstash S3 Plugin Error01](https://discuss.elastic.co/t/logstash-s3-plugin-error01/137437)
**Author:** [@Abhinaya\_S](https://discuss.elastic.co/u/Abhinaya_S)\ **Replies:** 3\ **Last updated:** [June 26, 2018, 4:46pm UTC](https://discuss.elastic.co/t/logstash-s3-plugin-error01/137437 "2018-06-26T16:46:18Z")
"A plugin had an unrecoverable error. Will restart this plugin.\\n Plugin: \"xxxxxxxxxxxxxxxx", bucket=\>"s3.amazonaws.com/mulelogs56", region=\>"us-east-1", secret\_access\_key=\>"xxxxxxxx… --- ## [Need to create multiple logstash into the same server to monitor apache and nginx together](https://discuss.elastic.co/t/need-to-create-multiple-logstash-into-the-same-server-to-monitor-apache-and-nginx-together/137290)
**Author:** [@m\_5amy](https://discuss.elastic.co/u/m_5amy)\ **Replies:** 9\ **Last updated:** [June 26, 2018, 4:23pm UTC](https://discuss.elastic.co/t/need-to-create-multiple-logstash-into-the-same-server-to-monitor-apache-and-nginx-together/137290 "2018-06-26T16:23:33Z")
i'm trying to use ELK "Elasticsearch , logstash and kibana" to monitor apache and nginx logs and centralize logs into kibana. now I have got logs of apache into logstash and kibana so, what should i do to get ngnix lo… --- ## [Windows Logs to Logstash](https://discuss.elastic.co/t/windows-logs-to-logstash/137005)
**Author:** [@gavin.mcmenemy](https://discuss.elastic.co/u/gavin.mcmenemy)\ **Replies:** 8\ **Last updated:** [June 26, 2018, 3:36pm UTC](https://discuss.elastic.co/t/windows-logs-to-logstash/137005 "2018-06-26T15:36:15Z")
Hi All, First of all I'm very new to ELK etc. I have 2 servers. One configured with the ELK stack: Elasticsearch Logstash Kibana Another server which is handling WEF (windows event forwarding). All servers in this… --- ## [org.logstash.LockException](https://discuss.elastic.co/t/org-logstash-lockexception/137464)
**Author:** [@sivakr](https://discuss.elastic.co/u/sivakr)\ **Replies:** 0\ **Last updated:** [June 26, 2018, 3:29pm UTC](https://discuss.elastic.co/t/org-logstash-lockexception/137464 "2018-06-26T15:29:07Z")
Im using logstash with 6 replication in kubernetes. Here is my config file Pipeline Filter filter { if \[kubernetes\]\[container\]\[name\] == "envoy" { grok { match =\> { "message" =\> '\\\[%{DATA:envoy\_log\_ty… --- ## [Keep DB extracts up to date](https://discuss.elastic.co/t/keep-db-extracts-up-to-date/137433)
**Author:** [@hgpit](https://discuss.elastic.co/u/hgpit)\ **Replies:** 1\ **Last updated:** [June 26, 2018, 3:02pm UTC](https://discuss.elastic.co/t/keep-db-extracts-up-to-date/137433 "2018-06-26T15:02:12Z")
Hi, Totally new to Logstash (and ELK in general). We've set everything up following the many online guides to extract product information from our DB and be able to query it in ES. But if the info changes in our DB, we … --- ## [Ruby code for array iteration](https://discuss.elastic.co/t/ruby-code-for-array-iteration/137450)
**Author:** [@Eswar\_Kumar\_Musiboin](https://discuss.elastic.co/u/Eswar_Kumar_Musiboin)\ **Replies:** 6\ **Last updated:** [June 26, 2018, 2:57pm UTC](https://discuss.elastic.co/t/ruby-code-for-array-iteration/137450 "2018-06-26T14:57:13Z")
ruby { code =\> " event.get('db\_user\_details').each\_index do |i| if \[db\_user\_details\]\[i\]\[country\] == "india" event.set('flagger', 'done') end … --- ## [Problem with logstash](https://discuss.elastic.co/t/problem-with-logstash/137445)
**Author:** [@joce455](https://discuss.elastic.co/u/joce455)\ **Replies:** 1\ **Last updated:** [June 26, 2018, 2:21pm UTC](https://discuss.elastic.co/t/problem-with-logstash/137445 "2018-06-26T14:21:15Z")
Hi all, I have a problem to start logstash i think that the cause is my logstash.conf, input { udp { port =\> 5000 codec =\> json } } filter { if \[docker\]\[image\] =~ /^rancher/ or \[docker\]\[image\] =~ /^bekt/ { drop… --- ## [Document not created](https://discuss.elastic.co/t/document-not-created/137266)
**Author:** [@waitangi](https://discuss.elastic.co/u/waitangi)\ **Replies:** 4\ **Last updated:** [June 26, 2018, 1:47pm UTC](https://discuss.elastic.co/t/document-not-created/137266 "2018-06-26T13:47:36Z")
I'm newbie in elasticsearch. I'm running logstash with command 'bin\\logstash --verbose -f logstash,conf'. Logstash.conf is: #input { stdin { } } input { file { path =\> "C:\\Users\\pawnow\\logstash\\logstash\_input.log" … --- ## [Parse multiline xml via syslog](https://discuss.elastic.co/t/parse-multiline-xml-via-syslog/136138)
**Author:** [@Eric\_Mueller](https://discuss.elastic.co/u/Eric_Mueller)\ **Replies:** 26\ **Last updated:** [June 26, 2018, 1:40pm UTC](https://discuss.elastic.co/t/parse-multiline-xml-via-syslog/136138 "2018-06-26T13:40:18Z")
Hello, I am trying to parse the following log below. I assume I will need combination of grok to grab the timestamp and then using multiline codec with xml parser. I shrunk the output but I can be about a few hundred IP… --- ## [Ruby filter - How to access some elastic index data from a ruby filter](https://discuss.elastic.co/t/ruby-filter-how-to-access-some-elastic-index-data-from-a-ruby-filter/136701)
**Author:** [@akapit](https://discuss.elastic.co/u/akapit)\ **Replies:** 22\ **Last updated:** [June 26, 2018, 12:56pm UTC](https://discuss.elastic.co/t/ruby-filter-how-to-access-some-elastic-index-data-from-a-ruby-filter/136701 "2018-06-26T12:56:40Z")
Hi, I'm trying to let logstash add a field based in the calculation of some incoming event and data that I have in another index or external parameter. How can I access to data from any elasticsearch index in the ruby … --- ## [Tabulation in field causes error](https://discuss.elastic.co/t/tabulation-in-field-causes-error/137274)
**Author:** [@Khaled\_Saidi](https://discuss.elastic.co/u/Khaled_Saidi)\ **Replies:** 1\ **Last updated:** [June 26, 2018, 12:53pm UTC](https://discuss.elastic.co/t/tabulation-in-field-causes-error/137274 "2018-06-26T12:53:43Z")
Hi everybovy, I'm trying to parse a log file and i'm facing a case that causes an error message when i test my filter. in the last field, it sometimes begin with a tabulation (not several blank spaces). Example : 201… --- ## [Logstash cluster where one node has very high CPU usage](https://discuss.elastic.co/t/logstash-cluster-where-one-node-has-very-high-cpu-usage/137429)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 0\ **Last updated:** [June 26, 2018, 12:28pm UTC](https://discuss.elastic.co/t/logstash-cluster-where-one-node-has-very-high-cpu-usage/137429 "2018-06-26T12:28:32Z")
Hi I have two logstash nodes in my cluster which is running the same configuration. I can see and monitor both nodes via Kibana but i'm having an issue where logstash1 is spiking in CPU usage. If i have both nodes runni… --- ## [Error, cannot retrieve cgroups information](https://discuss.elastic.co/t/error-cannot-retrieve-cgroups-information/137428)
**Author:** [@sp1thas](https://discuss.elastic.co/u/sp1thas)\ **Replies:** 0\ **Last updated:** [June 26, 2018, 12:03pm UTC](https://discuss.elastic.co/t/error-cannot-retrieve-cgroups-information/137428 "2018-06-26T12:03:08Z")
Hi, I'm trying to run logstash using specific config file: $ /usr/share/logstash/bin/logstash --debug -f $HOME/conf.d/conf2.conf --path.settings /etc/logstash and I'm getting this error: \[2018-06-26T15:00:41,046\]\[DEB… --- ## [Can't run Logstash ? Error: Could not find or load main class org.logstash.Logstash](https://discuss.elastic.co/t/cant-run-logstash-error-could-not-find-or-load-main-class-org-logstash-logstash/136213)
**Author:** [@Divit\_Sharma](https://discuss.elastic.co/u/Divit_Sharma)\ **Replies:** 4\ **Last updated:** [June 26, 2018, 11:58am UTC](https://discuss.elastic.co/t/cant-run-logstash-error-could-not-find-or-load-main-class-org-logstash-logstash/136213 "2018-06-26T11:58:47Z")
Error: Could not find or load main class org.logstash.Logstash Command : bin/logstash -f /home/hduser/Desktop/data/logstash\_cars.config I am running it in Ubuntu using vmware hduser@ubuntu:~$ java -version java versio… --- ## [Jdbc\_static filter parsing issue](https://discuss.elastic.co/t/jdbc-static-filter-parsing-issue/137422)
**Author:** [@Eswar\_Kumar\_Musiboin](https://discuss.elastic.co/u/Eswar_Kumar_Musiboin)\ **Replies:** 0\ **Last updated:** [June 26, 2018, 11:34am UTC](https://discuss.elastic.co/t/jdbc-static-filter-parsing-issue/137422 "2018-06-26T11:34:35Z")
my configuration file is input { generator { lines =\> \[ '{"from\_ip": "10.2.3.20", "app": "foobar", "amount": 32.95}', '{"from\_ip": "10.2.3.30", "app": "barfoo", "amount": 82.95}', '{"from\_ip": "10.2.3.40", "app"… --- ## [Grok pattern for custom application log](https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864)
**Author:** [@pkshara](https://discuss.elastic.co/u/pkshara)\ **Replies:** 9\ **Last updated:** [June 26, 2018, 11:25am UTC](https://discuss.elastic.co/t/grok-pattern-for-custom-application-log/133864 "2018-06-26T11:25:09Z")
Hi Team, I have error log as mentioned below , can you suggest me the grok pattern for this, 20180209 00:00:08,696 ERROR WebContainer : 34989 dao.OrderServiceDAOImpl Acct\_Nr=724377 Imp\_Acct\_Nr= Exception in the metho… --- ## [Parse XML log lines](https://discuss.elastic.co/t/parse-xml-log-lines/137276)
**Author:** [@Boukhdhira](https://discuss.elastic.co/u/Boukhdhira)\ **Replies:** 6\ **Last updated:** [June 26, 2018, 11:23am UTC](https://discuss.elastic.co/t/parse-xml-log-lines/137276 "2018-06-26T11:23:44Z")
I would like some recommendation on how to parse a given xml document splited into log lines with logstash. My document looks like this: |2018-06-19T07:29:09+02:00 127.0.0.1 ping - |\