# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=804 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 805 --- ## [Read jdbc password from shell script output](https://discuss.elastic.co/t/read-jdbc-password-from-shell-script-output/123543)
**Author:** [@rons1404](https://discuss.elastic.co/u/rons1404)\ **Replies:** 3\ **Last updated:** [March 12, 2018, 1:37pm UTC](https://discuss.elastic.co/t/read-jdbc-password-from-shell-script-output/123543 "2018-03-12T13:37:31Z")
Hey, Is it possible to set jdbc\_password as output of some script? I want to read the password from a global variable, but it's encrypted so I need to call some shell script to decrypt it. The only option I see is to e… --- ## [Can I use internal IP to connect to kafka? (logstash-input-kafka)](https://discuss.elastic.co/t/can-i-use-internal-ip-to-connect-to-kafka-logstash-input-kafka/123539)
**Author:** [@michaelcheung](https://discuss.elastic.co/u/michaelcheung)\ **Replies:** 1\ **Last updated:** [March 12, 2018, 12:09pm UTC](https://discuss.elastic.co/t/can-i-use-internal-ip-to-connect-to-kafka-logstash-input-kafka/123539 "2018-03-12T12:09:12Z")
I have both my kafka and logstash in same region on GCP, separating machine. When I check the bandwidth usage of kafka's machine (iftop -P), I found that a number of bandwidth is from logstash machine's external ip. But … --- ## [Problem to copy a field from one event to another event](https://discuss.elastic.co/t/problem-to-copy-a-field-from-one-event-to-another-event/123522)
**Author:** [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\ **Replies:** 2\ **Last updated:** [March 12, 2018, 12:07pm UTC](https://discuss.elastic.co/t/problem-to-copy-a-field-from-one-event-to-another-event/123522 "2018-03-12T12:07:15Z")
Two event's of postfix-mail log fields are below.: source: /var/log/mail.log postfix\_queueid:24C4C681F19 host: mail program: postfix/amavis/smtp postfix\_to: jhon@gmail.com time: 10:53:30 postfix\_status: sent post… --- ## [Date parsing (timestamp)](https://discuss.elastic.co/t/date-parsing-timestamp/123526)
**Author:** [@cilzzz](https://discuss.elastic.co/u/cilzzz)\ **Replies:** 2\ **Last updated:** [March 12, 2018, 11:37am UTC](https://discuss.elastic.co/t/date-parsing-timestamp/123526 "2018-03-12T11:37:47Z")
Hi how to parse this date in a log file? 2018-02-22T18:15:34.000+0100 thanks in advance --- ## [Parse log where is json and grok](https://discuss.elastic.co/t/parse-log-where-is-json-and-grok/123490)
**Author:** [@Marek\_Pastier](https://discuss.elastic.co/u/Marek_Pastier)\ **Replies:** 1\ **Last updated:** [March 12, 2018, 10:50am UTC](https://discuss.elastic.co/t/parse-log-where-is-json-and-grok/123490 "2018-03-12T10:50:56Z")
Hello team. I would like ask for help. I have log file with next records. Mar 12 07:04:07 HATAVSM002 ERAServer\[3720\] \{"event\_type":"Threat\_Event","ipv4":"10.1.3.115","hostname":"test.sebb.com","source\_uuid"… --- ## [How to check the result of a query to elasticsearch into the filter plugin](https://discuss.elastic.co/t/how-to-check-the-result-of-a-query-to-elasticsearch-into-the-filter-plugin/123096)
**Author:** [@mirkomirko](https://discuss.elastic.co/u/mirkomirko)\ **Replies:** 3\ **Last updated:** [March 12, 2018, 10:47am UTC](https://discuss.elastic.co/t/how-to-check-the-result-of-a-query-to-elasticsearch-into-the-filter-plugin/123096 "2018-03-12T10:47:28Z")
Hi all, I've a problem when trying to check if a query, done inside the elasticsearch filter plugin, returns me a document. How can i check this situation? Currently I'm using the fields attribute to map old event inside… --- ## [ELK on microsoft AZURE](https://discuss.elastic.co/t/elk-on-microsoft-azure/123249)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 6\ **Last updated:** [March 12, 2018, 9:25am UTC](https://discuss.elastic.co/t/elk-on-microsoft-azure/123249 "2018-03-12T09:25:20Z")
HI team, Im new azure Elasticstack and i have implement ELK in Azure cloud. I have implemented my datanode act as a masternode. I can open my kibana and find the elasticsearch staus by using machine IP Now i want to i… --- ## [MongoDB Logstash Integration \[Solved\]](https://discuss.elastic.co/t/mongodb-logstash-integration-solved/122299)
**Author:** [@panda2004](https://discuss.elastic.co/u/panda2004)\ **Replies:** 1\ **Last updated:** [March 12, 2018, 9:19am UTC](https://discuss.elastic.co/t/mongodb-logstash-integration-solved/122299 "2018-03-12T09:19:37Z")
I recently worked a lot about querying and writing values straight into MongoDB 3.6.2 from logstash. I noticed that many people in the forum tried to do the same thing and didn't succeed. I wanted to publish my solution,… --- ## [Split array with multiple field](https://discuss.elastic.co/t/split-array-with-multiple-field/122702)
**Author:** [@stefan\_f](https://discuss.elastic.co/u/stefan_f)\ **Replies:** 7\ **Last updated:** [March 12, 2018, 9:18am UTC](https://discuss.elastic.co/t/split-array-with-multiple-field/122702 "2018-03-12T09:18:16Z")
Hi there, I have tried by my own, but I'm unable to figure out what I'm doing wrong here. I get a reply from a SQL database as an enrichment of my data. The SQL reply is an array and looks like: { "servicename": "Sky… --- ## [Logstash not sending logs to elastic](https://discuss.elastic.co/t/logstash-not-sending-logs-to-elastic/123239)
**Author:** [@muthu\_kumar1](https://discuss.elastic.co/u/muthu_kumar1)\ **Replies:** 8\ **Last updated:** [March 12, 2018, 9:10am UTC](https://discuss.elastic.co/t/logstash-not-sending-logs-to-elastic/123239 "2018-03-12T09:10:02Z")
\[2018-03-09T09:46:31,839\]\[WARN \]\[logstash.outputs.elasticsearch\] Detected a 6.x and above cluster: the type event field won't be used to determine the document \_type {:es\_version=\>6} \[2018-03-09T09:46:31,840\]\[INFO \]\[log… --- ## [Logstash conf with stdin input not pushing data into elasticsearch](https://discuss.elastic.co/t/logstash-conf-with-stdin-input-not-pushing-data-into-elasticsearch/123485)
**Author:** [@Mohan\_M](https://discuss.elastic.co/u/Mohan_M)\ **Replies:** 6\ **Last updated:** [March 12, 2018, 8:07am UTC](https://discuss.elastic.co/t/logstash-conf-with-stdin-input-not-pushing-data-into-elasticsearch/123485 "2018-03-12T08:07:23Z")
Hello all, I am new to elastic stack I have configured a elastic stack with xpack. when it tried to run a simple logstash conf with stdin input and elasticsearch output. it is not getting inserted into elasticsearch but… --- ## [Need to create filter for logstash](https://discuss.elastic.co/t/need-to-create-filter-for-logstash/123042)
**Author:** [@mlala](https://discuss.elastic.co/u/mlala)\ **Replies:** 5\ **Last updated:** [March 12, 2018, 8:06am UTC](https://discuss.elastic.co/t/need-to-create-filter-for-logstash/123042 "2018-03-12T08:06:26Z")
Hi, I am trying to prepare filter for F5 ASM but using GROK Debugger i am facing issue related to same can anyone help with with proper logstash filter. i will paste the log file for your convenience. \<134\>Mar 8 12:40:… --- ## [Use redis to create session and add the session id to each log event in logstash](https://discuss.elastic.co/t/use-redis-to-create-session-and-add-the-session-id-to-each-log-event-in-logstash/123488)
**Author:** [@Arvind\_Kumar1988](https://discuss.elastic.co/u/Arvind_Kumar1988)\ **Replies:** 0\ **Last updated:** [March 12, 2018, 6:55am UTC](https://discuss.elastic.co/t/use-redis-to-create-session-and-add-the-session-id-to-each-log-event-in-logstash/123488 "2018-03-12T06:55:14Z")
Filebeat sends log events to our logstash server. Each log event has a client id which is basically a machine id of the client who generate the event. Using this client id I want to generate a session id and save this in… --- ## [XML Input Issues](https://discuss.elastic.co/t/xml-input-issues/118751)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 27\ **Last updated:** [March 12, 2018, 4:17am UTC](https://discuss.elastic.co/t/xml-input-issues/118751 "2018-03-12T04:17:45Z")
I posted a topic earlier about this but it got bogged down with craziness, hoping that is the reason there was limited assistance from the community. I am trying to ingest DMARC aggregate XML reports. Here is my pipeli… --- ## [How to Logstash gork filter with different infos in syslog message field](https://discuss.elastic.co/t/how-to-logstash-gork-filter-with-different-infos-in-syslog-message-field/123078)
**Author:** [@jiona](https://discuss.elastic.co/u/jiona)\ **Replies:** 5\ **Last updated:** [March 11, 2018, 10:53pm UTC](https://discuss.elastic.co/t/how-to-logstash-gork-filter-with-different-infos-in-syslog-message-field/123078 "2018-03-11T22:53:06Z")
Hi All, I want to parse a firewall syslog stream. Depends on firewall action the syslog message field contains additional information like the outbound interface or the service-name of well known ports or not. I have … --- ## [Logstash-input-jms real slow or losing messages](https://discuss.elastic.co/t/logstash-input-jms-real-slow-or-losing-messages/123460)
**Author:** [@slpkx2](https://discuss.elastic.co/u/slpkx2)\ **Replies:** 0\ **Last updated:** [March 11, 2018, 10:31pm UTC](https://discuss.elastic.co/t/logstash-input-jms-real-slow-or-losing-messages/123460 "2018-03-11T22:31:52Z")
I have set up the plug-in to subscribe to an AMQ topic and send the messages to stdout (just for initial testing). I have separate Java client subscribing to the same topic and I can see that the message rate is about 3 … --- ## [Logstash - \_source sibblings in output are not supported?](https://discuss.elastic.co/t/logstash-source-sibblings-in-output-are-not-supported/123406)
**Author:** [@ehtishamkhan](https://discuss.elastic.co/u/ehtishamkhan)\ **Replies:** 1\ **Last updated:** [March 11, 2018, 9:50pm UTC](https://discuss.elastic.co/t/logstash-source-sibblings-in-output-are-not-supported/123406 "2018-03-11T21:50:52Z")
By default, elements in \_source fields are accessible. How do I reference \_source siblings in logstash output config? e.g. highlight, \_id, \_score, \_index etc. --- ## [How do I add a field in my logs?](https://discuss.elastic.co/t/how-do-i-add-a-field-in-my-logs/123149)
**Author:** [@Murat89](https://discuss.elastic.co/u/Murat89)\ **Replies:** 3\ **Last updated:** [March 11, 2018, 9:27pm UTC](https://discuss.elastic.co/t/how-do-i-add-a-field-in-my-logs/123149 "2018-03-11T21:27:55Z")
Hi, im new in elastic stack and want to try out something. I have Pexip Logs below, my issue is that the duration is not recognized as a Field (in Kibana), I would like to have the duration value so i can calculate the … --- ## [Accepting messages in CEF format](https://discuss.elastic.co/t/accepting-messages-in-cef-format/123436)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 1\ **Last updated:** [March 11, 2018, 9:18pm UTC](https://discuss.elastic.co/t/accepting-messages-in-cef-format/123436 "2018-03-11T21:18:51Z")
Hi there, I have devices which are logging and sending events to logstash in CEF format. Can someone confirm how do I accept messages with those? Does logstash support CEF input plugin? Or is there any other alternativ… --- ## [Why \_source host = 'localhost' , that's not my machine's host name](https://discuss.elastic.co/t/why-source-host-localhost-thats-not-my-machines-host-name/123182)
**Author:** [@liumianmian](https://discuss.elastic.co/u/liumianmian)\ **Replies:** 1\ **Last updated:** [March 11, 2018, 7:55pm UTC](https://discuss.elastic.co/t/why-source-host-localhost-thats-not-my-machines-host-name/123182 "2018-03-11T19:55:05Z")
my logstash output result : { "path" =\> "/var/log/log\_2018-3-117\_submit.log", "@version" =\> "1", "status" =\> 0, "host" =\> "localhost", "@timestamp" =\> 2018-03-09T03:04:41.672Z, "type" =\> "submit\_log" } how can i… --- ## [Ignore date offset in logstash](https://discuss.elastic.co/t/ignore-date-offset-in-logstash/123419)
**Author:** [@yishain11](https://discuss.elastic.co/u/yishain11)\ **Replies:** 3\ **Last updated:** [March 11, 2018, 7:34pm UTC](https://discuss.elastic.co/t/ignore-date-offset-in-logstash/123419 "2018-03-11T19:34:00Z")
HI, I have the following date format: "2017-12-01 04:39:34-04:000" Currently, I use this configuration: date{ match =\> \["\[documentsList\]\[0\]\[commandTimestamp\]","yyyy-MM-dd HH:mm:ssZZ'0'"\] … --- ## ["Illegal Latitude" for some docs but not others](https://discuss.elastic.co/t/illegal-latitude-for-some-docs-but-not-others/123284)
**Author:** [@GregMeadows](https://discuss.elastic.co/u/GregMeadows)\ **Replies:** 1\ **Last updated:** [March 11, 2018, 4:15am UTC](https://discuss.elastic.co/t/illegal-latitude-for-some-docs-but-not-others/123284 "2018-03-11T04:15:20Z")
I have 14 years of data which loaded great into Logstash 5.4. We began seeing discrepancies in doc counts in both v5.4 and v6.2. We decided to add new columns to the data and reload from scratch using Logstash CSV. Our d… --- ## [Does Ruby version matter when building Gems?](https://discuss.elastic.co/t/does-ruby-version-matter-when-building-gems/123397)
**Author:** [@wakinov](https://discuss.elastic.co/u/wakinov)\ **Replies:** 1\ **Last updated:** [March 11, 2018, 3:58am UTC](https://discuss.elastic.co/t/does-ruby-version-matter-when-building-gems/123397 "2018-03-11T03:58:00Z")
It seems like if I build my project, it just places my code files in /path/to/logstash/vendor/local\_gems/random\_id/plugin-goes-here and adds a reference in the Gemfile. Does it even matter if I build my gem with Jruby or… --- ## [Mutate converts integer to string](https://discuss.elastic.co/t/mutate-converts-integer-to-string/123092)
**Author:** [@cawoodm](https://discuss.elastic.co/u/cawoodm)\ **Replies:** 4\ **Last updated:** [March 10, 2018, 8:38pm UTC](https://discuss.elastic.co/t/mutate-converts-integer-to-string/123092 "2018-03-10T20:38:32Z")
We have the following filter (Logstash 6.2 Windows): if \[category\] == "hybrisserver" and \[raw\] =~ "Starting up hybris server" { mutate { update =\> {"priority" =\> 2 "category" =\> "ServerStart" } } } The incoming events… --- ## [Filebeat 6.2.1 and logstash 6.2.1 and doc type](https://discuss.elastic.co/t/filebeat-6-2-1-and-logstash-6-2-1-and-doc-type/123180)
**Author:** [@bloke](https://discuss.elastic.co/u/bloke)\ **Replies:** 1\ **Last updated:** [March 10, 2018, 6:13am UTC](https://discuss.elastic.co/t/filebeat-6-2-1-and-logstash-6-2-1-and-doc-type/123180 "2018-03-10T06:13:57Z")
Hi, I recently updated ELK from 5.4 to 6.2.1 My upgrade went reasonably well and i modified some custom templates on the elasticsearch side in relation to SRX data and geoip. That all works. My question is really in r… --- ## [Grok filter failing](https://discuss.elastic.co/t/grok-filter-failing/123312)
**Author:** [@millerb1](https://discuss.elastic.co/u/millerb1)\ **Replies:** 1\ **Last updated:** [March 10, 2018, 4:19am UTC](https://discuss.elastic.co/t/grok-filter-failing/123312 "2018-03-10T04:19:01Z")
I have created a grok filter for JBoss logs. The grok fails when I include %{TIME:time} %{LOGLEVEL:level} and I can not read time and level grok { match =\> \[ "%{TIME:time} %{LOGLEVEL:level} \[(?\[^\]\]+)\] ((?\[^)\]+)) %{GRE… --- ## [What would be the best heap space setting for Logstash?](https://discuss.elastic.co/t/what-would-be-the-best-heap-space-setting-for-logstash/123332)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 1\ **Last updated:** [March 10, 2018, 1:24am UTC](https://discuss.elastic.co/t/what-would-be-the-best-heap-space-setting-for-logstash/123332 "2018-03-10T01:24:56Z")
Hi, Can someone please suggest me the -Xms and -Xms heap space settings for Logstash should be? Thanks --- ## [Logstash fails to send data to Elasticsearch](https://discuss.elastic.co/t/logstash-fails-to-send-data-to-elasticsearch/123347)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 1\ **Last updated:** [March 10, 2018, 12:33am UTC](https://discuss.elastic.co/t/logstash-fails-to-send-data-to-elasticsearch/123347 "2018-03-10T00:33:39Z")
Hi, I have a Logstash running which sends data to Elasticsearch nodes and also to other Logstash over http output plugin. Problem I see is, whenever the Losgtash(destination) is stopped(for whatever reason), Logstash(so… --- ## [Pipeline aborted due to "Grok::PatternError"](https://discuss.elastic.co/t/pipeline-aborted-due-to-grok-patternerror/123343)
**Author:** [@bo9354](https://discuss.elastic.co/u/bo9354)\ **Replies:** 2\ **Last updated:** [March 9, 2018, 9:58pm UTC](https://discuss.elastic.co/t/pipeline-aborted-due-to-grok-patternerror/123343 "2018-03-09T21:58:22Z")
Configuration OK Blockquote Adding pattern {"SYSLOG5424LINE"=\>"%{SYSLOG5424BASE} +%{GREEDYDATA:syslog5424\_msg}", :level=\>:info, :file=\>"grok-pure.rb", :line=\>"62", :method=\>"add\_pattern"} Adding pattern {"MCOLLECTIVE… --- ## [Help with logstash json output to split a field in multiple variables and combine them to a new field](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592)
**Author:** [@micruzz82](https://discuss.elastic.co/u/micruzz82)\ **Replies:** 11\ **Last updated:** [March 9, 2018, 9:57pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592 "2018-03-09T21:57:10Z")
Hi All I have this JSON output as below. { "totalCount": "1", "imdata": \[ { "l1PhysIf": { "attributes": { "adminSt": "up", "autoNeg": "on", "brkoutMap": "none", "bw": "0", "childAction": "", "delay": "1", "descr": "srv… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=803) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=805)