# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=753 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 754 --- ## [Extract subfields in JSON Field](https://discuss.elastic.co/t/extract-subfields-in-json-field/133111)
**Author:** [@Alessio\_Frabotta](https://discuss.elastic.co/u/Alessio_Frabotta)\ **Replies:** 7\ **Last updated:** [May 25, 2018, 8:45am UTC](https://discuss.elastic.co/t/extract-subfields-in-json-field/133111 "2018-05-25T08:45:46Z")
I have this JSON log message (from logstash-logback-encoder): { "@version" =\> "1", "@timestamp" =\> 2018-05-24T09:49:43.336Z, "message" =\> "{ "@timestamp":"2018-05-24T11:49:40.521+02:00", "thr… --- ## [Regarding Logstash Persistent Queus](https://discuss.elastic.co/t/regarding-logstash-persistent-queus/133261)
**Author:** [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\ **Replies:** 0\ **Last updated:** [May 25, 2018, 6:22am UTC](https://discuss.elastic.co/t/regarding-logstash-persistent-queus/133261 "2018-05-25T06:22:37Z")
Hello all, This issue bugs me for one week now. Setup We have a cluster of 4 ES nodes. 3 of it is a master/data eligible and 1 coordinating node. we also have logstash running on the same server of master/data eligib… --- ## [When I use input-jdbc and output webhdfs to do something occur %{host} %{message}](https://discuss.elastic.co/t/when-i-use-input-jdbc-and-output-webhdfs-to-do-something-occur-host-message/133243)
**Author:** [@hfq-jingsi](https://discuss.elastic.co/u/hfq-jingsi)\ **Replies:** 3\ **Last updated:** [May 25, 2018, 6:18am UTC](https://discuss.elastic.co/t/when-i-use-input-jdbc-and-output-webhdfs-to-do-something-occur-host-message/133243 "2018-05-25T06:18:55Z")
when I testing , anything is ok. But using jdbc plug-ins, the information is " %{host} %{message}" all . Please help me . --- ## [How to monitor logstash logs](https://discuss.elastic.co/t/how-to-monitor-logstash-logs/133077)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 7\ **Last updated:** [May 25, 2018, 6:16am UTC](https://discuss.elastic.co/t/how-to-monitor-logstash-logs/133077 "2018-05-25T06:16:33Z")
Please suggest me how to monitor Logstash logs. I am trying below solution but it's not working... Generating my logstash logs in json format in logstash log folder... By using File input to logstash ingesting this lo… --- ## [How to keep my entries ordered as in my file](https://discuss.elastic.co/t/how-to-keep-my-entries-ordered-as-in-my-file/133226)
**Author:** [@czem](https://discuss.elastic.co/u/czem)\ **Replies:** 0\ **Last updated:** [May 24, 2018, 9:55pm UTC](https://discuss.elastic.co/t/how-to-keep-my-entries-ordered-as-in-my-file/133226 "2018-05-24T21:55:41Z")
I'm ingesting a log file with Filebeat and sending to my ElasticSearch through Logstash. My log file has the following pattern: Timestamp - TransactionId - Header (Username) Timestamp - TransactionId - Events (This line… --- ## [Palo Alto filter](https://discuss.elastic.co/t/palo-alto-filter/132838)
**Author:** [@asalma](https://discuss.elastic.co/u/asalma)\ **Replies:** 1\ **Last updated:** [May 24, 2018, 8:45pm UTC](https://discuss.elastic.co/t/palo-alto-filter/132838 "2018-05-24T20:45:03Z")
Please is this part of config file correcte : else if \[fields\]\[log\_type\] == "paloalto" { csv { "data" =\> "FUTURE\_USE", "Receive Time", "Serial Number", "Type", "Subtype", "FUTURE\_… --- ## [Push PostgresDB Server complete data to logstash](https://discuss.elastic.co/t/push-postgresdb-server-complete-data-to-logstash/133204)
**Author:** [@kaushalshriyan](https://discuss.elastic.co/u/kaushalshriyan)\ **Replies:** 1\ **Last updated:** [May 24, 2018, 8:38pm UTC](https://discuss.elastic.co/t/push-postgresdb-server-complete-data-to-logstash/133204 "2018-05-24T20:38:58Z")
Hi, I have run /usr/share/logstash/bin/logstash-plugin install logstash-input-jdbc after following https://www.elastic.co/blog/logstash-jdbc-input-plugin. Is there a way to import complete Postgres Database server data … --- ## [Need help with 403 errors Logstash 6.2.4 with X-Pack](https://discuss.elastic.co/t/need-help-with-403-errors-logstash-6-2-4-with-x-pack/132876)
**Author:** [@jn1965](https://discuss.elastic.co/u/jn1965)\ **Replies:** 1\ **Last updated:** [May 24, 2018, 7:29pm UTC](https://discuss.elastic.co/t/need-help-with-403-errors-logstash-6-2-4-with-x-pack/132876 "2018-05-24T19:29:54Z")
Everythign was working fine until I installed X-Pack on a remote logstash server. Now I am geting this error: \[2018-05-22T18:30:12,500\]\[ERROR\]\[logstash.config.sourceloader\] Could not fetch all the sources {:exception=\>L… --- ## [How to use 'AND' in Logstash?](https://discuss.elastic.co/t/how-to-use-and-in-logstash/132695)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 11\ **Last updated:** [May 24, 2018, 7:12pm UTC](https://discuss.elastic.co/t/how-to-use-and-in-logstash/132695 "2018-05-24T19:12:31Z")
Hi, Can someone please say if it's correct to use the following, if \[message\] =~ /department=technology&space=dev/{ ... } Basically I am looking in \[message\] for the fields department and space. I am using operator &,… --- ## [Kafka (Confluent Platform) input - broken message encoding](https://discuss.elastic.co/t/kafka-confluent-platform-input-broken-message-encoding/132881)
**Author:** [@LipatovNikita](https://discuss.elastic.co/u/LipatovNikita)\ **Replies:** 1\ **Last updated:** [May 24, 2018, 5:19pm UTC](https://discuss.elastic.co/t/kafka-confluent-platform-input-broken-message-encoding/132881 "2018-05-24T17:19:42Z")
Hi, I have a Confluent Platform (version 4.1.1). It is configured to read data from the database. The configuration for this is: name = source-mysql-requests connection.url = jdbc:mysql://localhost:3306/Requests connect… --- ## [How to create a logstash instance in Elastic Cloud](https://discuss.elastic.co/t/how-to-create-a-logstash-instance-in-elastic-cloud/131731)
**Author:** [@EloaCardoso](https://discuss.elastic.co/u/EloaCardoso)\ **Replies:** 8\ **Last updated:** [May 24, 2018, 5:14pm UTC](https://discuss.elastic.co/t/how-to-create-a-logstash-instance-in-elastic-cloud/131731 "2018-05-24T17:14:36Z")
I am a bit lost in how to create a logstash instance in the EC. I've changed the following fields within logstash.yml file: cloud.id: \:\ cloud.auth: \:\ http.host: "\#\\>](https://discuss.elastic.co/t/error-nomethoderror-undefined-method-truncate-for-iowriter-0x3e9b038-io-zlib-gzipwriter-0x164b00d/132392)
**Author:** [@alexus](https://discuss.elastic.co/u/alexus)\ **Replies:** 3\ **Last updated:** [May 24, 2018, 4:51pm UTC](https://discuss.elastic.co/t/error-nomethoderror-undefined-method-truncate-for-iowriter-0x3e9b038-io-zlib-gzipwriter-0x164b00d/132392 "2018-05-24T16:51:11Z")
I'm using docker.elastic.co/logstash/logstash:6.2.4 and while running bellow pipeline configuration, I get the following: output { file { path =\> "/tmp/data.json.gz" write\_behavior =\> overwrite gzip =\> true } } … --- ## [Ruby script file with dynamic param](https://discuss.elastic.co/t/ruby-script-file-with-dynamic-param/133130)
**Author:** [@jonviously](https://discuss.elastic.co/u/jonviously)\ **Replies:** 3\ **Last updated:** [May 24, 2018, 3:47pm UTC](https://discuss.elastic.co/t/ruby-script-file-with-dynamic-param/133130 "2018-05-24T15:47:39Z")
hi everyone ! in order to factoring my logstash configuration i wanna replace many similar ruby inline code by a ruby script with dynamic param. what i got before my ruby code : \[ 'key1 =\> 'value1', 'key2 =\> 'value2… --- ## [I'm just confused in using grok](https://discuss.elastic.co/t/im-just-confused-in-using-grok/133153)
**Author:** [@Ben\_Chaabene\_Wissem](https://discuss.elastic.co/u/Ben_Chaabene_Wissem)\ **Replies:** 9\ **Last updated:** [May 24, 2018, 3:33pm UTC](https://discuss.elastic.co/t/im-just-confused-in-using-grok/133153 "2018-05-24T15:33:00Z")
Hello, I'm beginner with grok and I'm trying to extract log patterns using match property. This what I do : grok { match =\> { "message" =\> "REQUEST\\t%{WORD:senario\_group}\\t{NOTSPACE:user\_id}\\t% {WORD:senario\_name}\\t%… --- ## [Logstash too many open files error](https://discuss.elastic.co/t/logstash-too-many-open-files-error/133182)
**Author:** [@Kaushik](https://discuss.elastic.co/u/Kaushik)\ **Replies:** 0\ **Last updated:** [May 24, 2018, 3:13pm UTC](https://discuss.elastic.co/t/logstash-too-many-open-files-error/133182 "2018-05-24T15:13:35Z")
Hello there, We have set up ELK on a single VM. Kibana was not able to display desired logs for a period of 10 days. So the error seemed to be with logstash as per error logs. The error was dial udp x.x.x.x:53: socket:… --- ## [Logstash Not using environment proxy setting -- Authentication 407 error](https://discuss.elastic.co/t/logstash-not-using-environment-proxy-setting-authentication-407-error/133176)
**Author:** [@ysumit](https://discuss.elastic.co/u/ysumit)\ **Replies:** 0\ **Last updated:** [May 24, 2018, 2:51pm UTC](https://discuss.elastic.co/t/logstash-not-using-environment-proxy-setting-authentication-407-error/133176 "2018-05-24T14:51:25Z")
Hi Team Logstash 6.2 not using proxy setting of env as well as mentioned in configuration file: -\> proxy =\> "http://username:password@northproxy:4145" in file. -\> export http\_proxy=http://username:password@northprox… --- ## [/var/log/logstash-plain.log is filling the disk space](https://discuss.elastic.co/t/var-log-logstash-plain-log-is-filling-the-disk-space/133112)
**Author:** [@venki522](https://discuss.elastic.co/u/venki522)\ **Replies:** 4\ **Last updated:** [May 24, 2018, 2:22pm UTC](https://discuss.elastic.co/t/var-log-logstash-plain-log-is-filling-the-disk-space/133112 "2018-05-24T14:22:05Z")
Hello all, I am using logstash for netflow collection and iam running logstash 6.2.During the process of collection the /var/log/logstash-plain.log is filling my disk space.Once the disk space is full the netflow collec… --- ## [RSS NoMethodError nil:NilClass](https://discuss.elastic.co/t/rss-nomethoderror-nil-nilclass/133080)
**Author:** [@Ole](https://discuss.elastic.co/u/Ole)\ **Replies:** 2\ **Last updated:** [May 24, 2018, 1:29pm UTC](https://discuss.elastic.co/t/rss-nomethoderror-nil-nilclass/133080 "2018-05-24T13:29:33Z")
I've just started working with Logstash 6.2.4 and want to use it with RSS feeds. I've installed the RSS plugin and I'm using this as my pipeline.conf: input { rss { url =\> "http://www.nrk.no/rss" interval =\> 60 tag… --- ## [How to parse xml from a single line](https://discuss.elastic.co/t/how-to-parse-xml-from-a-single-line/131975)
**Author:** [@gonzalo](https://discuss.elastic.co/u/gonzalo)\ **Replies:** 6\ **Last updated:** [May 24, 2018, 1:01pm UTC](https://discuss.elastic.co/t/how-to-parse-xml-from-a-single-line/131975 "2018-05-24T13:01:32Z")
Hello, Before starting just thank you all. The fact is that I am trying to parse an XML through the XML filter from logstash but I am getting blocked. The XML format is as follows: \ **Author:** [@motts](https://discuss.elastic.co/u/motts)\ **Replies:** 0\ **Last updated:** [May 24, 2018, 1:01pm UTC](https://discuss.elastic.co/t/create-a-field-for-a-specific-line-number/133151 "2018-05-24T13:01:04Z") Hello, I have these logs that have fields and values, except for the 3rd line of every log. Is there something I can do in Logstash that gives the 3rd line a field for that value? Here is an example of my data in JSON: … --- ## [Multiline codec with file input](https://discuss.elastic.co/t/multiline-codec-with-file-input/132895)
**Author:** [@Badger](https://discuss.elastic.co/u/Badger)\ **Replies:** 4\ **Last updated:** [May 24, 2018, 12:55pm UTC](https://discuss.elastic.co/t/multiline-codec-with-file-input/132895 "2018-05-24T12:55:18Z")
No, it's not the usual problem. Using 6.2.4 I have a file that contains 1 1 2 2 3 They really are blank lines. # od -a bar.txt 0000000 1 nl 1 nl nl 2 nl 2 nl nl 3 nl With a stdin input, a multiline… --- ## [Webhdfs Output plugin does not login with Keytab](https://discuss.elastic.co/t/webhdfs-output-plugin-does-not-login-with-keytab/132164)
**Author:** [@Felipe\_Manzur](https://discuss.elastic.co/u/Felipe_Manzur)\ **Replies:** 6\ **Last updated:** [May 24, 2018, 12:05pm UTC](https://discuss.elastic.co/t/webhdfs-output-plugin-does-not-login-with-keytab/132164 "2018-05-24T12:05:32Z")
Hi, I configured a pipeline using the webhdfs output plugin, but i'm having trouble to login into kerberos using a keytab. If i login by command line using the keytab "kinit -kt \ " , i get the kerberos tic… --- ## [Logstash Configuration IlegalStateException](https://discuss.elastic.co/t/logstash-configuration-ilegalstateexception/133121)
**Author:** [@Tempy](https://discuss.elastic.co/u/Tempy)\ **Replies:** 2\ **Last updated:** [May 24, 2018, 11:31am UTC](https://discuss.elastic.co/t/logstash-configuration-ilegalstateexception/133121 "2018-05-24T11:31:47Z")
Hello, I ran /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/filebeat.conf --config.test\_and\_exit and got this as an output \[FATAL\] 2018-05-24 12:24:15.136 \[LogStash::Runner\] runner - The given configuratio… --- ## [Logstash Keystore password can be read from Linux command](https://discuss.elastic.co/t/logstash-keystore-password-can-be-read-from-linux-command/133119)
**Author:** [@keysersozee](https://discuss.elastic.co/u/keysersozee)\ **Replies:** 1\ **Last updated:** [May 24, 2018, 11:01am UTC](https://discuss.elastic.co/t/logstash-keystore-password-can-be-read-from-linux-command/133119 "2018-05-24T11:01:37Z")
Hello, the command cat /proc/$LOGSTASH\_PID/environ can tell about the logstash keystore password since it's a an environment variable (LOGSTASH\_KEYSTORE\_PASS) Is there a way to secure this password ? Thank you --- ## [Are tags kept after pruning](https://discuss.elastic.co/t/are-tags-kept-after-pruning/132466)
**Author:** [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Replies:** 2\ **Last updated:** [May 24, 2018, 10:11am UTC](https://discuss.elastic.co/t/are-tags-kept-after-pruning/132466 "2018-05-24T10:11:08Z")
I am using multiple pipelines, one of which (pipeline\_1) adds some tags and sends its data downstream to pipeline\_2 via tcp output/input binding. pipeline\_2, as soon as it receives the event, it applies a prune, e.g.: f… --- ## [Logstash-codec-cloudtrail hung when parsing a large file](https://discuss.elastic.co/t/logstash-codec-cloudtrail-hung-when-parsing-a-large-file/133110)
**Author:** [@nikhilo](https://discuss.elastic.co/u/nikhilo)\ **Replies:** 0\ **Last updated:** [May 24, 2018, 10:00am UTC](https://discuss.elastic.co/t/logstash-codec-cloudtrail-hung-when-parsing-a-large-file/133110 "2018-05-24T10:00:23Z")
I'm seeing a problem with logstash-codec-cloudtrail where the processing just hangs without any error or debug logs when the codec encounters a large file. Tried enabling debug logs for the codec, but nothing is printed… --- ## [Logstash-6.2.4 Pipeline has terminated](https://discuss.elastic.co/t/logstash-6-2-4-pipeline-has-terminated/133010)
**Author:** [@Vadim\_Klimenko](https://discuss.elastic.co/u/Vadim_Klimenko)\ **Replies:** 3\ **Last updated:** [May 24, 2018, 9:15am UTC](https://discuss.elastic.co/t/logstash-6-2-4-pipeline-has-terminated/133010 "2018-05-24T09:15:41Z")
vadim@vadim-HP:~/www/FULL/Search/logstash-6.2.4$ bin/logstash -f logstash-simple.conf Sending Logstash's logs to /home/vadim/www/FULL/Search/logstash-6.2.4/logs which is now configured via log4j2.properties \[2018-05-2… --- ## [Logstash-6.2.4 Error while running conf](https://discuss.elastic.co/t/logstash-6-2-4-error-while-running-conf/132982)
**Author:** [@Vadim\_Klimenko](https://discuss.elastic.co/u/Vadim_Klimenko)\ **Replies:** 7\ **Last updated:** [May 24, 2018, 8:52am UTC](https://discuss.elastic.co/t/logstash-6-2-4-error-while-running-conf/132982 "2018-05-24T08:52:30Z")
Plugin:LogStash::Inputs::Jdbc jdbc\_connection\_string=\>"jdbc:mysql://localhost:3306/pall\_new", jdbc\_user=\>"root", jdbc\_password=\>, jdbc\_validate\_connection=\>true, jdbc\_driver\_library=\>"home/vadim/www/Full/Search/logstash… --- ## [Logstash data to Elasticsearch cloud](https://discuss.elastic.co/t/logstash-data-to-elasticsearch-cloud/132845)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 1\ **Last updated:** [May 24, 2018, 7:06am UTC](https://discuss.elastic.co/t/logstash-data-to-elasticsearch-cloud/132845 "2018-05-24T07:06:49Z")
Hey Guys, I have been trying to fetch data from serviceNow through logstash http\_poller method, i was able to achieve when elasticsearch is on premise. When i try to use the same method to send data to elasticsearch c… --- ## [Couldn't decode protobuf: #\ **Author:** [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\ **Replies:** 0\ **Last updated:** [May 24, 2018, 6:01am UTC](https://discuss.elastic.co/t/couldnt-decode-protobuf-runtimeerror-protocol-message-tag-had-invalid-wire-type/133076 "2018-05-24T06:01:39Z") Hi, I am using protobuf logstash plugin for decoding of messages which were encoded using ruby. Here is my .proto used file syntax = "proto3"; message SearchRequest { string query = 1; int64 page\_number = 2; int64… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=752) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=754)