# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=751 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 752 --- ## [Mutate Gsub not working](https://discuss.elastic.co/t/mutate-gsub-not-working/133304)
**Author:** [@npontes](https://discuss.elastic.co/u/npontes)\ **Replies:** 14\ **Last updated:** [May 29, 2018, 12:20pm UTC](https://discuss.elastic.co/t/mutate-gsub-not-working/133304 "2018-05-29T12:20:24Z")
Hello, I'm currently extracting data from Elasticsearch to a CSV using logstash, however I have some fields that returns a null, so I would have in my CSV, something like: 1,,2,3,4,,,,2. My configuration is the follo… --- ## [Logstash two different log types](https://discuss.elastic.co/t/logstash-two-different-log-types/130002)
**Author:** [@JensVanDeynse](https://discuss.elastic.co/u/JensVanDeynse)\ **Replies:** 7\ **Last updated:** [May 29, 2018, 11:56am UTC](https://discuss.elastic.co/t/logstash-two-different-log-types/130002 "2018-05-29T11:56:02Z")
Hello there I've been experimenting with logstash but ran into a problem when trying to parse haproxy logs. The issue is as follows, I've been able to implement a grok parser function for the most common logs but somet… --- ## [Logstash grok pattern: Unexpected output](https://discuss.elastic.co/t/logstash-grok-pattern-unexpected-output/133616)
**Author:** [@rahulnama](https://discuss.elastic.co/u/rahulnama)\ **Replies:** 4\ **Last updated:** [May 29, 2018, 11:48am UTC](https://discuss.elastic.co/t/logstash-grok-pattern-unexpected-output/133616 "2018-05-29T11:48:25Z")
Hello, This is the sample log pattern I'm parsing. I'm using grok but it's not exactly as what I expected 180528 8:46:26 2 Query SELECT 1 To parse this log my grok pattern is %{NUMBER:date} %{NOTSPACE:time}%{INT:pid}… --- ## [Logstash heap size vs. Elasticsearch heap size](https://discuss.elastic.co/t/logstash-heap-size-vs-elasticsearch-heap-size/133662)
**Author:** [@dr01](https://discuss.elastic.co/u/dr01)\ **Replies:** 2\ **Last updated:** [May 29, 2018, 11:19am UTC](https://discuss.elastic.co/t/logstash-heap-size-vs-elasticsearch-heap-size/133662 "2018-05-29T11:19:33Z")
Hello, As a rule of thumb, does it make sense to set a Logstash heap size equal (or proportional) to the Elasticsearch heap size, or there are other factors to take into consideration? For instance, would an ELK stack … --- ## [Is it possible to configure LogStash with external input file?](https://discuss.elastic.co/t/is-it-possible-to-configure-logstash-with-external-input-file/133653)
**Author:** [@clash\_of\_clanss](https://discuss.elastic.co/u/clash_of_clanss)\ **Replies:** 3\ **Last updated:** [May 29, 2018, 10:33am UTC](https://discuss.elastic.co/t/is-it-possible-to-configure-logstash-with-external-input-file/133653 "2018-05-29T10:33:08Z")
Now I am working with ELK(ElasticSearch, LogStash, Kibana). I would like to know, how to configure LogStash with external input configuration file. Lets say for example If I am having an external xml configuration file l… --- ## [Logstash leaving temporary jar files in Windows\\Temp](https://discuss.elastic.co/t/logstash-leaving-temporary-jar-files-in-windows-temp/133654)
**Author:** [@Matt\_Pearce](https://discuss.elastic.co/u/Matt_Pearce)\ **Replies:** 0\ **Last updated:** [May 29, 2018, 10:08am UTC](https://discuss.elastic.co/t/logstash-leaving-temporary-jar-files-in-windows-temp/133654 "2018-05-29T10:08:51Z")
Hi, I visited a client last week who were running Logstash 6.2.2 on Windows, having installed it as a service using NSSM. They'd run out of disk space on their main (C:) drive, which we eventually tracked down to the Wi… --- ## [Logstash 6.2.3 Arcsight Module setup error](https://discuss.elastic.co/t/logstash-6-2-3-arcsight-module-setup-error/133648)
**Author:** [@shaigb](https://discuss.elastic.co/u/shaigb)\ **Replies:** 1\ **Last updated:** [May 29, 2018, 9:48am UTC](https://discuss.elastic.co/t/logstash-6-2-3-arcsight-module-setup-error/133648 "2018-05-29T09:48:58Z")
Hi , I'm trying to Integrate arcsight SIEM and ELK. I don't know why, but every version of ELK that I'm trying to deploy and install, I always ger an error in the Logstash Arcsight Module Setup when I run it from the c… --- ## [Performance tuning](https://discuss.elastic.co/t/performance-tuning/133623)
**Author:** [@nickytd](https://discuss.elastic.co/u/nickytd)\ **Replies:** 1\ **Last updated:** [May 29, 2018, 9:19am UTC](https://discuss.elastic.co/t/performance-tuning/133623 "2018-05-29T09:19:33Z")
Hi I am trying to chase a CPU utilisation on logstash and aways get "oneagentautosensor" as the hottest thread Hot threads at 2018-05-29T06:29:15Z, busiestThreads=3: 12 % of cpu usage, state: runnable, thread name: 'on… --- ## [ELb logs from s3 to ELK server](https://discuss.elastic.co/t/elb-logs-from-s3-to-elk-server/133168)
**Author:** [@Suresh\_Pal](https://discuss.elastic.co/u/Suresh_Pal)\ **Replies:** 3\ **Last updated:** [May 29, 2018, 7:45am UTC](https://discuss.elastic.co/t/elb-logs-from-s3-to-elk-server/133168 "2018-05-29T07:45:21Z")
I'm trying to ship my elb logs from the s3 bucket to ELK server. I'm getting following error /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/s3\_elb\_logs.conf WARNING: Could not find logstash.yml which is typic… --- ## [Can't dup NilClass event.get()](https://discuss.elastic.co/t/cant-dup-nilclass-event-get/132276)
**Author:** [@vicennt](https://discuss.elastic.co/u/vicennt)\ **Replies:** 8\ **Last updated:** [May 29, 2018, 6:38am UTC](https://discuss.elastic.co/t/cant-dup-nilclass-event-get/132276 "2018-05-29T06:38:18Z")
Hi there! First sorry for my English. I have a problem when I try to get the value from grok field. I'm using the event.get but when I try to pass this value to a Time.parse() I always get \`Can't dup NilClass error'. E… --- ## [Custom Filter Plugin - Performance Test](https://discuss.elastic.co/t/custom-filter-plugin-performance-test/133551)
**Author:** [@Emanuel9](https://discuss.elastic.co/u/Emanuel9)\ **Replies:** 2\ **Last updated:** [May 29, 2018, 6:34am UTC](https://discuss.elastic.co/t/custom-filter-plugin-performance-test/133551 "2018-05-29T06:34:12Z")
Hey all. I have set up the ELK-Stack and I have written my own Logstash filter plugins in ruby. Now I would like to add unit tests for my ruby filters and also check if I can improve my filters in any way. So I was loo… --- ## [Missing events using JDBC input](https://discuss.elastic.co/t/missing-events-using-jdbc-input/133585)
**Author:** [@Olatunde\_Tokun](https://discuss.elastic.co/u/Olatunde_Tokun)\ **Replies:** 1\ **Last updated:** [May 29, 2018, 5:59am UTC](https://discuss.elastic.co/t/missing-events-using-jdbc-input/133585 "2018-05-29T05:59:26Z")
To stream events from the database of a device that logs events in milliseconds using the logstash JDBC input. The input file is scheduled to update the index every second with new entries into the DB. The issue with th… --- ## [Parse log line into JSON even when some tags from log can miss](https://discuss.elastic.co/t/parse-log-line-into-json-even-when-some-tags-from-log-can-miss/133559)
**Author:** [@Vinitc](https://discuss.elastic.co/u/Vinitc)\ **Replies:** 4\ **Last updated:** [May 29, 2018, 5:55am UTC](https://discuss.elastic.co/t/parse-log-line-into-json-even-when-some-tags-from-log-can-miss/133559 "2018-05-29T05:55:30Z")
Hi, My log line looks like below. In my log it is possible that few tags could miss (example: CITY or ZIPCODE or any other tag may not always be present). I want to parse this log and show output as JSON. Log Line: 20… --- ## [Multiple pipeline](https://discuss.elastic.co/t/multiple-pipeline/133451)
**Author:** [@Saketh\_Chandra\_Kolis](https://discuss.elastic.co/u/Saketh_Chandra_Kolis)\ **Replies:** 6\ **Last updated:** [May 29, 2018, 5:35am UTC](https://discuss.elastic.co/t/multiple-pipeline/133451 "2018-05-29T05:35:13Z")
I am using a windows operating system.Where should i store my pipeline.yml file.i placed it in the config folder but i get an error. This is the error i am getting: C:\\logstash-5.4.1\>bin\\logstash Sending Logstash's lo… --- ## [Conditional expression to test for a list of possible partial matches in a string](https://discuss.elastic.co/t/conditional-expression-to-test-for-a-list-of-possible-partial-matches-in-a-string/133583)
**Author:** [@jlsam](https://discuss.elastic.co/u/jlsam)\ **Replies:** 3\ **Last updated:** [May 29, 2018, 12:16am UTC](https://discuss.elastic.co/t/conditional-expression-to-test-for-a-list-of-possible-partial-matches-in-a-string/133583 "2018-05-29T00:16:14Z")
I want to select and output only the uncommon messages in a log, based on a partial match (the beginning of the message). I can filter out one message like so output { if "foo" not in \[msg\] { But if it try to match … --- ## [Logstach automatic logs reader](https://discuss.elastic.co/t/logstach-automatic-logs-reader/132046)
**Author:** [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\ **Replies:** 12\ **Last updated:** [May 28, 2018, 2:36pm UTC](https://discuss.elastic.co/t/logstach-automatic-logs-reader/132046 "2018-05-28T14:36:54Z")
How can we read automatic log because if the time differs with the machine timestamp it dosen't display with the filters. --- ## [Grok filter: check if field exists](https://discuss.elastic.co/t/grok-filter-check-if-field-exists/132103)
**Author:** [@Alessio\_Frabotta](https://discuss.elastic.co/u/Alessio_Frabotta)\ **Replies:** 4\ **Last updated:** [May 28, 2018, 2:08pm UTC](https://discuss.elastic.co/t/grok-filter-check-if-field-exists/132103 "2018-05-28T14:08:29Z")
I have a log message with this structure: "message" =\> "{ "@timestamp":"201856T12:54:33.347+02:00", "thread":"main", "logger\_name":"org.elasticsearch.bootstrap", "level":"WARN", "message":"JNA no… --- ## [KV filter problem with value\_split](https://discuss.elastic.co/t/kv-filter-problem-with-value-split/133436)
**Author:** [@DanV](https://discuss.elastic.co/u/DanV)\ **Replies:** 1\ **Last updated:** [May 28, 2018, 2:02pm UTC](https://discuss.elastic.co/t/kv-filter-problem-with-value-split/133436 "2018-05-28T14:02:11Z")
Hello, i'm using the kv filter to parse a log, especially field named User that has a string value of: User Name: Daniel Venzi the kv filter is in the format: kv { … --- ## [Exception on configuring Logstash](https://discuss.elastic.co/t/exception-on-configuring-logstash/133510)
**Author:** [@Anca\_Turc](https://discuss.elastic.co/u/Anca_Turc)\ **Replies:** 4\ **Last updated:** [May 28, 2018, 1:17pm UTC](https://discuss.elastic.co/t/exception-on-configuring-logstash/133510 "2018-05-28T13:17:50Z")
Hi , I am trying to filter the traffic for search queries and I have followed this tutorial where is specified the logstash config file sniff\_search.conf. Input { beats { port =\> 5044 } } filter {… --- ## [Logstash CPU utilization is high](https://discuss.elastic.co/t/logstash-cpu-utilization-is-high/132727)
**Author:** [@mamta](https://discuss.elastic.co/u/mamta)\ **Replies:** 39\ **Last updated:** [May 28, 2018, 12:15pm UTC](https://discuss.elastic.co/t/logstash-cpu-utilization-is-high/132727 "2018-05-28T12:15:33Z")
Hi, I am using logstash 6.2.4 version. When I try to run any conf file at that time CPU utilization is high. So currently there is no conf file is running but still, CPU usages are high. I have attached a screenshot of … --- ## [Elasticsearch input running behind a proxy](https://discuss.elastic.co/t/elasticsearch-input-running-behind-a-proxy/133530)
**Author:** [@madou23](https://discuss.elastic.co/u/madou23)\ **Replies:** 0\ **Last updated:** [May 28, 2018, 11:59am UTC](https://discuss.elastic.co/t/elasticsearch-input-running-behind-a-proxy/133530 "2018-05-28T11:59:20Z")
Hi all, I'm trying to get data from an Elasticsearch cluster to another via Logstash the problem is that the input cluster is running behind a proxy. This is my configuration: input { elasticsearch { hosts =\> "my… --- ## [Aggregation through Logstash](https://discuss.elastic.co/t/aggregation-through-logstash/133129)
**Author:** [@Alok](https://discuss.elastic.co/u/Alok)\ **Replies:** 5\ **Last updated:** [May 28, 2018, 11:55am UTC](https://discuss.elastic.co/t/aggregation-through-logstash/133129 "2018-05-28T11:55:27Z")
Dear Experts I am using below aggregation to add child table entries to given parent ids - aggregate { task\_id =\> "%{id}" code =\> " map\['id'\] = event.get('id') map\['sizes'\] ||= \[\] map\['sizes'\] \<\< {'width' =\> event… --- ## [How to send grok field into ruby](https://discuss.elastic.co/t/how-to-send-grok-field-into-ruby/133381)
**Author:** [@veerendranath\_lukkan](https://discuss.elastic.co/u/veerendranath_lukkan)\ **Replies:** 8\ **Last updated:** [May 28, 2018, 11:54am UTC](https://discuss.elastic.co/t/how-to-send-grok-field-into-ruby/133381 "2018-05-28T11:54:50Z")
filter{ grok{ match =\>{"message" =\> "%{GREEDYDATA:selectedMessage}"} } Ruby{ path ="test.rb" script\_params =\> { "percentage" =\> selectedMessage } } } how can send any example! --- ## [Logstash data storage](https://discuss.elastic.co/t/logstash-data-storage/133468)
**Author:** [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\ **Replies:** 5\ **Last updated:** [May 28, 2018, 9:52am UTC](https://discuss.elastic.co/t/logstash-data-storage/133468 "2018-05-28T09:52:18Z")
Hello all I am running my ELk stack on docker so whenever my logstash loses connection with elasticsearch then the data which is getting parsed where is it stored. According to my observation it is piling up the logsta… --- ## [Grok filter for selecting and formatting certain logs lines](https://discuss.elastic.co/t/grok-filter-for-selecting-and-formatting-certain-logs-lines/131892)
**Author:** [@Parthgandhi](https://discuss.elastic.co/u/Parthgandhi)\ **Replies:** 8\ **Last updated:** [May 28, 2018, 9:46am UTC](https://discuss.elastic.co/t/grok-filter-for-selecting-and-formatting-certain-logs-lines/131892 "2018-05-28T09:46:01Z")
HI Team, I am writing up a grok filter for parsing my application log which is unstructured. What i need is to look for certain lines and generate output in a specific format. e.g below are my logs 2018-05-07 01:19:40… --- ## [Encoding charset logstash accents log problem](https://discuss.elastic.co/t/encoding-charset-logstash-accents-log-problem/133087)
**Author:** [@dyl](https://discuss.elastic.co/u/dyl)\ **Replies:** 10\ **Last updated:** [May 28, 2018, 8:42am UTC](https://discuss.elastic.co/t/encoding-charset-logstash-accents-log-problem/133087 "2018-05-28T08:42:49Z")
Hello ELK community :slight\_smile: I've got a problem, I listen a "Vega" log file thanks toFilebeat and then I transfer these logs into Logstash to grok them etc... For exemple, a log line in my log file looks like : … --- ## [Is logstash working?!?!](https://discuss.elastic.co/t/is-logstash-working/133299)
**Author:** [@MrNerd](https://discuss.elastic.co/u/MrNerd)\ **Replies:** 5\ **Last updated:** [May 28, 2018, 7:45am UTC](https://discuss.elastic.co/t/is-logstash-working/133299 "2018-05-28T07:45:41Z")
Hy there I recently lost all my data, and im trying to reintall all the logstash sistem i had before, the problem here is that I don't know if logstash is tacking packets from this ipaddress: 192.168.128.51 Especificat… --- ## [How to run multiple conf file continusly in backend](https://discuss.elastic.co/t/how-to-run-multiple-conf-file-continusly-in-backend/133079)
**Author:** [@mamta](https://discuss.elastic.co/u/mamta)\ **Replies:** 10\ **Last updated:** [May 25, 2018, 6:03am UTC](https://discuss.elastic.co/t/how-to-run-multiple-conf-file-continusly-in-backend/133079 "2018-05-25T06:03:02Z")
Hey, I have installed ELK 5.6.8 version on Linux machine in a production environment. I have multiple conf files for each log.Currently I am running conf file from bin# ./logstash -f . I also tried to run from /etc/logs… --- ## [Trying to parse XML-logs](https://discuss.elastic.co/t/trying-to-parse-xml-logs/132749)
**Author:** [@Petersiemens](https://discuss.elastic.co/u/Petersiemens)\ **Replies:** 10\ **Last updated:** [May 28, 2018, 6:29am UTC](https://discuss.elastic.co/t/trying-to-parse-xml-logs/132749 "2018-05-28T06:29:12Z")
Hello, im currently trying to parse a XML-log. I worked with logstash the past month to parse simple single- or multiline logs, so im familiar with the functionality of input{} / filter{} / output{}. Now i want to par… --- ## [How to give class name more than two class in protobuf](https://discuss.elastic.co/t/how-to-give-class-name-more-than-two-class-in-protobuf/133459)
**Author:** [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\ **Replies:** 0\ **Last updated:** [May 28, 2018, 5:52am UTC](https://discuss.elastic.co/t/how-to-give-class-name-more-than-two-class-in-protobuf/133459 "2018-05-28T05:52:32Z")
I am having two .proto with different class names as person.proto syntax = "proto3"; message Person { int32 id = 1; string name = 2; string email = 3; } and second is message.proto syntax = "proto3"; message Se… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=750) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=752)