# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=813 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 814 --- ## [Logstash - UNEXPECTED POOL ERROR](https://discuss.elastic.co/t/logstash-unexpected-pool-error/121524)
**Author:** [@przemolb](https://discuss.elastic.co/u/przemolb)\ **Replies:** 1\ **Last updated:** [February 26, 2018, 5:33pm UTC](https://discuss.elastic.co/t/logstash-unexpected-pool-error/121524 "2018-02-26T17:33:22Z")
\[2018-02-26T16:28:52,669\]\[WARN \]\[logstash.outputs.elasticsearch\] UNEXPECTED POOL ERROR {:e=\>#\} \[2018-02-26T16:28:5… --- ## [Compress Json events (gzip, snappy, ...) before output](https://discuss.elastic.co/t/compress-json-events-gzip-snappy-before-output/121532)
**Author:** [@gghdr](https://discuss.elastic.co/u/gghdr)\ **Replies:** 0\ **Last updated:** [February 26, 2018, 5:24pm UTC](https://discuss.elastic.co/t/compress-json-events-gzip-snappy-before-output/121532 "2018-02-26T17:24:54Z")
Hello, i need to do exactly what i wrote in the topic, to compress my json event before shipping it to the output. My pipeline is: Kafka (json) input -\> filters -\> Jms (IBM Websphere MQ) output From my searches it see… --- ## [Confused and concerned](https://discuss.elastic.co/t/confused-and-concerned/120500)
**Author:** [@QuizzyRascal](https://discuss.elastic.co/u/QuizzyRascal)\ **Replies:** 1\ **Last updated:** [February 26, 2018, 5:11pm UTC](https://discuss.elastic.co/t/confused-and-concerned/120500 "2018-02-26T17:11:18Z")
Hi all, I spent my weekend building a CentOS 7 ELK (6.2.1) implementation to collect syslogs (redirecting 514 udp/tcp up to 5000). Everything in the end was bullet proof and I came into work feeling great. It didn't la… --- ## [Help: Trying to get logstash to start as a service](https://discuss.elastic.co/t/help-trying-to-get-logstash-to-start-as-a-service/120647)
**Author:** [@QuizzyRascal](https://discuss.elastic.co/u/QuizzyRascal)\ **Replies:** 1\ **Last updated:** [February 26, 2018, 5:10pm UTC](https://discuss.elastic.co/t/help-trying-to-get-logstash-to-start-as-a-service/120647 "2018-02-26T17:10:50Z")
Hi, I am really struggling to launch logstash as a service on CentOS 7 (see confused and concerned topic below). Since I cannot figure out what or where to set the -DJava.io.tmpdir= variable (which apparently would so… --- ## [Logstash service Centos](https://discuss.elastic.co/t/logstash-service-centos/120903)
**Author:** [@QuizzyRascal](https://discuss.elastic.co/u/QuizzyRascal)\ **Replies:** 1\ **Last updated:** [February 26, 2018, 5:10pm UTC](https://discuss.elastic.co/t/logstash-service-centos/120903 "2018-02-26T17:10:20Z")
Is there anyway to get debug information when starting a logstash service through sudo systemctl start logstash My listener isn't starting and so my service runs without the listener- I want to try and get the informat… --- ## [Kv Filter is not working](https://discuss.elastic.co/t/kv-filter-is-not-working/121523)
**Author:** [@Adansar](https://discuss.elastic.co/u/Adansar)\ **Replies:** 0\ **Last updated:** [February 26, 2018, 4:19pm UTC](https://discuss.elastic.co/t/kv-filter-is-not-working/121523 "2018-02-26T16:19:00Z")
Hello, I had a problem while parsing some logs, example : 2018-01-26 13:05:36.072Z ERROR \[Camel (context-autoprov) thread #2 - JmsConsumer\[inject\]\] n.a.cle.business.PairingAutoProcess - \[PL-MROV-002\]\[RO:REM\_JUnit\_2\]\[X… --- ## [CSV Filter - Quote character causing \_csvparsefailure](https://discuss.elastic.co/t/csv-filter-quote-character-causing-csvparsefailure/121273)
**Author:** [@perry29](https://discuss.elastic.co/u/perry29)\ **Replies:** 5\ **Last updated:** [February 26, 2018, 4:14pm UTC](https://discuss.elastic.co/t/csv-filter-quote-character-causing-csvparsefailure/121273 "2018-02-26T16:14:49Z")
The context is logstash and bro http logs. Both logstash and bro are minimally configured, just enough to test. Bro is generating tsv files and logstash is using the csv filter to process. In testing, logstash ocasion… --- ## [Error while testing pipeline](https://discuss.elastic.co/t/error-while-testing-pipeline/121431)
**Author:** [@amiyadutta](https://discuss.elastic.co/u/amiyadutta)\ **Replies:** 6\ **Last updated:** [February 26, 2018, 4:01pm UTC](https://discuss.elastic.co/t/error-while-testing-pipeline/121431 "2018-02-26T16:01:08Z")
Hi, When we run the following command to test the pipeline - curl -XGET 'localhost:9200/logstash-$DATE/\_search?pretty&q=response=200' output is - { "error" : { "root\_cause" : \[ { "type" : "index\_not\_found\_excepti… --- ## [What is the sickdb path](https://discuss.elastic.co/t/what-is-the-sickdb-path/121488)
**Author:** [@Srikanth\_FT](https://discuss.elastic.co/u/Srikanth_FT)\ **Replies:** 1\ **Last updated:** [February 26, 2018, 3:37pm UTC](https://discuss.elastic.co/t/what-is-the-sickdb-path/121488 "2018-02-26T15:37:32Z")
Hi, What is the path information where sickdb data stores. I need to reindex the full data sets again. best Mudireddy --- ## [How to transfer log to influxDB via elasticsearch](https://discuss.elastic.co/t/how-to-transfer-log-to-influxdb-via-elasticsearch/121437)
**Author:** [@abhisek](https://discuss.elastic.co/u/abhisek)\ **Replies:** 3\ **Last updated:** [February 26, 2018, 3:36pm UTC](https://discuss.elastic.co/t/how-to-transfer-log-to-influxdb-via-elasticsearch/121437 "2018-02-26T15:36:50Z")
Hi All, I need to transfer log to influxdb from elasticsearch, i have gone through the following document : https://www.elastic.co/guide/en/logstash/5.4/plugins-outputs-influxdb.html and came to the conclusion that i … --- ## [Logstash 6.2.1, 2 instances, both crashing with GC and filled heap memory](https://discuss.elastic.co/t/logstash-6-2-1-2-instances-both-crashing-with-gc-and-filled-heap-memory/121497)
**Author:** [@nkl](https://discuss.elastic.co/u/nkl)\ **Replies:** 0\ **Last updated:** [February 26, 2018, 1:59pm UTC](https://discuss.elastic.co/t/logstash-6-2-1-2-instances-both-crashing-with-gc-and-filled-heap-memory/121497 "2018-02-26T13:59:24Z")
Hi. We are having a lot of problems with our two Logstash instances. The instances have 72 cores and 128GB heap memory per host. I'll upload the monitoring screenshots below so you can have better insight into our conf… --- ## [Beats input w/ json codec](https://discuss.elastic.co/t/beats-input-w-json-codec/120060)
**Author:** [@joshday](https://discuss.elastic.co/u/joshday)\ **Replies:** 2\ **Last updated:** [February 26, 2018, 1:50pm UTC](https://discuss.elastic.co/t/beats-input-w-json-codec/120060 "2018-02-26T13:50:05Z")
So I know events coming from filebeat into logstash have a "source" field related to the full path of the file. The problem is my json file that I am sending will occasionally also have a 'source' field, which seems to b… --- ## [Elasticsearch / Logstash mapping error after upgrade](https://discuss.elastic.co/t/elasticsearch-logstash-mapping-error-after-upgrade/119814)
**Author:** [@petegriggs](https://discuss.elastic.co/u/petegriggs)\ **Replies:** 3\ **Last updated:** [February 26, 2018, 1:34pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-mapping-error-after-upgrade/119814 "2018-02-26T13:34:57Z")
Hi, Just upgraded to Elasticsearch and Logstash 6.2.0, everything was going well however logstash now can't put data in to elasticsearch. We are using this for IDS traffic the error we are getting is: \[2018-02-14T14:11… --- ## [Logstash utilising 100% CPU and java heap error](https://discuss.elastic.co/t/logstash-utilising-100-cpu-and-java-heap-error/121486)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 0\ **Last updated:** [February 26, 2018, 12:49pm UTC](https://discuss.elastic.co/t/logstash-utilising-100-cpu-and-java-heap-error/121486 "2018-02-26T12:49:53Z")
I am using HTTP as input and elastic search as output. I am getting logs from 100 different HTTP sources. For each source, i am creating different conf file input and output. when I am running each conf file CPU util… --- ## [Transform array objects in fields](https://discuss.elastic.co/t/transform-array-objects-in-fields/121269)
**Author:** [@Leonardo\_Santana](https://discuss.elastic.co/u/Leonardo_Santana)\ **Replies:** 1\ **Last updated:** [February 26, 2018, 12:08pm UTC](https://discuss.elastic.co/t/transform-array-objects-in-fields/121269 "2018-02-26T12:08:05Z")
hi, I have a object like this: "fields": \[ { "value": "123", "id": "0" }, { "value": "456", "id": "1" }, { "value": "llllobj", "id": "2" } \] but i want change the id to a field in my object like this: "f… --- ## [Despite of change in Lostash error pointing to same place to look into](https://discuss.elastic.co/t/despite-of-change-in-lostash-error-pointing-to-same-place-to-look-into/121385)
**Author:** [@TechGeek](https://discuss.elastic.co/u/TechGeek)\ **Replies:** 3\ **Last updated:** [February 26, 2018, 5:43am UTC](https://discuss.elastic.co/t/despite-of-change-in-lostash-error-pointing-to-same-place-to-look-into/121385 "2018-02-26T05:43:52Z")
Today I started learning about ES and LS using latest versions. Like any other curious newbie, experienced various issues and solved them one by one. I am hoping i.e. below mentioned problem will the last one before I w… --- ## [Path filename pattern problem](https://discuss.elastic.co/t/path-filename-pattern-problem/121462)
**Author:** [@Majuan](https://discuss.elastic.co/u/Majuan)\ **Replies:** 1\ **Last updated:** [February 26, 2018, 10:55am UTC](https://discuss.elastic.co/t/path-filename-pattern-problem/121462 "2018-02-26T10:55:03Z")
When I'm trying to make a filename pattern to get all log files : "logs\\Testlog4j\*.log" , it doesn't work, no data is sent to Elasticsearch. But when I specify the log file name, it works. --- ## [Logstash elasticsearch output plugin: configure retry count](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-configure-retry-count/121472)
**Author:** [@ov7a](https://discuss.elastic.co/u/ov7a)\ **Replies:** 0\ **Last updated:** [February 26, 2018, 10:51am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-configure-retry-count/121472 "2018-02-26T10:51:45Z")
My configuration is the following java app =\> log4j2 SocketAppender (tcp) =\> Logstash: tcp input, es output =\> elasticsearch Scenario: Elasticsearch cluster is red and do not accept indexing by some reason. Current b… --- ## [Unable to process the file input which has Key Value pairs](https://discuss.elastic.co/t/unable-to-process-the-file-input-which-has-key-value-pairs/118370)
**Author:** [@madeti](https://discuss.elastic.co/u/madeti)\ **Replies:** 7\ **Last updated:** [February 26, 2018, 10:51am UTC](https://discuss.elastic.co/t/unable-to-process-the-file-input-which-has-key-value-pairs/118370 "2018-02-26T10:51:07Z")
Hello, My input file is a single line file which has multiple key Value pairs in it. Of them the field or Key “content” is the only one that is needed. I can drop the rest. The key/field “content” has a lot of events an… --- ## [Can't see logs population on ELK machine from different different filebeat machines?](https://discuss.elastic.co/t/cant-see-logs-population-on-elk-machine-from-different-different-filebeat-machines/121456)
**Author:** [@ashish\_khulbe](https://discuss.elastic.co/u/ashish_khulbe)\ **Replies:** 0\ **Last updated:** [February 26, 2018, 9:02am UTC](https://discuss.elastic.co/t/cant-see-logs-population-on-elk-machine-from-different-different-filebeat-machines/121456 "2018-02-26T09:02:59Z")
Hi Team, I am new to ELK , just installed ELK and filebeat 2 days ago. I am able to see logs from 2 different filebeat clients logs in ELK machine, when working in ubuntu for syslog(default logs). But, When working w… --- ## [Custom Pattern is not working in Logstash](https://discuss.elastic.co/t/custom-pattern-is-not-working-in-logstash/121443)
**Author:** [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)\ **Replies:** 2\ **Last updated:** [February 26, 2018, 8:58am UTC](https://discuss.elastic.co/t/custom-pattern-is-not-working-in-logstash/121443 "2018-02-26T08:58:24Z")
Hi i have written custom pattern to match a String in my log file the pattern is \*REQUEST ^/WebApp\_. \*\* my logstash configuration is grok{ patterns\_dir =\> \["/home/Desktop/patterns"\] match =\> { "message" =\> '%{IP… --- ## [Syslog working. added NetFlow module and it stops](https://discuss.elastic.co/t/syslog-working-added-netflow-module-and-it-stops/121411)
**Author:** [@dpackham](https://discuss.elastic.co/u/dpackham)\ **Replies:** 1\ **Last updated:** [February 26, 2018, 7:52am UTC](https://discuss.elastic.co/t/syslog-working-added-netflow-module-and-it-stops/121411 "2018-02-26T07:52:42Z")
Like in the title. I have Syslog recieving data from multiple sources and indexing fine. I read up on the Netflow module and ran the --setup ok and it created all the needed indexes etc. when I add the following to th… --- ## [Jdbc input plugins how to show sql](https://discuss.elastic.co/t/jdbc-input-plugins-how-to-show-sql/121441)
**Author:** [@7f20ad61b3937e80aedb](https://discuss.elastic.co/u/7f20ad61b3937e80aedb)\ **Replies:** 0\ **Last updated:** [February 26, 2018, 7:11am UTC](https://discuss.elastic.co/t/jdbc-input-plugins-how-to-show-sql/121441 "2018-02-26T07:11:05Z")
jdbc input plugins how to show sql? --- ## [Log event with multi-line](https://discuss.elastic.co/t/log-event-with-multi-line/118207)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 4\ **Last updated:** [February 26, 2018, 6:10am UTC](https://discuss.elastic.co/t/log-event-with-multi-line/118207 "2018-02-26T06:10:28Z")
hi there, may i ask about multi-line pattern matching in grok filter. in input part, i use tcp input plugin then grok in filter part. eg: 2018-02-02 15:44:44,805 (null) 19 WARN : WEBFRORM: Somepage:: 342juajpobzxv4sv… --- ## [Get the Value of an array, in the output config](https://discuss.elastic.co/t/get-the-value-of-an-array-in-the-output-config/121404)
**Author:** [@Maher\_Ben\_Taleb\_Ali](https://discuss.elastic.co/u/Maher_Ben_Taleb_Ali)\ **Replies:** 3\ **Last updated:** [February 26, 2018, 12:13am UTC](https://discuss.elastic.co/t/get-the-value-of-an-array-in-the-output-config/121404 "2018-02-26T00:13:13Z")
Hi, I have the corresponding config file : input { elasticsearch { hosts =\> "localhost" index =\> "articles" add\_field =\> { "\[@metadata\]\[kafka\_topics\]" =\> \["test1", "test2"\] } } } output { stdout { cod… --- ## [SOLVED - Logstash Filter Help - Hash](https://discuss.elastic.co/t/solved-logstash-filter-help-hash/121400)
**Author:** [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\ **Replies:** 2\ **Last updated:** [February 25, 2018, 10:14pm UTC](https://discuss.elastic.co/t/solved-logstash-filter-help-hash/121400 "2018-02-25T22:14:34Z")
Team, I have an SQS message which has an embedded message that I want to parse and take out each element. I have SQS input working - but when it gets to ES it looks for the message part. Example "REALTIMEEVENT": { "ty… --- ## [HTTP\_Poller Plugin - Guidance Is Appreciated](https://discuss.elastic.co/t/http-poller-plugin-guidance-is-appreciated/121356)
**Author:** [@Criviere](https://discuss.elastic.co/u/Criviere)\ **Replies:** 2\ **Last updated:** [February 25, 2018, 8:48pm UTC](https://discuss.elastic.co/t/http-poller-plugin-guidance-is-appreciated/121356 "2018-02-25T20:48:45Z")
Hello, I'm trying to bring in data to logstash via http. My application's logs are hosted on http in this format: http://subdomain.domain.com:portnumber/path\_to\_log/ Now, my question is, how can I configure my logatas… --- ## [Logstash refusing connections from Filebeat - failed to publish events caused by: read connection reset by peer](https://discuss.elastic.co/t/logstash-refusing-connections-from-filebeat-failed-to-publish-events-caused-by-read-connection-reset-by-peer/121386)
**Author:** [@elapse2039](https://discuss.elastic.co/u/elapse2039)\ **Replies:** 0\ **Last updated:** [February 25, 2018, 2:20pm UTC](https://discuss.elastic.co/t/logstash-refusing-connections-from-filebeat-failed-to-publish-events-caused-by-read-connection-reset-by-peer/121386 "2018-02-25T14:20:38Z")
filbeat: 6.2.2 (arm), libbeat 6.2.2 logstash: 6.2.2 Filebeat after 'some time' ended up logging: 2018-02-25T13:37:42.058Z ERROR logstash/async.go:235 Failed to publish events caused by: read tcp 192.168.1.1… --- ## [Scaling vertically?](https://discuss.elastic.co/t/scaling-vertically/121360)
**Author:** [@wakinov](https://discuss.elastic.co/u/wakinov)\ **Replies:** 1\ **Last updated:** [February 25, 2018, 11:07am UTC](https://discuss.elastic.co/t/scaling-vertically/121360 "2018-02-25T11:07:15Z")
Version 2,4 in case it matters. The documentation really only mentions scaling horizontally. I'm in a position where I can get access to very "large" machines with 12-cores and 64GB+ of RAM. I'm just wondering if there … --- ## [Logstash DNS Filter - Enrichment security example](https://discuss.elastic.co/t/logstash-dns-filter-enrichment-security-example/121040)
**Author:** [@Raj\_Kumar](https://discuss.elastic.co/u/Raj_Kumar)\ **Replies:** 4\ **Last updated:** [February 25, 2018, 9:30am UTC](https://discuss.elastic.co/t/logstash-dns-filter-enrichment-security-example/121040 "2018-02-25T09:30:05Z")
Hi All, I was trying to execute the Data enrichment logstash -IP addresses exist in an external DNSBL (DNS Blocklist)? filter { grok { match =\> { "message" =\> "%{WORD:addr1}.%{WORD:addr2}.%{WORD:addr3}.%{WORD… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=812) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=814)