# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=542 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 543 --- ## [Does Logstash (7.0.1) support gzip file input?](https://discuss.elastic.co/t/does-logstash-7-0-1-support-gzip-file-input/181862)
**Author:** [@nobound](https://discuss.elastic.co/u/nobound)\ **Replies:** 6\ **Last updated:** [May 23, 2019, 5:49pm UTC](https://discuss.elastic.co/t/does-logstash-7-0-1-support-gzip-file-input/181862 "2019-05-23T17:49:01Z")
Hi all, It seems there are issues regarding gzip file support, but I can't really find any post that will give me the answer. I have tried to use gzip\_lines codec. It does not work. input { file { path =\> "… --- ## [How to properly parse multi line json?](https://discuss.elastic.co/t/how-to-properly-parse-multi-line-json/182464)
**Author:** [@ck1](https://discuss.elastic.co/u/ck1)\ **Replies:** 5\ **Last updated:** [May 23, 2019, 5:45pm UTC](https://discuss.elastic.co/t/how-to-properly-parse-multi-line-json/182464 "2019-05-23T17:45:21Z")
Hello all, I am having trouble parsing the following json input in tpc { "color": { "pixel": "152", "shade": "grey" }, "color": { "pixel": "157", "shade": "dark" } } Result I want : 2 documents into Elas… --- ## [Grok Debugger - hard to find identification string for my difficult example](https://discuss.elastic.co/t/grok-debugger-hard-to-find-identification-string-for-my-difficult-example/182016)
**Author:** [@Datakids](https://discuss.elastic.co/u/Datakids)\ **Replies:** 10\ **Last updated:** [May 23, 2019, 3:38pm UTC](https://discuss.elastic.co/t/grok-debugger-hard-to-find-identification-string-for-my-difficult-example/182016 "2019-05-23T15:38:32Z")
{Result=\[{Index=1, PropertyName=, ResourceA1=here, ResourceA2=heretoo, ResourceA3=hereagain, ResourceA4=herewego, ResourceKey=transactions.ui.rfc.messages.label.defaultMessage, ServiceKey='6457A5674GF01ED98EAC1Z99349097… --- ## [XML filter, doc\_as\_upsert and pipeline.workers](https://discuss.elastic.co/t/xml-filter-doc-as-upsert-and-pipeline-workers/182479)
**Author:** [@MaryFu](https://discuss.elastic.co/u/MaryFu)\ **Replies:** 0\ **Last updated:** [May 23, 2019, 3:27pm UTC](https://discuss.elastic.co/t/xml-filter-doc-as-upsert-and-pipeline-workers/182479 "2019-05-23T15:27:02Z")
Hello, My colleague @Anabella\_Cristaldi and me have an XML like the following structure \ \feline\ \meow\ \ \ … --- ## [No hits after indexing Mysql](https://discuss.elastic.co/t/no-hits-after-indexing-mysql/182463)
**Author:** [@Gonzalo\_Garcia\_Roces](https://discuss.elastic.co/u/Gonzalo_Garcia_Roces)\ **Replies:** 3\ **Last updated:** [May 23, 2019, 3:05pm UTC](https://discuss.elastic.co/t/no-hits-after-indexing-mysql/182463 "2019-05-23T15:05:06Z")
Hello I'm fairly new to ES and I'm not quite sure what is not working here. I have indexed a Mysql table with the following logstash config. input { jdbc { jdbc\_connection\_string =\> "jdbc:mysql://127.0.0.1:3306/?use… --- ## [Postgresql toLogstash does not work](https://discuss.elastic.co/t/postgresql-tologstash-does-not-work/182474)
**Author:** [@victorMagalhaesPache](https://discuss.elastic.co/u/victorMagalhaesPache)\ **Replies:** 0\ **Last updated:** [May 23, 2019, 2:57pm UTC](https://discuss.elastic.co/t/postgresql-tologstash-does-not-work/182474 "2019-05-23T14:57:30Z")
Hello, I'm new to the elk stack. Operating system: centos ELK version 6.7 I'm trying to migrate some data from postgresql to the elastic search using logstash, but it does not work! I have the file called: /etc/logs… --- ## [LibreNMS/RRD files to Elastic?](https://discuss.elastic.co/t/librenms-rrd-files-to-elastic/182459)
**Author:** [@Sean\_Lambert](https://discuss.elastic.co/u/Sean_Lambert)\ **Replies:** 0\ **Last updated:** [May 23, 2019, 1:55pm UTC](https://discuss.elastic.co/t/librenms-rrd-files-to-elastic/182459 "2019-05-23T13:55:46Z")
There was one topic here a while back that went nowhere. I'm trying to move data from LibreNMS RRD files to Elastic. Has anyone done this? --- ## [How to use grok filter on XML field](https://discuss.elastic.co/t/how-to-use-grok-filter-on-xml-field/181718)
**Author:** [@Yoav\_Ben\_Moha](https://discuss.elastic.co/u/Yoav_Ben_Moha)\ **Replies:** 5\ **Last updated:** [May 23, 2019, 1:34pm UTC](https://discuss.elastic.co/t/how-to-use-grok-filter-on-xml-field/181718 "2019-05-23T13:34:33Z")
Hi, I have a csv files . One of the fields has an xml strutcher. I need to return the value that exists between the first apperence of : In this case i need to return : 0-GOLD \\\0-G… --- ## [Something is wrong with my confifg file - Oracle DB Connection](https://discuss.elastic.co/t/something-is-wrong-with-my-confifg-file-oracle-db-connection/182436)
**Author:** [@boyaronur](https://discuss.elastic.co/u/boyaronur)\ **Replies:** 3\ **Last updated:** [May 23, 2019, 1:21pm UTC](https://discuss.elastic.co/t/something-is-wrong-with-my-confifg-file-oracle-db-connection/182436 "2019-05-23T13:21:41Z")
Hello, I try to connect Oracle DB. I am using config file for that reason. My config is input { jdbc { jdbc\_connection\_string =\> "jdbc:oracle:thin:@//00.000.00.0:0000/PRELIVE1" jdbc\_user =\> "\*\*\*\*\*" jdbc\_pas… --- ## [How to dynamic parse fieldname and replace?](https://discuss.elastic.co/t/how-to-dynamic-parse-fieldname-and-replace/179858)
**Author:** [@ck1](https://discuss.elastic.co/u/ck1)\ **Replies:** 2\ **Last updated:** [May 23, 2019, 1:13pm UTC](https://discuss.elastic.co/t/how-to-dynamic-parse-fieldname-and-replace/179858 "2019-05-23T13:13:16Z")
Hello, I am trying data that looks like this unwanted.field1 unwanted.field2 categories.colors.blue.t.type First I need to remove every unwanted field, then I want to change categories.colors to only blue.t.type. I… --- ## [Cannot combine two indexes to one using logstash](https://discuss.elastic.co/t/cannot-combine-two-indexes-to-one-using-logstash/182083)
**Author:** [@aldol](https://discuss.elastic.co/u/aldol)\ **Replies:** 6\ **Last updated:** [May 23, 2019, 12:44pm UTC](https://discuss.elastic.co/t/cannot-combine-two-indexes-to-one-using-logstash/182083 "2019-05-23T12:44:33Z")
I have the following two indexes: PUT twitter1/doc/1 { "name" : "Mike", "c\_num" : 1, "tags" : "in1" } PUT twitter2/doc/1 { "resume" : "Mike blah, blah, blah", "c\_num" : 1, "tags" : "in2" } I would like to ha… --- ## [Logstash Elasticsearch Output into wrong index](https://discuss.elastic.co/t/logstash-elasticsearch-output-into-wrong-index/182417)
**Author:** [@Spitza\_Spitza](https://discuss.elastic.co/u/Spitza_Spitza)\ **Replies:** 0\ **Last updated:** [May 23, 2019, 11:48am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-into-wrong-index/182417 "2019-05-23T11:48:21Z")
Hi, I upgraded from 6.8 to 7.1 yesterday and it works well, except that I do not write to the default index "logstash-%{+YYYY.MM.dd}" anymore, instead everything goes into an index just called logstash with no date atta… --- ## [JMS input getting an error JMS Consumer Died](https://discuss.elastic.co/t/jms-input-getting-an-error-jms-consumer-died/181877)
**Author:** [@bigster](https://discuss.elastic.co/u/bigster)\ **Replies:** 8\ **Last updated:** [May 23, 2019, 11:26am UTC](https://discuss.elastic.co/t/jms-input-getting-an-error-jms-consumer-died/181877 "2019-05-23T11:26:03Z")
Hi all, I've got a JMS input in my logstash config getting messages from a IBM MQ and i'm getting this error: JMS Consumer died {:exception=\>com.ibm.msg.client.jms.DetailedMessageEOFException: JMSWMQ2002: Failed to get… --- ## [Parsing grok](https://discuss.elastic.co/t/parsing-grok/182248)
**Author:** [@angie](https://discuss.elastic.co/u/angie)\ **Replies:** 4\ **Last updated:** [May 23, 2019, 10:55am UTC](https://discuss.elastic.co/t/parsing-grok/182248 "2019-05-23T10:55:18Z")
I'm using logstash 6.6.0 and I would like to convert this value: "\[34mems\_1 |\[0m" for this: "ems". I look forward to hearing from you --- ## [Unable to push data from Mysql to elastic search through LogStash](https://discuss.elastic.co/t/unable-to-push-data-from-mysql-to-elastic-search-through-logstash/182403)
**Author:** [@Abhinav\_Manoj](https://discuss.elastic.co/u/Abhinav_Manoj)\ **Replies:** 0\ **Last updated:** [May 23, 2019, 10:39am UTC](https://discuss.elastic.co/t/unable-to-push-data-from-mysql-to-elastic-search-through-logstash/182403 "2019-05-23T10:39:35Z")
Hello All I'm unable to push data from Mysql to Elastic search. I'm using Logstash to push data to ElasticSearch My config file is given below input { jdbc { jdbc\_driver\_library =\> "D:/ELK/mysql-connector-java-8.0.16… --- ## [Prevent Logstash Service logging to /var/log/messages](https://discuss.elastic.co/t/prevent-logstash-service-logging-to-var-log-messages/181836)
**Author:** [@anshul27jain](https://discuss.elastic.co/u/anshul27jain)\ **Replies:** 2\ **Last updated:** [May 23, 2019, 10:35am UTC](https://discuss.elastic.co/t/prevent-logstash-service-logging-to-var-log-messages/181836 "2019-05-23T10:35:56Z")
Anyone please guide me in preventing Logstash logging to /var/log/messages? I have configured Syslog Input module on my server & sending those logs to Elasticsearch. Along with that all log related to logsatsh is also g… --- ## [Problem with the KV plugin that prevents logstash to process logs](https://discuss.elastic.co/t/problem-with-the-kv-plugin-that-prevents-logstash-to-process-logs/182377)
**Author:** [@maxjoe](https://discuss.elastic.co/u/maxjoe)\ **Replies:** 0\ **Last updated:** [May 23, 2019, 8:41am UTC](https://discuss.elastic.co/t/problem-with-the-kv-plugin-that-prevents-logstash-to-process-logs/182377 "2019-05-23T08:41:09Z")
Hi everyone, I'm new to ELK stack and I've been stuck for two days with a problem that I think is due to kv plugin. I need to ingest log from our Fortimail. This is one sample of my Fortimail log : \<42\>date=2019-05-22… --- ## [Add\_field with a name generated from message data](https://discuss.elastic.co/t/add-field-with-a-name-generated-from-message-data/182375)
**Author:** [@jrihds](https://discuss.elastic.co/u/jrihds)\ **Replies:** 1\ **Last updated:** [May 23, 2019, 9:19am UTC](https://discuss.elastic.co/t/add-field-with-a-name-generated-from-message-data/182375 "2019-05-23T09:19:06Z")
Hello, Is it possible to add a new field whose name is based on data from the message? e.g. add\_field =\> { "%{\[parsed-json\]\[0\]\[key\]}" =\> "%{\[parsed-json\]\[0\]\[value\]}" } I am currently getting an 'Invalid FieldReferen… --- ## [Logstash is not working](https://discuss.elastic.co/t/logstash-is-not-working/182378)
**Author:** [@kmohd](https://discuss.elastic.co/u/kmohd)\ **Replies:** 1\ **Last updated:** [May 23, 2019, 8:54am UTC](https://discuss.elastic.co/t/logstash-is-not-working/182378 "2019-05-23T08:54:55Z")
Logstash has an issue as per the below, i hope anyone have an idea how to solve it. C:\\Users\\Administrator\\Desktop\\logstash-7.1.0\\logstash-7.1.0\\bin\>logstash -f logstash.conf Sending Logstash logs to C:/Users/Administr… --- ## [Logs are overwritten in the specified index under the same \_id](https://discuss.elastic.co/t/logs-are-overwritten-in-the-specified-index-under-the-same-id/182208)
**Author:** [@madhanbaskar](https://discuss.elastic.co/u/madhanbaskar)\ **Replies:** 6\ **Last updated:** [May 23, 2019, 7:38am UTC](https://discuss.elastic.co/t/logs-are-overwritten-in-the-specified-index-under-the-same-id/182208 "2019-05-23T07:38:17Z")
Hi There, I'm using Logstash - 6.5.1 and elasticsearch - 6.5.1. Below is my Filebeat.yml filebeat.prospectors: type: log paths: var/log/message fields: type: apache\_access tags: \["ApacheAccessLogs"\] type: … --- ## [Memcached performance](https://discuss.elastic.co/t/memcached-performance/182237)
**Author:** [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 1:37pm UTC](https://discuss.elastic.co/t/memcached-performance/182237 "2019-05-22T13:37:48Z")
Hello. As I have not encountered any post related to the memcached so far. I have a very powerful machine, 512 GB of ram, RAID0 of SSD, 80 CPU cores of Xeon v4.... so the hardware is not the limit, potentially my misco… --- ## [Separate qt2612532-162](https://discuss.elastic.co/t/separate-qt2612532-162/181188)
**Author:** [@Pooja\_Chowdhary](https://discuss.elastic.co/u/Pooja_Chowdhary)\ **Replies:** 2\ **Last updated:** [May 23, 2019, 6:37am UTC](https://discuss.elastic.co/t/separate-qt2612532-162/181188 "2019-05-23T06:37:22Z")
I want to filter qt2612532-162 using grok I want the output as instance\_name : qt2612532 thread\_id : 162 Help is appreciated! --- ## [Test log stash performance](https://discuss.elastic.co/t/test-log-stash-performance/182334)
**Author:** [@Yungyoung\_Ok](https://discuss.elastic.co/u/Yungyoung_Ok)\ **Replies:** 0\ **Last updated:** [May 23, 2019, 2:00am UTC](https://discuss.elastic.co/t/test-log-stash-performance/182334 "2019-05-23T02:00:53Z")
I want to test logstash performance. The items I would like to test are: The time at which the data in the logstash filter is processed Number of data exported from logstash to Elasticsearch Number of data received fr… --- ## [Translate filter - substitute not working / bug](https://discuss.elastic.co/t/translate-filter-substitute-not-working-bug/182320)
**Author:** [@nufje](https://discuss.elastic.co/u/nufje)\ **Replies:** 2\ **Last updated:** [May 22, 2019, 9:26pm UTC](https://discuss.elastic.co/t/translate-filter-substitute-not-working-bug/182320 "2019-05-22T21:26:12Z")
Hello i'am using the logstash translate filter, but substitution on same source field is not working. For example i want to change color in product\_name\_search from "Iphone black" to "iphone zwart" When i use this snip… --- ## [Configure logstash to input JSON file into elasticsearch](https://discuss.elastic.co/t/configure-logstash-to-input-json-file-into-elasticsearch/182313)
**Author:** [@esportuga](https://discuss.elastic.co/u/esportuga)\ **Replies:** 3\ **Last updated:** [May 22, 2019, 8:27pm UTC](https://discuss.elastic.co/t/configure-logstash-to-input-json-file-into-elasticsearch/182313 "2019-05-22T20:27:49Z")
Hello, I'm accessing tweets with Tweepy and generating complete .JSON files for each Tweet. I set up my logstash to read each of these files and index them in Elasticsearch See below my logstash configuration file: i… --- ## [Logstash-filter-cef instead of codec](https://discuss.elastic.co/t/logstash-filter-cef-instead-of-codec/182302)
**Author:** [@paniccontrol](https://discuss.elastic.co/u/paniccontrol)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 6:40pm UTC](https://discuss.elastic.co/t/logstash-filter-cef-instead-of-codec/182302 "2019-05-22T18:40:27Z")
I'm receiving several type of log over syslog, some of them are customed, json, cef,.. other syslog. I've noticed there is no built in cef filter, like the json one. I found this one: Why it's not included into log… --- ## [Logstash-input-jdbc for Oracle or Postgres with VPD (Virtual Private Database)](https://discuss.elastic.co/t/logstash-input-jdbc-for-oracle-or-postgres-with-vpd-virtual-private-database/182286)
**Author:** [@hive](https://discuss.elastic.co/u/hive)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 4:38pm UTC](https://discuss.elastic.co/t/logstash-input-jdbc-for-oracle-or-postgres-with-vpd-virtual-private-database/182286 "2019-05-22T16:38:55Z")
Hi, I am trying to use "jdbc" with logstash to move data from oracle to elasticcache. My Oracle DB uses VPD. So I have set VPD before I can execute any SQL statement configured in logstash config file. Does logstash a… --- ## [Logstash Config Files (Am I doing something wrong)](https://discuss.elastic.co/t/logstash-config-files-am-i-doing-something-wrong/182025)
**Author:** [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Replies:** 7\ **Last updated:** [May 22, 2019, 3:31pm UTC](https://discuss.elastic.co/t/logstash-config-files-am-i-doing-something-wrong/182025 "2019-05-22T15:31:20Z")
Hi everyone, I have been working with the elastic stack for a short while. I have been following a udemy class about setting all of it up. Right now I am facing a really confusing problem with the Logstash setup. I am wo… --- ## [Sugesstions on logstash ingesting data from other database containers in docker](https://discuss.elastic.co/t/sugesstions-on-logstash-ingesting-data-from-other-database-containers-in-docker/182257)
**Author:** [@decodeit](https://discuss.elastic.co/u/decodeit)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 2:49pm UTC](https://discuss.elastic.co/t/sugesstions-on-logstash-ingesting-data-from-other-database-containers-in-docker/182257 "2019-05-22T14:49:10Z")
I have elasticsearch, kibana and logstash containers on docker, ingesting data from the web service. I want however to be able to ingest data from the database. How can it be done. Can someone point me towards right ar… --- ## [Logstash ruby exception](https://discuss.elastic.co/t/logstash-ruby-exception/182247)
**Author:** [@zcx](https://discuss.elastic.co/u/zcx)\ **Replies:** 0\ **Last updated:** [May 22, 2019, 2:13pm UTC](https://discuss.elastic.co/t/logstash-ruby-exception/182247 "2019-05-22T14:13:58Z")
when Embed ruby scripts in the filter and an error occur as follows: \[ERROR\]\[logstash.filters.ruby \] Ruby exception occurred: undefined method \`filter\_method' for #LogStash::Filters::Ruby:0x4cd94f01 Did you mean? f… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=541) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=543)