# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=617 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 618 --- ## [How to use IF ELSE conduction in grok pattern in logstash](https://discuss.elastic.co/t/how-to-use-if-else-conduction-in-grok-pattern-in-logstash/164458)
**Author:** [@Vinit\_Kumar](https://discuss.elastic.co/u/Vinit_Kumar)\ **Replies:** 4\ **Last updated:** [January 16, 2019, 8:12pm UTC](https://discuss.elastic.co/t/how-to-use-if-else-conduction-in-grok-pattern-in-logstash/164458 "2019-01-16T20:12:44Z")
I have web and API log combined and I want to save it separately in elasticsearch. So I want to write one pattern if the request is for API then if past should execute, the request is web then else part of the log should… --- ## [Logstash Output to ES hosted on AWS, authentication not working](https://discuss.elastic.co/t/logstash-output-to-es-hosted-on-aws-authentication-not-working/164504)
**Author:** [@Smitto](https://discuss.elastic.co/u/Smitto)\ **Replies:** 2\ **Last updated:** [January 16, 2019, 7:11pm UTC](https://discuss.elastic.co/t/logstash-output-to-es-hosted-on-aws-authentication-not-working/164504 "2019-01-16T19:11:06Z")
I've been trying to send my Logstash output to an Elasticsearch instance being hosted on AWS. Following the instructions here I've set up my conf file's output as the following: output { elasticsearch { host… --- ## [Expected one of #, \\", ', -, \[, / at line 77, column 52 (byte 2039)](https://discuss.elastic.co/t/expected-one-of-at-line-77-column-52-byte-2039/164505)
**Author:** [@exocore123](https://discuss.elastic.co/u/exocore123)\ **Replies:** 5\ **Last updated:** [January 16, 2019, 6:53pm UTC](https://discuss.elastic.co/t/expected-one-of-at-line-77-column-52-byte-2039/164505 "2019-01-16T18:53:47Z")
Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, ", ', -, \[, / at line 77, column 52 (byte 2039) after filte… --- ## [XML plugin + split + ruby code to generate dynamic fields](https://discuss.elastic.co/t/xml-plugin-split-ruby-code-to-generate-dynamic-fields/164491)
**Author:** [@Anabella\_Cristaldi](https://discuss.elastic.co/u/Anabella_Cristaldi)\ **Replies:** 3\ **Last updated:** [January 16, 2019, 4:38pm UTC](https://discuss.elastic.co/t/xml-plugin-split-ruby-code-to-generate-dynamic-fields/164491 "2019-01-16T16:38:43Z")
Hi, I have a compplex XML with information about phone network cells we need to parse. I managed to ingest the data using the xml and the split filter in order to have the desired information in separate documents into … --- ## [Update/Add multiple records in elastic using logstash](https://discuss.elastic.co/t/update-add-multiple-records-in-elastic-using-logstash/164316)
**Author:** [@amit.patra](https://discuss.elastic.co/u/amit.patra)\ **Replies:** 2\ **Last updated:** [January 16, 2019, 4:20pm UTC](https://discuss.elastic.co/t/update-add-multiple-records-in-elastic-using-logstash/164316 "2019-01-16T16:20:27Z")
We have a requirement, We have multiple json data like below : { "\_id": "1", "\_index": "vnfsdb-20181214", "\_type": "l4", "\_source": { "@timestamp": "2018-10-01", "Flagvalue": "1", "Terminaltype": "T2", "ObjectF… --- ## [Best way to remove CDATA Infos in xml](https://discuss.elastic.co/t/best-way-to-remove-cdata-infos-in-xml/164470)
**Author:** [@humalog](https://discuss.elastic.co/u/humalog)\ **Replies:** 1\ **Last updated:** [January 16, 2019, 3:55pm UTC](https://discuss.elastic.co/t/best-way-to-remove-cdata-infos-in-xml/164470 "2019-01-16T15:55:20Z")
Hello, I have an XML with several CDATA Info in my values. I don't want them there. What is the best way to delete this infos? I tried with using filter { mutate { gsub }} but is there an easier solution? Here is an e… --- ## [How to date match "January 16th 2019, 07:00:09.257"?](https://discuss.elastic.co/t/how-to-date-match-january-16th-2019-0709-257/164463)
**Author:** [@JimP](https://discuss.elastic.co/u/JimP)\ **Replies:** 3\ **Last updated:** [January 16, 2019, 2:26pm UTC](https://discuss.elastic.co/t/how-to-date-match-january-16th-2019-0709-257/164463 "2019-01-16T14:26:45Z")
This should be easy, but for some reason I am unable to do this. Can someone help, please? January 16th 2019, 07:00:09.257 Logstash 6.3.2 --- ## [Logstash - json filter removes backslash](https://discuss.elastic.co/t/logstash-json-filter-removes-backslash/164285)
**Author:** [@Francisca\_Lima](https://discuss.elastic.co/u/Francisca_Lima)\ **Replies:** 4\ **Last updated:** [January 16, 2019, 2:08pm UTC](https://discuss.elastic.co/t/logstash-json-filter-removes-backslash/164285 "2019-01-16T14:08:57Z")
Hello, When I am doing a json filter in my logstash file, in fields where I have two backlashes ("\\") one of them is removed. Why this is happening? Besides replacing \\ to \\, what can I do? Thank you. --- ## [Use Field as Fingerprint Instead of Generating One](https://discuss.elastic.co/t/use-field-as-fingerprint-instead-of-generating-one/164465)
**Author:** [@Aralex](https://discuss.elastic.co/u/Aralex)\ **Replies:** 2\ **Last updated:** [January 16, 2019, 1:41pm UTC](https://discuss.elastic.co/t/use-field-as-fingerprint-instead-of-generating-one/164465 "2019-01-16T13:41:30Z")
Hi, since I have a field with a unique value for each of my csv entries, how can I use that field value as a fingerprint instead of generating one, to avoid duplicates? I experimented with the csv filter for while, and… --- ## [Compose logstash configuration from multiple files (import config fragments)](https://discuss.elastic.co/t/compose-logstash-configuration-from-multiple-files-import-config-fragments/164476)
**Author:** [@Presence](https://discuss.elastic.co/u/Presence)\ **Replies:** 1\ **Last updated:** [January 16, 2019, 1:39pm UTC](https://discuss.elastic.co/t/compose-logstash-configuration-from-multiple-files-import-config-fragments/164476 "2019-01-16T13:39:57Z")
I have a few pipelines that contain common configuration (filters, mainly). Is there a logstash mechanism whereby I can lift the common config out into a separate file and have this imported into the primary config file … --- ## [Pipeline stats API: in/out/filtered](https://discuss.elastic.co/t/pipeline-stats-api-in-out-filtered/163742)
**Author:** [@jmb](https://discuss.elastic.co/u/jmb)\ **Replies:** 3\ **Last updated:** [January 16, 2019, 1:16pm UTC](https://discuss.elastic.co/t/pipeline-stats-api-in-out-filtered/163742 "2019-01-16T13:16:40Z")
What do the in/out/filtered values in the events section of the pipeline stats API actually represent? I had understood them to mean in: total events entering the pipeline (sum of input plugins production) out: tota… --- ## [Winlogbeat to SysLog Server](https://discuss.elastic.co/t/winlogbeat-to-syslog-server/164474)
**Author:** [@B4nd00](https://discuss.elastic.co/u/B4nd00)\ **Replies:** 0\ **Last updated:** [January 16, 2019, 1:10pm UTC](https://discuss.elastic.co/t/winlogbeat-to-syslog-server/164474 "2019-01-16T13:10:03Z")
Hi, i want to store my Data from my Winlogbeat on my Syslog Server. I use the SysLog Server from my Qnap I am running Logstash on a RedHat VM on my Qnap. The Syslog Output Plugin is already installed. I am now sendi… --- ## [How to Increase Performance Speed of Logstash for using MongoDB](https://discuss.elastic.co/t/how-to-increase-performance-speed-of-logstash-for-using-mongodb/163698)
**Author:** [@urcreations](https://discuss.elastic.co/u/urcreations)\ **Replies:** 12\ **Last updated:** [January 16, 2019, 11:09am UTC](https://discuss.elastic.co/t/how-to-increase-performance-speed-of-logstash-for-using-mongodb/163698 "2019-01-16T11:09:33Z")
Hi... I'm using logstash 5.6.13. I'm trying to export data from a CSV file (Size 975mb and contain 1 crore fields) to Mongodb using logstash. My system configuration in 16GB RAM and 320GB HDD. System have core i7 proce… --- ## [I get XML elements back when using xpath](https://discuss.elastic.co/t/i-get-xml-elements-back-when-using-xpath/164423)
**Author:** [@humalog](https://discuss.elastic.co/u/humalog)\ **Replies:** 1\ **Last updated:** [January 16, 2019, 10:07am UTC](https://discuss.elastic.co/t/i-get-xml-elements-back-when-using-xpath/164423 "2019-01-16T10:07:34Z")
Hello, when using XPATH to parse my XML file, i get the correct field but in the content I also get the element name back, what I dont want. What I need to change? My XML looks like this: \ \ **Author:** [@chaire](https://discuss.elastic.co/u/chaire)\ **Replies:** 3\ **Last updated:** [January 16, 2019, 6:09am UTC](https://discuss.elastic.co/t/join-multiple-input-with-different-input-types/164123 "2019-01-16T06:09:08Z") I'm using Logstash along with Filebeat, Elasticsearch, and Kibana. I have 2 inputs to join, one is a filelog (Apache file) and another one is database file (sql). I can't find any documentation that explains how to use … --- ## [Dynamic Parsing with Grok](https://discuss.elastic.co/t/dynamic-parsing-with-grok/164295)
**Author:** [@HugoSousa](https://discuss.elastic.co/u/HugoSousa)\ **Replies:** 5\ **Last updated:** [January 16, 2019, 9:13am UTC](https://discuss.elastic.co/t/dynamic-parsing-with-grok/164295 "2019-01-16T09:13:30Z")
Hi I have the following log file sample: #Version: 1.0 #Fields: s-dns date time x-duration c-ip c-port c-vx-zone ac1.lg9ams1d1.cdn 2019-01-14 05:00:01 0.001 172.30.116.146 46684 … --- ## [Archive index to gzip - cant decompressed](https://discuss.elastic.co/t/archive-index-to-gzip-cant-decompressed/160108)
**Author:** [@bmatoki](https://discuss.elastic.co/u/bmatoki)\ **Replies:** 10\ **Last updated:** [January 16, 2019, 9:02am UTC](https://discuss.elastic.co/t/archive-index-to-gzip-cant-decompressed/160108 "2019-01-16T09:02:06Z")
Hey all. im trying to archive logs every 1hour to gzip - its working and saving every 1 hour to gzip. while im trying to decompressed it via node js \\ linux \\ gunzip \\ cygnwin - got an error " Gzip outputting invalid… --- ## [Logstash kv filter extract json array issue](https://discuss.elastic.co/t/logstash-kv-filter-extract-json-array-issue/163835)
**Author:** [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\ **Replies:** 6\ **Last updated:** [January 16, 2019, 8:33am UTC](https://discuss.elastic.co/t/logstash-kv-filter-extract-json-array-issue/163835 "2019-01-16T08:33:05Z")
Hi, I have below json file, and the json object contains array. I want to use kv filter ( not json filter) to extract corresponding kv paris, { "DP": "dbpool1", "CMID": "TALSCMK", "CMN": "HuaWei", "UID": "lokamoto1… --- ## [Nested json as a string field](https://discuss.elastic.co/t/nested-json-as-a-string-field/164257)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 2\ **Last updated:** [January 16, 2019, 7:01am UTC](https://discuss.elastic.co/t/nested-json-as-a-string-field/164257 "2019-01-16T07:01:59Z")
hi i use json filter plugin in my logstash config. here if i face nested json, i got \_jsonparsefailure tag. my target is that i just want nested json as string. not another json. eg: { "agent": "Mozilla/5.0 (compat… --- ## [Logstash sybase jdbc error](https://discuss.elastic.co/t/logstash-sybase-jdbc-error/164410)
**Author:** [@Bhupendra\_Bagde](https://discuss.elastic.co/u/Bhupendra_Bagde)\ **Replies:** 0\ **Last updated:** [January 16, 2019, 6:24am UTC](https://discuss.elastic.co/t/logstash-sybase-jdbc-error/164410 "2019-01-16T06:24:27Z")
After I configure the logstash.conf input{ jdbc { jdbc\_driver\_library =\> "/data/logstash/vendor/jar/jdbc/jconn3.jar" jdbc\_driver\_class =\> "com.sybase.jdbc3.jdbc.SybDriver" jdbc\_connection\_string =\> "jdbc:sybase:Tds:… --- ## [Logstash uses the time in the log to resolve to @timestamp, the parsing format in the match is disassembled, and a matching exception occurs!](https://discuss.elastic.co/t/logstash-uses-the-time-in-the-log-to-resolve-to-timestamp-the-parsing-format-in-the-match-is-disassembled-and-a-matching-exception-occurs/164305)
**Author:** [@sun\_changlong](https://discuss.elastic.co/u/sun_changlong)\ **Replies:** 2\ **Last updated:** [January 16, 2019, 1:47am UTC](https://discuss.elastic.co/t/logstash-uses-the-time-in-the-log-to-resolve-to-timestamp-the-parsing-format-in-the-match-is-disassembled-and-a-matching-exception-occurs/164305 "2019-01-16T01:47:57Z")
The format of the time field in the log: version=1.0 time="2019-01-15 18:20:05" dev="WAF01.PUB.BEIJING-B" pri="0" Parsing in match: date { #"time" =\> "2019-01-15 18:20:05" match =\> \["time", "yyyy-MM-dd HH:mm:ss"\] … --- ## [Splitting a huge field into multiple fields](https://discuss.elastic.co/t/splitting-a-huge-field-into-multiple-fields/163433)
**Author:** [@mayanks8](https://discuss.elastic.co/u/mayanks8)\ **Replies:** 6\ **Last updated:** [January 15, 2019, 10:16pm UTC](https://discuss.elastic.co/t/splitting-a-huge-field-into-multiple-fields/163433 "2019-01-15T22:16:00Z")
Hi All, I am getting data in Json format in my eleasticsearch. but one one the fields called "messages" has over 30K lines, I need to split lines into the "message" based on the format. i have a special character where… --- ## [Increase speed processing with more logstash consumer](https://discuss.elastic.co/t/increase-speed-processing-with-more-logstash-consumer/164380)
**Author:** [@elk2](https://discuss.elastic.co/u/elk2)\ **Replies:** 0\ **Last updated:** [January 15, 2019, 10:09pm UTC](https://discuss.elastic.co/t/increase-speed-processing-with-more-logstash-consumer/164380 "2019-01-15T22:09:44Z")
Someone can explicate how can I sizing correctly the number of logstash consumer if I want to process a single kafka topic with 3 partitions? --- ## [How to use the jdbc\_static plugin?](https://discuss.elastic.co/t/how-to-use-the-jdbc-static-plugin/164355)
**Author:** [@roblopes](https://discuss.elastic.co/u/roblopes)\ **Replies:** 1\ **Last updated:** [January 15, 2019, 9:13pm UTC](https://discuss.elastic.co/t/how-to-use-the-jdbc-static-plugin/164355 "2019-01-15T21:13:52Z")
Hi, I did some tests with the jdbc\_static plugin, and it is working fine, however it is not 100% clear to me how to use it in my pipelines. Adding the jdbc\_static block with loaders, local\_db\_objects and local\_lookups … --- ## [Variable on Logstash](https://discuss.elastic.co/t/variable-on-logstash/164356)
**Author:** [@rcd1189](https://discuss.elastic.co/u/rcd1189)\ **Replies:** 2\ **Last updated:** [January 15, 2019, 8:57pm UTC](https://discuss.elastic.co/t/variable-on-logstash/164356 "2019-01-15T20:57:46Z")
Hello everyone! I'm creating a Centralized Logs with Filebeat, Logstash, and Elasticsearch. My Beat and Elasticsearch work well. I'm getting a problem in my logstash on output. logstash.conf input { hits port =\> 50… --- ## [Unable to see my index on kibana](https://discuss.elastic.co/t/unable-to-see-my-index-on-kibana/164359)
**Author:** [@mailtosuresh](https://discuss.elastic.co/u/mailtosuresh)\ **Replies:** 0\ **Last updated:** [January 15, 2019, 7:01pm UTC](https://discuss.elastic.co/t/unable-to-see-my-index-on-kibana/164359 "2019-01-15T19:01:28Z")
hi Dear I load some sample datasets in csv format ,,,using logsash ....iam getting log message is " \[2019-01-15T10:44:42,242\]\[WARN \]\[logstash.config.source.multilocal\] Ignoring the 'pipelines.yml' file because modules… --- ## [Logstash Drop Lines](https://discuss.elastic.co/t/logstash-drop-lines/163273)
**Author:** [@rcd1189](https://discuss.elastic.co/u/rcd1189)\ **Replies:** 2\ **Last updated:** [January 15, 2019, 6:48pm UTC](https://discuss.elastic.co/t/logstash-drop-lines/163273 "2019-01-15T18:48:04Z")
Hi All. I Have a Weblogic log how below: Jan 7, 2019 5:29:13 PM com.clarify.common.filter.SearchFilterBaseWB Start doLoad() from SearchFilterBaseWB INFO: ENTRY Warning (Not critical): Query will be done against view.… --- ## [java.lang.IllegalArgumentException: Illegal pattern component: t](https://discuss.elastic.co/t/java-lang-illegalargumentexception-illegal-pattern-component-t/164348)
**Author:** [@Marcos\_OS](https://discuss.elastic.co/u/Marcos_OS)\ **Replies:** 2\ **Last updated:** [January 15, 2019, 6:06pm UTC](https://discuss.elastic.co/t/java-lang-illegalargumentexception-illegal-pattern-component-t/164348 "2019-01-15T18:06:05Z")
EN: I have a problem with Logstash and would like help in trying to resolve. PT: Eu tenho um problema com Logstash e gostaria de ajuda para tentar resolver. EN: I'm trying to extract data from a relational database and… --- ## [How to write custom gork for logstash](https://discuss.elastic.co/t/how-to-write-custom-gork-for-logstash/163858)
**Author:** [@Rocky\_RK](https://discuss.elastic.co/u/Rocky_RK)\ **Replies:** 8\ **Last updated:** [January 15, 2019, 5:37pm UTC](https://discuss.elastic.co/t/how-to-write-custom-gork-for-logstash/163858 "2019-01-15T17:37:28Z")
I'm trying to test the some custom log filter for logstash but somehow i'm not able to get it, I googled and looked over many examples but I am not able to create a one I want. Below is my log patterns: testhost-in2,19… --- ## [Logstash Kafka Output: UNKNOWN\_TOPIC\_OR\_PARTITION blocks all other logs](https://discuss.elastic.co/t/logstash-kafka-output-unknown-topic-or-partition-blocks-all-other-logs/164344)
**Author:** [@Jeff\_Bursik](https://discuss.elastic.co/u/Jeff_Bursik)\ **Replies:** 0\ **Last updated:** [January 15, 2019, 4:32pm UTC](https://discuss.elastic.co/t/logstash-kafka-output-unknown-topic-or-partition-blocks-all-other-logs/164344 "2019-01-15T16:32:46Z")
I'm curious if anyone else has faced this or if theres a recommended approach: Here is the setup: Docker Swarm node runs several docker containers Logstash parses all docker containers on the node Logstash pushes to a… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=616) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=618)