# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=592 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 593 --- ## [Logstash - how to configure CSV filter for joining 2 CSV files based on a common field and send the joined log to ES](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125)
**Author:** [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\ **Replies:** 28\ **Last updated:** [February 28, 2019, 1:29pm UTC](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125 "2019-02-28T13:29:55Z")
Hi Team, I need your help for my below scenario which I am suffering a lot since last week. I googled a lot but can't figure out a good anwser for my case. My Scenario is I have 2 CVS file which contains a same column … --- ## [Index parent-child data from csv to elasticsearch](https://discuss.elastic.co/t/index-parent-child-data-from-csv-to-elasticsearch/170035)
**Author:** [@chandan.sing1](https://discuss.elastic.co/u/chandan.sing1)\ **Replies:** 1\ **Last updated:** [February 28, 2019, 12:12pm UTC](https://discuss.elastic.co/t/index-parent-child-data-from-csv-to-elasticsearch/170035 "2019-02-28T12:12:31Z")
Hi, I have 2 csv files with 1 to many relationship. How would i index both the files in same index with relationship. So, that i can generate the deep dive visualisation in detail based on selection. CSV 1 : customerID… --- ## [Installing keystore](https://discuss.elastic.co/t/installing-keystore/163793)
**Author:** [@kb2295](https://discuss.elastic.co/u/kb2295)\ **Replies:** 5\ **Last updated:** [February 28, 2019, 11:57am UTC](https://discuss.elastic.co/t/installing-keystore/163793 "2019-02-28T11:57:20Z")
Hello, I'm trying to install keystore using the following command: bin/logstash-keystore create and I get this error: bin/logstash-keystore: No such file or directory I'm in my logstash directory and don't under… --- ## ["Successfully start Logstash API endpoint {:port =\>9600}" Stuck](https://discuss.elastic.co/t/successfully-start-logstash-api-endpoint-port-9600-stuck/170258)
**Author:** [@rb6](https://discuss.elastic.co/u/rb6)\ **Replies:** 4\ **Last updated:** [February 28, 2019, 10:13am UTC](https://discuss.elastic.co/t/successfully-start-logstash-api-endpoint-port-9600-stuck/170258 "2019-02-28T10:13:33Z")
Hi everyone, currently stuck trying to setup Logstash to accept iis logs. I'm on a windows 10 workstation with the logs being copied from external servers. At the moment when logstash is ran, it starts, then stops on the… --- ## [Multiline Codec: Message mismatch between ES and ruby plugin logger](https://discuss.elastic.co/t/multiline-codec-message-mismatch-between-es-and-ruby-plugin-logger/170194)
**Author:** [@Anabella\_Cristaldi](https://discuss.elastic.co/u/Anabella_Cristaldi)\ **Replies:** 2\ **Last updated:** [February 28, 2019, 9:44am UTC](https://discuss.elastic.co/t/multiline-codec-message-mismatch-between-es-and-ruby-plugin-logger/170194 "2019-02-28T09:44:44Z")
Hi all, I'm processing some NMAP XML output using multiline input plugin and XML filter plugin. And either I'm doing something wrong or an strange behavior is taking place. In order to check if multiline codec were sp… --- ## [Sending logs from Logstash to ArcSight via Syslog CEF](https://discuss.elastic.co/t/sending-logs-from-logstash-to-arcsight-via-syslog-cef/170247)
**Author:** [@elic1997](https://discuss.elastic.co/u/elic1997)\ **Replies:** 2\ **Last updated:** [February 28, 2019, 9:18am UTC](https://discuss.elastic.co/t/sending-logs-from-logstash-to-arcsight-via-syslog-cef/170247 "2019-02-28T09:18:32Z")
Hello, I've configured my Logstash to send event to an ArcSight Connector via the syslog output plugin with cef codec. The data I receive looks like this: {"name": SERVER\_NAME} LOGSTASH\[\*\]: CEF:0|Elasticsearch|Logsta… --- ## [Logstash configuration dynamic management](https://discuss.elastic.co/t/logstash-configuration-dynamic-management/170299)
**Author:** [@xodn0812](https://discuss.elastic.co/u/xodn0812)\ **Replies:** 0\ **Last updated:** [February 28, 2019, 9:07am UTC](https://discuss.elastic.co/t/logstash-configuration-dynamic-management/170299 "2019-02-28T09:07:06Z")
I run logstash in the following manner. bin/logstash -f config/log\_test.conf --config.reload.automatic Contents: input{} filter{~} output{ if { ~~~~~ } else if { ~~~~~~~ } } I want to dynamically change the c… --- ## [Offline to install logstash-output-mongodb plugin](https://discuss.elastic.co/t/offline-to-install-logstash-output-mongodb-plugin/170273)
**Author:** [@linsie](https://discuss.elastic.co/u/linsie)\ **Replies:** 0\ **Last updated:** [February 28, 2019, 5:07am UTC](https://discuss.elastic.co/t/offline-to-install-logstash-output-mongodb-plugin/170273 "2019-02-28T05:07:35Z")
Hi Guys, Recently, I want to install logstash-output-mongodb plugin offline, but not successful. So could anyone help me ? Thank you so much. system version: centos 6.8 logstash version: 6.6.1 jdk version: 1… --- ## [Logstash can not get access to elasticsearch and can not update the mapping](https://discuss.elastic.co/t/logstash-can-not-get-access-to-elasticsearch-and-can-not-update-the-mapping/170255)
**Author:** [@kesha](https://discuss.elastic.co/u/kesha)\ **Replies:** 2\ **Last updated:** [February 27, 2019, 11:13pm UTC](https://discuss.elastic.co/t/logstash-can-not-get-access-to-elasticsearch-and-can-not-update-the-mapping/170255 "2019-02-27T23:13:52Z")
Elasticsearch: Version 6.5.4 Logstash: Version 6.5.4 OS: CentOS 6 elasticsearch.yml: node.name: dev node.data: true path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 127.0.0.1 http.por… --- ## [Persistent queue only filling on page](https://discuss.elastic.co/t/persistent-queue-only-filling-on-page/170240)
**Author:** [@smbullet](https://discuss.elastic.co/u/smbullet)\ **Replies:** 0\ **Last updated:** [February 27, 2019, 8:22pm UTC](https://discuss.elastic.co/t/persistent-queue-only-filling-on-page/170240 "2019-02-27T20:22:19Z")
Here is my logstash.yml: config.reload.automatic: true dead\_letter\_queue.enable: true http.host: 0.0.0.0 path.config: /home/logstash\_user/logstash/conf.d/ path.logs: /var/log/logstash pipeline.batch.size: 250 pip… --- ## [Gsub match specific string and replace with blank](https://discuss.elastic.co/t/gsub-match-specific-string-and-replace-with-blank/170225)
**Author:** [@Arnav\_Sengupta](https://discuss.elastic.co/u/Arnav_Sengupta)\ **Replies:** 1\ **Last updated:** [February 27, 2019, 7:20pm UTC](https://discuss.elastic.co/t/gsub-match-specific-string-and-replace-with-blank/170225 "2019-02-27T19:20:42Z")
I am trying to replace a specific string using gsub but it's not working correctly. Input -\> /ICW/PDR/backend Desired Output -\> /ICW/PDR What I have used is, gsub =\> \["backend\_name", "\[\\/backend\]", ""\] However, it… --- ## [Do global variables override the result of matching?](https://discuss.elastic.co/t/do-global-variables-override-the-result-of-matching/170042)
**Author:** [@jcaballero](https://discuss.elastic.co/u/jcaballero)\ **Replies:** 7\ **Last updated:** [February 27, 2019, 4:30pm UTC](https://discuss.elastic.co/t/do-global-variables-override-the-result-of-matching/170042 "2019-02-27T16:30:07Z")
Hi maybe the subject is not clear. I will try to explain it. I have a log file that looks like this 06/22/18 00:41:11.719 PID = 1036 ... 06/22/18 00:41:11.757 host is "host1" ... 06/22/18 00:41:11.772 JOB ID: 87.113 .… --- ## [PipeLine-\> Filters-\> (tags?)-\> Is it possible to parse message by specific keywords after pattern parsing log message](https://discuss.elastic.co/t/pipeline-filters-tags-is-it-possible-to-parse-message-by-specific-keywords-after-pattern-parsing-log-message/169973)
**Author:** [@DanaDascalu](https://discuss.elastic.co/u/DanaDascalu)\ **Replies:** 4\ **Last updated:** [February 27, 2019, 1:18pm UTC](https://discuss.elastic.co/t/pipeline-filters-tags-is-it-possible-to-parse-message-by-specific-keywords-after-pattern-parsing-log-message/169973 "2019-02-27T13:18:54Z")
Hi, I new to using ELK stack, and need to create dashboards of meaningful data from our log files. Currently i'm using filebeat to read log files, then parsing the messages using LogStash, and later on view the data in… --- ## [Dissect pattern not found](https://discuss.elastic.co/t/dissect-pattern-not-found/170143)
**Author:** [@OrangeDog](https://discuss.elastic.co/u/OrangeDog)\ **Replies:** 10\ **Last updated:** [February 27, 2019, 11:37am UTC](https://discuss.elastic.co/t/dissect-pattern-not-found/170143 "2019-02-27T11:37:51Z")
I cannot see why this isn't working (other event fields removed for clarity). Dissector mapping, pattern not found {"field"=\>"message", "pattern"=\>" \[%{level-\>}\] %{}", "event"=\>{"message"=\>" \[WARNING \] Test"}} --- ## [Using Grok with KV filter](https://discuss.elastic.co/t/using-grok-with-kv-filter/170136)
**Author:** [@spadhi](https://discuss.elastic.co/u/spadhi)\ **Replies:** 5\ **Last updated:** [February 27, 2019, 11:23am UTC](https://discuss.elastic.co/t/using-grok-with-kv-filter/170136 "2019-02-27T11:23:34Z")
Hi, I am pretty new to ELK stack. Currently I am trying to parse my application log using grok pattern. But since my logs are not structured i may have to write too many grok conditions, which will not scale well. Sampl… --- ## [Failure with receiving network data - CURLE\_RECV\_ERROR (56)](https://discuss.elastic.co/t/failure-with-receiving-network-data-curle-recv-error-56/169556)
**Author:** [@Dominique\_Fabre](https://discuss.elastic.co/u/Dominique_Fabre)\ **Replies:** 1\ **Last updated:** [February 27, 2019, 10:50am UTC](https://discuss.elastic.co/t/failure-with-receiving-network-data-curle-recv-error-56/169556 "2019-02-27T10:50:13Z")
Hi, I created a self-signed certificate like this openssl req -subj '/CN=logstash\_hostname/' -x509 -days 3652 -batch -nodes -newkey rsa:2048 -keyout logstash.key -out logstash.crt Then i set the logstash configuration… --- ## [Re-Index data from elasticsearch back to another elasticsearch using Logstash](https://discuss.elastic.co/t/re-index-data-from-elasticsearch-back-to-another-elasticsearch-using-logstash/169217)
**Author:** [@manikandanid](https://discuss.elastic.co/u/manikandanid)\ **Replies:** 0\ **Last updated:** [February 20, 2019, 1:14pm UTC](https://discuss.elastic.co/t/re-index-data-from-elasticsearch-back-to-another-elasticsearch-using-logstash/169217 "2019-02-20T13:14:38Z")
Want to use logstash to query ES for every 5 min and the output should be sent to another index in ES. How to implement this using elasticsearch input plugin. Have to use scroll so that querying large data shouln't impac… --- ## [Log stash is not reading last record in a file](https://discuss.elastic.co/t/log-stash-is-not-reading-last-record-in-a-file/169979)
**Author:** [@Mayurbiw](https://discuss.elastic.co/u/Mayurbiw)\ **Replies:** 3\ **Last updated:** [February 27, 2019, 10:20am UTC](https://discuss.elastic.co/t/log-stash-is-not-reading-last-record-in-a-file/169979 "2019-02-27T10:20:17Z")
I am reading an audit file using log stash. The problem is log stash doesn't read the last record in a file let's call that record X. when I push another record in a file let's call it Y. it reads the X and presents it … --- ## [Logstash log4j configuration pipeline.id](https://discuss.elastic.co/t/logstash-log4j-configuration-pipeline-id/169455)
**Author:** [@janmg](https://discuss.elastic.co/u/janmg)\ **Replies:** 1\ **Last updated:** [February 27, 2019, 9:47am UTC](https://discuss.elastic.co/t/logstash-log4j-configuration-pipeline-id/169455 "2019-02-27T09:47:19Z")
I created an logstash input plugin in jruby and when I do an @logger.info("log entry"), I'd like it to have a pipeline.id in the logstash-plain.log as generated by log4j2. Since I have several pipelines it's nice to use … --- ## [Patterns\_dir in logstash not working](https://discuss.elastic.co/t/patterns-dir-in-logstash-not-working/170124)
**Author:** [@Tomas\_Langer](https://discuss.elastic.co/u/Tomas_Langer)\ **Replies:** 0\ **Last updated:** [February 27, 2019, 9:11am UTC](https://discuss.elastic.co/t/patterns-dir-in-logstash-not-working/170124 "2019-02-27T09:11:07Z")
Hello, I have a problem with logstasth configuration. I will try to explain my problem: I run ELK stack in Docker image on Virtual Linux machine from Win 10. I have this part in file logstash.conf (UTF-8 LF line endin… --- ## [Round robin with syslog output](https://discuss.elastic.co/t/round-robin-with-syslog-output/169898)
**Author:** [@Vincent\_Maury](https://discuss.elastic.co/u/Vincent_Maury)\ **Replies:** 2\ **Last updated:** [February 27, 2019, 8:42am UTC](https://discuss.elastic.co/t/round-robin-with-syslog-output/169898 "2019-02-27T08:42:04Z")
Hi, is it possible to have several hosts in the syslog output plugins (with round robin LB)? Thank you in advance! --- ## [Json parsing problem tcp input](https://discuss.elastic.co/t/json-parsing-problem-tcp-input/169562)
**Author:** [@hoff](https://discuss.elastic.co/u/hoff)\ **Replies:** 8\ **Last updated:** [February 27, 2019, 7:59am UTC](https://discuss.elastic.co/t/json-parsing-problem-tcp-input/169562 "2019-02-27T07:59:17Z")
I have a problem with logs in syslog. Logstash logging warn like a \[WARN \]\[logstash.codecs.jsonlines\] JSON parse error, original data now in message field {:error=\>#\, :… --- ## [How can use a field in jdbc statement of jdbc streaming](https://discuss.elastic.co/t/how-can-use-a-field-in-jdbc-statement-of-jdbc-streaming/169942)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 2\ **Last updated:** [February 27, 2019, 5:01am UTC](https://discuss.elastic.co/t/how-can-use-a-field-in-jdbc-statement-of-jdbc-streaming/169942 "2019-02-27T05:01:21Z")
hi all, I am using following logstash.conf file: input { jdbc { jdbc\_driver\_library =\> "F:\\driver\\sqljdbc\_6.0\\enu\\jre8\\sqljdbc42.jar" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc… --- ## [Logstash: Not able to parse date field from csv](https://discuss.elastic.co/t/logstash-not-able-to-parse-date-field-from-csv/169994)
**Author:** [@rparmar812](https://discuss.elastic.co/u/rparmar812)\ **Replies:** 2\ **Last updated:** [February 27, 2019, 4:18am UTC](https://discuss.elastic.co/t/logstash-not-able-to-parse-date-field-from-csv/169994 "2019-02-27T04:18:22Z")
I'm not able to parse date field ("Time") from given csv file. it's giving error all the time. Help will be appreciated. CSV Time,TotalMsg,CancelMsg,NEWMsg,ModifyMsg,CancelThrottle,ModifyThrottle,NEWThrottle,TOTALThro… --- ## [How to use split filter on the field using logstash](https://discuss.elastic.co/t/how-to-use-split-filter-on-the-field-using-logstash/169725)
**Author:** [@muralasandeep](https://discuss.elastic.co/u/muralasandeep)\ **Replies:** 4\ **Last updated:** [February 27, 2019, 4:16am UTC](https://discuss.elastic.co/t/how-to-use-split-filter-on-the-field-using-logstash/169725 "2019-02-27T04:16:13Z")
Hi everyone , i am new at logstash here i am trying to fatch data from .csv file and i have 6 fields total and in that 6 fields one field have actually data i need to split that field as multiple fields , here is my co… --- ## [ECS & Palo Alto Logs](https://discuss.elastic.co/t/ecs-palo-alto-logs/167542)
**Author:** [@davidhowell.tx](https://discuss.elastic.co/u/davidhowell.tx)\ **Replies:** 3\ **Last updated:** [February 27, 2019, 3:00am UTC](https://discuss.elastic.co/t/ecs-palo-alto-logs/167542 "2019-02-27T03:00:22Z")
Hello. I've seen some logstash pipelines for parsing Palo Alto logs by using CSV plugin. I'm curious if anyone has taken this a step further and aligned the fields with the Elastic Common Schema. I am working on doing … --- ## [Threat Intelligence Platform Integration](https://discuss.elastic.co/t/threat-intelligence-platform-integration/169949)
**Author:** [@andrewjo](https://discuss.elastic.co/u/andrewjo)\ **Replies:** 1\ **Last updated:** [February 27, 2019, 2:43am UTC](https://discuss.elastic.co/t/threat-intelligence-platform-integration/169949 "2019-02-27T02:43:24Z")
Hi Guys, Any idea , how integrate Threat Intelligence Platform with ELK? Or I just create some config in logstash and manually lookup csv from TI data source feed? Thanks. --- ## [Logstash Kafka Client Version 0.8 input and Elasticsearch-6.6 output](https://discuss.elastic.co/t/logstash-kafka-client-version-0-8-input-and-elasticsearch-6-6-output/170087)
**Author:** [@rpuserh](https://discuss.elastic.co/u/rpuserh)\ **Replies:** 0\ **Last updated:** [February 27, 2019, 2:26am UTC](https://discuss.elastic.co/t/logstash-kafka-client-version-0-8-input-and-elasticsearch-6-6-output/170087 "2019-02-27T02:26:32Z")
I need to find logstash which one will be compatible with Kafka Client Version 0.8 on input and Elasticsearch-6.6 on output I have kafka cluster and Kafka Client Version 0.8 compatible with it I'm running logstash-2.4 … --- ## [Parse multi line application log](https://discuss.elastic.co/t/parse-multi-line-application-log/169805)
**Author:** [@Worradorn\_Sirisang](https://discuss.elastic.co/u/Worradorn_Sirisang)\ **Replies:** 2\ **Last updated:** [February 26, 2019, 11:11pm UTC](https://discuss.elastic.co/t/parse-multi-line-application-log/169805 "2019-02-26T23:11:16Z")
Hello all, I am new for elk stack. I try to parsing application using grok -\> using grok debugger for creating grok pattern but on grok debugger is alert me about miss match. the my log pattern is outlined below \[2019… --- ## [Logstash storing JDBC DB results set in some cache and not updating data set from DB](https://discuss.elastic.co/t/logstash-storing-jdbc-db-results-set-in-some-cache-and-not-updating-data-set-from-db/170028)
**Author:** [@ApekshaB](https://discuss.elastic.co/u/ApekshaB)\ **Replies:** 1\ **Last updated:** [February 26, 2019, 11:06pm UTC](https://discuss.elastic.co/t/logstash-storing-jdbc-db-results-set-in-some-cache-and-not-updating-data-set-from-db/170028 "2019-02-26T23:06:18Z")
Dear Team, KIndly find my losgtash conf file: input { jdbc { jdbc\_validate\_connection =\> true jdbc\_connection\_string =\> "jdbc:oracle:thin:@10.1.50.79:1521/ndaie2" jdbc\_user =\> "AXIA\_SPRINT\_… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=591) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=593)