# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=929 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 930 --- ## [Sequence ID in logstash](https://discuss.elastic.co/t/sequence-id-in-logstash/96467)
**Author:** [@bilancocueno](https://discuss.elastic.co/u/bilancocueno)\ **Replies:** 0\ **Last updated:** [August 9, 2017, 3:14pm UTC](https://discuss.elastic.co/t/sequence-id-in-logstash/96467 "2017-08-09T15:14:45Z")
Hello, I’m pretty new to the elastic Stack and i would like to do something that i don t find anywhere. I’m saving syslogs in elasticsearch through logstash and i would like to add a “sequence id” field to know in what… --- ## [SSL Certificates](https://discuss.elastic.co/t/ssl-certificates/95705)
**Author:** [@runtman](https://discuss.elastic.co/u/runtman)\ **Replies:** 2\ **Last updated:** [August 9, 2017, 2:54pm UTC](https://discuss.elastic.co/t/ssl-certificates/95705 "2017-08-09T14:54:58Z")
Excuse me if this is a silly question, but you are able to secure Elasticsearch using SSL certificate and authorities rather than keystores, but it seems you can’t set up Logstash to talk to Elasticsearch in the same way… --- ## [Can logstash automatically identify KV pairs in the logs and parse them](https://discuss.elastic.co/t/can-logstash-automatically-identify-kv-pairs-in-the-logs-and-parse-them/96464)
**Author:** [@shroh](https://discuss.elastic.co/u/shroh)\ **Replies:** 0\ **Last updated:** [August 9, 2017, 2:53pm UTC](https://discuss.elastic.co/t/can-logstash-automatically-identify-kv-pairs-in-the-logs-and-parse-them/96464 "2017-08-09T14:53:59Z")
Can Logstash automatically identify KV pairs in the logs and parse them just like how splunk does. I actually wrote one GROK filter which parses the fields and defined a KV filter so that the additional fields can be pa… --- ## [Grok after CSV filter](https://discuss.elastic.co/t/grok-after-csv-filter/96303)
**Author:** [@somu411](https://discuss.elastic.co/u/somu411)\ **Replies:** 12\ **Last updated:** [August 9, 2017, 1:54pm UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303 "2017-08-09T13:54:38Z")
Hi I would be thankful who helps me in extracting substring between two forward slashes using grok after csv filter My CSV file is GIS,/Dianon/CancerRegPHNMS,22876365,96589706,ID\_Dianon\_170708235043,2905,2017-07-09,01… --- ## [Really big retrieval lag for Logstash Kafka inputs producing data irregularly](https://discuss.elastic.co/t/really-big-retrieval-lag-for-logstash-kafka-inputs-producing-data-irregularly/96435)
**Author:** [@mwysokin](https://discuss.elastic.co/u/mwysokin)\ **Replies:** 0\ **Last updated:** [August 9, 2017, 12:40pm UTC](https://discuss.elastic.co/t/really-big-retrieval-lag-for-logstash-kafka-inputs-producing-data-irregularly/96435 "2017-08-09T12:40:49Z")
Hi! I’m using logstash 2.4 with kafka input 5.1.6. In my config I created a field called input\_lag in order to monitor how much time it takes logstash to process logs: ruby { code =\> "event\['lag\_seconds'\] = (Time… --- ## [Configure a list of integers with Logstash](https://discuss.elastic.co/t/configure-a-list-of-integers-with-logstash/96098)
**Author:** [@marwa](https://discuss.elastic.co/u/marwa)\ **Replies:** 8\ **Last updated:** [August 9, 2017, 12:04pm UTC](https://discuss.elastic.co/t/configure-a-list-of-integers-with-logstash/96098 "2017-08-09T12:04:39Z")
Hi, i want to load datasets (csv file) to elasticseach viaa logstash,this is a snippet of my data sets transaction\_date,customer\_name,age,job,fidelity\_card\_id,product\_id\_sold, 2/24/2017;14:15:43,Roberto Michel,25,Anal… --- ## [fIlter by hour given timestamp](https://discuss.elastic.co/t/filter-by-hour-given-timestamp/96426)
**Author:** [@tabs11](https://discuss.elastic.co/u/tabs11)\ **Replies:** 0\ **Last updated:** [August 9, 2017, 11:11am UTC](https://discuss.elastic.co/t/filter-by-hour-given-timestamp/96426 "2017-08-09T11:11:17Z")
Hi, I have a huge amount of logs and I want to only parse the data between specific hours. In each log I have a all day information and I want to extract the data in specific hours of the day. I have a timestamp: “@tim… --- ## [Logstash stops running after running for few seconds](https://discuss.elastic.co/t/logstash-stops-running-after-running-for-few-seconds/95765)
**Author:** [@shroh](https://discuss.elastic.co/u/shroh)\ **Replies:** 13\ **Last updated:** [August 9, 2017, 11:06am UTC](https://discuss.elastic.co/t/logstash-stops-running-after-running-for-few-seconds/95765 "2017-08-09T11:06:16Z")
Hi, My logstash stops running after 4 seconds of starting it, i get this output for systemctl status logstash -l logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor prese… --- ## [Logstash CPU load metric always -1](https://discuss.elastic.co/t/logstash-cpu-load-metric-always-1/96419)
**Author:** [@aristen](https://discuss.elastic.co/u/aristen)\ **Replies:** 0\ **Last updated:** [August 9, 2017, 10:19am UTC](https://discuss.elastic.co/t/logstash-cpu-load-metric-always-1/96419 "2017-08-09T10:19:14Z")
Hey all. I am having some trouble with monitoring Logstash in our ELK setup that i hope some of you have experienced as well. When i look at the CPU load of our logstash nodes they always says -1. When i call ‘host:96… --- ## [Replace middle of string](https://discuss.elastic.co/t/replace-middle-of-string/96382)
**Author:** [@VincentNgai86](https://discuss.elastic.co/u/VincentNgai86)\ **Replies:** 2\ **Last updated:** [August 9, 2017, 9:39am UTC](https://discuss.elastic.co/t/replace-middle-of-string/96382 "2017-08-09T09:39:57Z")
Hi All , Have question about how to gsub or replace some of middle string The orginal body like this A=somestinghere1&B=somestringhere2&C=somestringhere3&D=somestringhere4 What i would like to do like remain A,C and … --- ## [PKI issue for Logstash monitoring](https://discuss.elastic.co/t/pki-issue-for-logstash-monitoring/96097)
**Author:** [@vb4t](https://discuss.elastic.co/u/vb4t)\ **Replies:** 2\ **Last updated:** [August 9, 2017, 9:21am UTC](https://discuss.elastic.co/t/pki-issue-for-logstash-monitoring/96097 "2017-08-09T09:21:10Z")
Hello. I switched logstash ES output from basic to PKI auth. Works like a charm: output { elasticsearch { hosts =\> \[ "${LOGSTASH\_CONFIGVAR\_ES\_SERVER\_1}", "${LOGSTASH\_CONFIGVAR\_ES\_SERVER\_2}", "${LOGSTASH\_CONFIGVAR\_… --- ## [\[CLOSED\]Grok pattern - Help](https://discuss.elastic.co/t/closed-grok-pattern-help/96380)
**Author:** [@Pedro\_Cabral](https://discuss.elastic.co/u/Pedro_Cabral)\ **Replies:** 4\ **Last updated:** [August 9, 2017, 9:17am UTC](https://discuss.elastic.co/t/closed-grok-pattern-help/96380 "2017-08-09T09:17:28Z")
Hello, My logs are like this, the part I’m interested in is after the absolute time, but if I need use all of the fields there is no problem. Aug 9 05:54:25 centos-2gb-lon1-01-bro bro\_conn: 1502257999.489181 CHQP… --- ## [Doc fields in CSV destination](https://discuss.elastic.co/t/doc-fields-in-csv-destination/96099)
**Author:** [@AHurtado](https://discuss.elastic.co/u/AHurtado)\ **Replies:** 2\ **Last updated:** [August 9, 2017, 8:37am UTC](https://discuss.elastic.co/t/doc-fields-in-csv-destination/96099 "2017-08-09T08:37:26Z")
Hi, I’m very new in ELK stack and I’m trying to learn. I’d like to enrich a document adding fields based on a translate filter and a dictionary\_path value. My question is: when building the dictionary\_path file, can I … --- ## [Multiple pattern matching](https://discuss.elastic.co/t/multiple-pattern-matching/96221)
**Author:** [@DFrant](https://discuss.elastic.co/u/DFrant)\ **Replies:** 3\ **Last updated:** [August 9, 2017, 8:04am UTC](https://discuss.elastic.co/t/multiple-pattern-matching/96221 "2017-08-09T08:04:43Z")
Hello, I have a filter defined as follow: grok { patterns\_dir =\> \["/etc/logstash/conf.d/patterns/"\] match =\> \[ "message", "^%{TIMESTAMP… --- ## [Writing grok filter for my usecase](https://discuss.elastic.co/t/writing-grok-filter-for-my-usecase/95861)
**Author:** [@vinod\_hy](https://discuss.elastic.co/u/vinod_hy)\ **Replies:** 4\ **Last updated:** [August 9, 2017, 7:46am UTC](https://discuss.elastic.co/t/writing-grok-filter-for-my-usecase/95861 "2017-08-09T07:46:26Z")
I have the following line in log file, \[2017-07-07 11:41:30.794\]\[DEBUG\]\[Hello\] for the above log, i have written the grok filter as shown below, match =\> {“message” =\>"\[%{TIMESTAMP\_ISO8601:TimeStamp}\]\[%{LOGLEVEL:logL… --- ## [Creating dynamic elastic seach indices using a json field](https://discuss.elastic.co/t/creating-dynamic-elastic-seach-indices-using-a-json-field/96112)
**Author:** [@AswinKakarot](https://discuss.elastic.co/u/AswinKakarot)\ **Replies:** 1\ **Last updated:** [August 9, 2017, 6:20am UTC](https://discuss.elastic.co/t/creating-dynamic-elastic-seach-indices-using-a-json-field/96112 "2017-08-09T06:20:20Z")
I need to create different outputs to elasticsearch for different inputs. My logs are coming in json format. I have created a field called “environment” in the json input. I need to filter that field’s value and create i… --- ## [How does logstash buffering work when sinks are slow?](https://discuss.elastic.co/t/how-does-logstash-buffering-work-when-sinks-are-slow/96332)
**Author:** [@jayunit100](https://discuss.elastic.co/u/jayunit100)\ **Replies:** 1\ **Last updated:** [August 9, 2017, 5:55am UTC](https://discuss.elastic.co/t/how-does-logstash-buffering-work-when-sinks-are-slow/96332 "2017-08-09T05:55:19Z")
Given issues such as Kafka input slow stop consuming , its clear that some people can see silent throughput failures that may be related to sinks of logstash. What happens when a logstash sink isn’t capable of consuming… --- ## [Importing MySQL column into Elasticsearch as nested field with jdbc plugin](https://discuss.elastic.co/t/importing-mysql-column-into-elasticsearch-as-nested-field-with-jdbc-plugin/96162)
**Author:** [@Ricky\_Jenkins](https://discuss.elastic.co/u/Ricky_Jenkins)\ **Replies:** 5\ **Last updated:** [August 9, 2017, 5:34am UTC](https://discuss.elastic.co/t/importing-mysql-column-into-elasticsearch-as-nested-field-with-jdbc-plugin/96162 "2017-08-09T05:34:18Z")
Hi, I am new to elasticsearch and have spent the last two weeks trying to learn everything i can to the best of my ability but as one can imagine it is somewhat overwhelming from a new-coming standpoint. I am trying to … --- ## [How to sync database from mysql to elasticsearch with programming?](https://discuss.elastic.co/t/how-to-sync-database-from-mysql-to-elasticsearch-with-programming/96364)
**Author:** [@Prapat\_Polchan](https://discuss.elastic.co/u/Prapat_Polchan)\ **Replies:** 0\ **Last updated:** [August 9, 2017, 2:40am UTC](https://discuss.elastic.co/t/how-to-sync-database-from-mysql-to-elasticsearch-with-programming/96364 "2017-08-09T02:40:33Z")
i want to sync database from sybase database but sybase database using ISO8859\_1 i found a problem character on elasticsearch ??? i used JDBC and set charset but not work. Can i writing code to encode charset custom in … --- ## [Tagging Patterns of data into a field](https://discuss.elastic.co/t/tagging-patterns-of-data-into-a-field/96164)
**Author:** [@Drift](https://discuss.elastic.co/u/Drift)\ **Replies:** 4\ **Last updated:** [August 9, 2017, 2:19am UTC](https://discuss.elastic.co/t/tagging-patterns-of-data-into-a-field/96164 "2017-08-09T02:19:32Z")
Hi, I have a data set which has a field that contain different values like: FieldA : AAAA-REL FieldA : AAAA-DEV FieldA : BBBB-REL FieldA : DDDD-REL FieldA : BBBB-DEV … and so on I want to tag or categorize these … --- ## [Grok patterns subdivision](https://discuss.elastic.co/t/grok-patterns-subdivision/96346)
**Author:** [@SandhyaRani](https://discuss.elastic.co/u/SandhyaRani)\ **Replies:** 0\ **Last updated:** [August 8, 2017, 9:47pm UTC](https://discuss.elastic.co/t/grok-patterns-subdivision/96346 "2017-08-08T21:47:51Z")
Hi, I have java exception log like this 2016-12-16 21:28:05,668 ERROR \[int-http-28\] \[nbiws::::\] c.t.d.s.impl.DiagnosticServiceImpl - Error running a diagnostic workflow : 9003: Invalid arguments com.twowire.dmc.listener… --- ## [Xml/xpath returning an array](https://discuss.elastic.co/t/xml-xpath-returning-an-array/96345)
**Author:** [@Badger](https://discuss.elastic.co/u/Badger)\ **Replies:** 0\ **Last updated:** [August 8, 2017, 9:35pm UTC](https://discuss.elastic.co/t/xml-xpath-returning-an-array/96345 "2017-08-08T21:35:41Z")
I am trying to parse SOAP messages, which take the following form \\\1\\\\2\\\ I need to get the name of the first element of the Body, which I tried to do… --- ## [Logstash progress in loading from oracle database?](https://discuss.elastic.co/t/logstash-progress-in-loading-from-oracle-database/96344)
**Author:** [@ruibinluo](https://discuss.elastic.co/u/ruibinluo)\ **Replies:** 0\ **Last updated:** [August 8, 2017, 9:15pm UTC](https://discuss.elastic.co/t/logstash-progress-in-loading-from-oracle-database/96344 "2017-08-08T21:15:11Z")
I have configured the logstash to load data (12 million records) from Oracle database. No error in the log, but logstash log does not display anything after the sql statement. Neither elasticsearch shows anything. I did … --- ## [Logstash convert string to integer](https://discuss.elastic.co/t/logstash-convert-string-to-integer/96325)
**Author:** [@bitesize](https://discuss.elastic.co/u/bitesize)\ **Replies:** 3\ **Last updated:** [August 8, 2017, 8:29pm UTC](https://discuss.elastic.co/t/logstash-convert-string-to-integer/96325 "2017-08-08T20:29:01Z")
I have json string array that I am decomposing into separate fields, such as: ’{“eventid”:1,“content”:\[“a”,“b”,“c”,“d”\]}' after adding some field names and remove\_field content. I get the following: { "@timestamp" =\> … --- ## [How well does the syslog output module handle queuing if the destination host goes down temporarily? Or what is the best way to send logs to a 3rd party?](https://discuss.elastic.co/t/how-well-does-the-syslog-output-module-handle-queuing-if-the-destination-host-goes-down-temporarily-or-what-is-the-best-way-to-send-logs-to-a-3rd-party/93443)
**Author:** [@dfinn](https://discuss.elastic.co/u/dfinn)\ **Replies:** 6\ **Last updated:** [August 8, 2017, 8:26pm UTC](https://discuss.elastic.co/t/how-well-does-the-syslog-output-module-handle-queuing-if-the-destination-host-goes-down-temporarily-or-what-is-the-best-way-to-send-logs-to-a-3rd-party/93443 "2017-08-08T20:26:41Z")
We've recently been asked to ship a portion of our logs to a client at their datacenter. We are currently using the syslog output module to send logs from logstash to another internal device at the end of our pipeline. … --- ## [Filtering on beats fields issue](https://discuss.elastic.co/t/filtering-on-beats-fields-issue/96028)
**Author:** [@Puk](https://discuss.elastic.co/u/Puk)\ **Replies:** 6\ **Last updated:** [August 8, 2017, 8:19pm UTC](https://discuss.elastic.co/t/filtering-on-beats-fields-issue/96028 "2017-08-08T20:19:51Z")
Fairly new to ELK Stack so please bear with me :slight\_smile: I have a single beats.conf file in Logstash at the moment for a new deployment and I thought it was all working fine, however when trying to make some change… --- ## [Not able to read large csv file in logstash](https://discuss.elastic.co/t/not-able-to-read-large-csv-file-in-logstash/95987)
**Author:** [@developer\_apk](https://discuss.elastic.co/u/developer_apk)\ **Replies:** 2\ **Last updated:** [August 8, 2017, 8:16pm UTC](https://discuss.elastic.co/t/not-able-to-read-large-csv-file-in-logstash/95987 "2017-08-08T20:16:06Z")
I am using below config to read large file of 1gb size having around 5m records, it is not giving any output or errors input { file { path =\> “/filepath/filename.txt” } } filter { csv { separator =\> “|” } } ou… --- ## [ES input : ingest aggregation result](https://discuss.elastic.co/t/es-input-ingest-aggregation-result/96140)
**Author:** [@smcooper](https://discuss.elastic.co/u/smcooper)\ **Replies:** 0\ **Last updated:** [August 7, 2017, 5:07pm UTC](https://discuss.elastic.co/t/es-input-ingest-aggregation-result/96140 "2017-08-07T17:07:03Z")
Hi, I would like to create an Elasticsearch index containing the result of an aggregation (some statistics), and not the documents of the source index. I get the expected result from Elasticsearch by including “size”:0… --- ## [Reached open files limit when running as service](https://discuss.elastic.co/t/reached-open-files-limit-when-running-as-service/96329)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 0\ **Last updated:** [August 8, 2017, 7:14pm UTC](https://discuss.elastic.co/t/reached-open-files-limit-when-running-as-service/96329 "2017-08-08T19:14:57Z")
I am not understanding something that when i run logstash from command line from logstash user itself it works perfectly fine and i see data being ingested. /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logst… --- ## [Logstash will read 3days old file](https://discuss.elastic.co/t/logstash-will-read-3days-old-file/95950)
**Author:** [@P\_Kumar](https://discuss.elastic.co/u/P_Kumar)\ **Replies:** 6\ **Last updated:** [August 8, 2017, 6:54pm UTC](https://discuss.elastic.co/t/logstash-will-read-3days-old-file/95950 "2017-08-08T18:54:58Z")
Is it possible to logstash5 will read 3 days old SystemOut.log file ? [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=928) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=930)