# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=508 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 509 --- ## [Pattern Creation for Filter](https://discuss.elastic.co/t/pattern-creation-for-filter/190832)
**Author:** [@tvsuhail](https://discuss.elastic.co/u/tvsuhail)\ **Replies:** 4\ **Last updated:** [July 18, 2019, 9:47am UTC](https://discuss.elastic.co/t/pattern-creation-for-filter/190832 "2019-07-18T09:47:05Z")
42 32 45 41 31 44 43 36 39 37 38 43 B2EB1DC6978C \[16/07/19\]\[08:28:14\]\[EzAHC3421\]\[7632\]IN \[0\] \<200\> \[16/07/19\]\[08:28:15\]\[EzAHC3421\]\[7632\]IN \[1\] \[16/07/19\]\[08:28:15\]\[EzAHC3421\]\[7632\]Value \[000000268… --- ## [Is there any way to use conditional grok filter?](https://discuss.elastic.co/t/is-there-any-way-to-use-conditional-grok-filter/191147)
**Author:** [@Shinw00](https://discuss.elastic.co/u/Shinw00)\ **Replies:** 0\ **Last updated:** [July 18, 2019, 7:57am UTC](https://discuss.elastic.co/t/is-there-any-way-to-use-conditional-grok-filter/191147 "2019-07-18T07:57:19Z")
Hi! I'm studying ELK Stacks these days and trying to analyze logs on my mac. I used Logstash to get information in my laptop(txt file) and trying to send it to ElasticSearch. On my way, I had a problem with grok filter… --- ## [Can you help me with Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError" problem?](https://discuss.elastic.co/t/can-you-help-me-with-failed-to-execute-action-action-logstash-create-pipeline-id-main-exception-logstash-configurationerror-problem/190949)
**Author:** [@Shinw00](https://discuss.elastic.co/u/Shinw00)\ **Replies:** 2\ **Last updated:** [July 18, 2019, 7:48am UTC](https://discuss.elastic.co/t/can-you-help-me-with-failed-to-execute-action-action-logstash-create-pipeline-id-main-exception-logstash-configurationerror-problem/190949 "2019-07-18T07:48:32Z")
Hi. First, thank you for your attention. Now, I'm trying to give log in my laptop(txt file) to Elasticsearch by logstash. I have a problem with filters. I checked whether it runs by deleting filter part. \[2019-07-17T… --- ## [Nested field in a grok filter](https://discuss.elastic.co/t/nested-field-in-a-grok-filter/190758)
**Author:** [@theo1991](https://discuss.elastic.co/u/theo1991)\ **Replies:** 9\ **Last updated:** [July 18, 2019, 7:21am UTC](https://discuss.elastic.co/t/nested-field-in-a-grok-filter/190758 "2019-07-18T07:21:29Z")
Hi, I have a pattern for log which is like this : inactive: No, sources : \[{id: 1, type: fr, name:custom}, {id:2, type:fr, name: random}\], assigned: Yes I wrote this custom grok filter : inactive: %{WORD:inactive}, s… --- ## [What field to save the state of the logs for Logstash JDBC input plugin?](https://discuss.elastic.co/t/what-field-to-save-the-state-of-the-logs-for-logstash-jdbc-input-plugin/190600)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 4\ **Last updated:** [July 17, 2019, 11:10pm UTC](https://discuss.elastic.co/t/what-field-to-save-the-state-of-the-logs-for-logstash-jdbc-input-plugin/190600 "2019-07-17T23:10:39Z")
Is there any way to save the state of the logs in Logstash JDBC input plugin? For example, for file input plugin, there is sincedb\_path which remembers where it left off from reading the logs. --- ## [Grok Expression for Nginx logs through logstash to Kibana](https://discuss.elastic.co/t/grok-expression-for-nginx-logs-through-logstash-to-kibana/191095)
**Author:** [@vijayakrishna.rg](https://discuss.elastic.co/u/vijayakrishna.rg)\ **Replies:** 2\ **Last updated:** [July 17, 2019, 10:06pm UTC](https://discuss.elastic.co/t/grok-expression-for-nginx-logs-through-logstash-to-kibana/191095 "2019-07-17T22:06:59Z")
Can someone please help me with Grok Expression to forward Nginx access logs to elasticsearch using filebeat is 1st preference, if filebeat doesn't support filter then through logstash to elasticsearch using below Grok E… --- ## [Json\_batch: I need set the buffer size and flush time](https://discuss.elastic.co/t/json-batch-i-need-set-the-buffer-size-and-flush-time/191090)
**Author:** [@DANIEL\_CORTES\_FERNAN](https://discuss.elastic.co/u/DANIEL_CORTES_FERNAN)\ **Replies:** 1\ **Last updated:** [July 17, 2019, 9:48pm UTC](https://discuss.elastic.co/t/json-batch-i-need-set-the-buffer-size-and-flush-time/191090 "2019-07-17T21:48:30Z")
Hello, I'm trying to use json\_batch in http output, how I can set the buffer size and time to flush? My sample pipeline: logstash -e 'input{stdin {}} filter{ mutate { add\_field =\> { "level" =\> "info" "mrId" =\> "mrTest… --- ## [Logstash-input-cloudwatch\_logs missing some CloudWatch log events](https://discuss.elastic.co/t/logstash-input-cloudwatch-logs-missing-some-cloudwatch-log-events/191075)
**Author:** [@timkitch](https://discuss.elastic.co/u/timkitch)\ **Replies:** 1\ **Last updated:** [July 17, 2019, 7:41pm UTC](https://discuss.elastic.co/t/logstash-input-cloudwatch-logs-missing-some-cloudwatch-log-events/191075 "2019-07-17T19:41:17Z")
We're using version 1.0.1 of the logstash-input-cloudwatch\_logs logstash plugin and are randomly encountering mysterious gaps in the CloudWatch log events. We periodically have an event that is clearly in CloudWatch that… --- ## [Convert Elasticsearch Date/Time field from UTC To Local Time](https://discuss.elastic.co/t/convert-elasticsearch-date-time-field-from-utc-to-local-time/191070)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 8\ **Last updated:** [July 17, 2019, 7:18pm UTC](https://discuss.elastic.co/t/convert-elasticsearch-date-time-field-from-utc-to-local-time/191070 "2019-07-17T19:18:49Z")
I am query elasticsearch in my pipeline for a date/time field. Elasticsearch stores those date/time fields in UTC time, regardless of of the timezone they were ingested in. Is it possible to convert the time to local t… --- ## [Converting a new field](https://discuss.elastic.co/t/converting-a-new-field/191019)
**Author:** [@luc1](https://discuss.elastic.co/u/luc1)\ **Replies:** 3\ **Last updated:** [July 17, 2019, 3:41pm UTC](https://discuss.elastic.co/t/converting-a-new-field/191019 "2019-07-17T15:41:16Z")
Hello, In my logstash filter I created a new field with a condition which works. if "ntf" in \[entities\]\[sdn\]{ mutate{add\_field=\>\["VNFc","ntf"\]}} Now, this VNFc field appear with a "?" in Kibana, so it can't be use … --- ## [Logstash nssm service error](https://discuss.elastic.co/t/logstash-nssm-service-error/190857)
**Author:** [@bmovva61](https://discuss.elastic.co/u/bmovva61)\ **Replies:** 6\ **Last updated:** [July 17, 2019, 3:40pm UTC](https://discuss.elastic.co/t/logstash-nssm-service-error/190857 "2019-07-17T15:40:21Z")
hi guys need some help new to elk . my logstash works fine when i run in cmd but when i start it from windows service i get errors . Ignoring the 'pipelines.yml' file because modules or command line options are specifie… --- ## [3 questions about memory leak](https://discuss.elastic.co/t/3-questions-about-memory-leak/188760)
**Author:** [@jcaballero](https://discuss.elastic.co/u/jcaballero)\ **Replies:** 9\ **Last updated:** [July 17, 2019, 3:01pm UTC](https://discuss.elastic.co/t/3-questions-about-memory-leak/188760 "2019-07-17T15:01:57Z")
Using LogStash 6.7.1 It really feels like the memory consumption increases day by day. So: Is really there a memory leak? I read in some of the "RELEASE NOTES" some comment about minimizing it, so I assume it's real… --- ## [Ruby Filter Syntax](https://discuss.elastic.co/t/ruby-filter-syntax/190839)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 4\ **Last updated:** [July 17, 2019, 2:19pm UTC](https://discuss.elastic.co/t/ruby-filter-syntax/190839 "2019-07-17T14:19:40Z")
I'm trying to take a pair of fields that contain a date and subtract one from the other. However, my code is wrong somewhere and i get an error. This error occurs three times for each event, so I assume it's a syntax e… --- ## [Upgrade Elastic Stack 6.3 to 7.2](https://discuss.elastic.co/t/upgrade-elastic-stack-6-3-to-7-2/191021)
**Author:** [@adrianolimit](https://discuss.elastic.co/u/adrianolimit)\ **Replies:** 0\ **Last updated:** [July 17, 2019, 2:07pm UTC](https://discuss.elastic.co/t/upgrade-elastic-stack-6-3-to-7-2/191021 "2019-07-17T14:07:22Z")
Hello everyone, I'm using Elastic Stack in version 6.3 (Elasticsearch - Logstash - Kibana - Beats) and I plan to upgrade to 7.2. I followed the webinar presenting the best practices to upgrade Elastic Stack. I think I … --- ## [How to rightly use geoip filter & get latitude & longitude to visualize in Map](https://discuss.elastic.co/t/how-to-rightly-use-geoip-filter-get-latitude-longitude-to-visualize-in-map/190665)
**Author:** [@madhan8](https://discuss.elastic.co/u/madhan8)\ **Replies:** 3\ **Last updated:** [July 17, 2019, 1:38pm UTC](https://discuss.elastic.co/t/how-to-rightly-use-geoip-filter-get-latitude-longitude-to-visualize-in-map/190665 "2019-07-17T13:38:45Z")
Hello Guys, I have geoip plugin installed in elasticsearch and I have used the below logstash config - if "ng" in \[tags\] { grok { match =\> { "message" =\> "%{IP:service\_ip} - - \[%{HTTPDATE:service\_timestamp}\] "%{WORD… --- ## [How to read gzip encoding from log file](https://discuss.elastic.co/t/how-to-read-gzip-encoding-from-log-file/190886)
**Author:** [@sumits1000](https://discuss.elastic.co/u/sumits1000)\ **Replies:** 1\ **Last updated:** [July 17, 2019, 1:32pm UTC](https://discuss.elastic.co/t/how-to-read-gzip-encoding-from-log-file/190886 "2019-07-17T13:32:35Z")
Hi, So I am using logstash to push data to elasticsearch. For backup we are writing the same logs into a file databakup.log. For this we are using gzip true as below: output{ file { path =\> "/home/ubuntu/logstash-fil… --- ## [How I can replace time with actual time in my log file?](https://discuss.elastic.co/t/how-i-can-replace-time-with-actual-time-in-my-log-file/190891)
**Author:** [@sbke\_baap](https://discuss.elastic.co/u/sbke_baap)\ **Replies:** 1\ **Last updated:** [July 17, 2019, 1:27pm UTC](https://discuss.elastic.co/t/how-i-can-replace-time-with-actual-time-in-my-log-file/190891 "2019-07-17T13:27:02Z")
While uploading my data, the time field is getting update with the uploading time. Given below is my sample data \<134\> id=firewall sn=C0EAE4F8239E time="2019-03-01 03:26:12 UTC" fw=203.109.120.130 pri=6 c=1024 m=537 m… --- ## [Counting of the csv file and sending the counted data to elasticsearch using logstash](https://discuss.elastic.co/t/counting-of-the-csv-file-and-sending-the-counted-data-to-elasticsearch-using-logstash/190760)
**Author:** [@amritaraj23](https://discuss.elastic.co/u/amritaraj23)\ **Replies:** 3\ **Last updated:** [July 17, 2019, 1:02pm UTC](https://discuss.elastic.co/t/counting-of-the-csv-file-and-sending-the-counted-data-to-elasticsearch-using-logstash/190760 "2019-07-17T13:02:19Z")
Hi All, I have a csv file with below mentioned data. Order,Location,Date 123,Paris,26-07-2019 203,Amsterdam,03-07-2019 456,Paris,08-09-2018 I want to count and group the data on the basis of location and want to se… --- ## [Lot of queuing files on logstash](https://discuss.elastic.co/t/lot-of-queuing-files-on-logstash/185427)
**Author:** [@Nishant\_Hurnekar](https://discuss.elastic.co/u/Nishant_Hurnekar)\ **Replies:** 5\ **Last updated:** [June 19, 2019, 11:16am UTC](https://discuss.elastic.co/t/lot-of-queuing-files-on-logstash/185427 "2019-06-19T11:16:41Z")
Hi, We are running with ELK cluster 5.6.16. Our system as below Filebeat --\> Logstash --\> Elasticsearch (3 server in cluster) --\> KIbana We are observing very high queuing on out logstash servers since quite some tim… --- ## [Generate SSL Certs for Filebeat-\>Logstash](https://discuss.elastic.co/t/generate-ssl-certs-for-filebeat-logstash/190932)
**Author:** [@Sundaramoorthy\_Anand](https://discuss.elastic.co/u/Sundaramoorthy_Anand)\ **Replies:** 0\ **Last updated:** [July 17, 2019, 9:13am UTC](https://discuss.elastic.co/t/generate-ssl-certs-for-filebeat-logstash/190932 "2019-07-17T09:13:43Z")
I need to establish an SSL connection b/w my logstash and filebeat. To generate Cert I reffered ESCertUtil But it is confusing.. To create CA (as zip), I did bin/elasticsearch-certutil ca --pem then I unzipped it usi… --- ## [LogStash::Json::ParserError: Unrecognized token : was expecting ('true', 'false' or 'null')](https://discuss.elastic.co/t/logstash-unrecognized-token-was-expecting-true-false-or-null/189058)
**Author:** [@Shawcs](https://discuss.elastic.co/u/Shawcs)\ **Replies:** 3\ **Last updated:** [July 17, 2019, 9:05am UTC](https://discuss.elastic.co/t/logstash-unrecognized-token-was-expecting-true-false-or-null/189058 "2019-07-17T09:05:01Z")
Hi, I have trouble sending data from my logstash to another one. here is the error I have : \[2019 - 07 - 05T10: 50: 13, 107\]\[WARN\]\[logstash.codecs.jsonlines\]JSON parse error, original data now in message field { : er… --- ## [Bug : Not able to uninstall/remove LS plugin](https://discuss.elastic.co/t/bug-not-able-to-uninstall-remove-ls-plugin/190890)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 0\ **Last updated:** [July 17, 2019, 7:05am UTC](https://discuss.elastic.co/t/bug-not-able-to-uninstall-remove-ls-plugin/190890 "2019-07-17T07:05:10Z")
Hi All I am Not able to uninstall/remove LS plugin. This is what i am using bin\>logstash-plugin.bat remove logstash-filter-csv ERROR: Operation aborted, cannot remove plugin, message: The path /bin/logstash-core doe… --- ## [Config for ingest pipeline from Filebeat](https://discuss.elastic.co/t/config-for-ingest-pipeline-from-filebeat/190874)
**Author:** [@smerzlyakov](https://discuss.elastic.co/u/smerzlyakov)\ **Replies:** 0\ **Last updated:** [July 17, 2019, 5:27am UTC](https://discuss.elastic.co/t/config-for-ingest-pipeline-from-filebeat/190874 "2019-07-17T05:27:10Z")
Hello. I have a problem with ingest pipeline best practicies. I want to parse system logs by Filebeat pipeline, but do not use Filebeat, because we already collect these logs. I do ./filebeat setup --pipelines for ins… --- ## [Logstash Setting up multiple config files in one pipeline](https://discuss.elastic.co/t/logstash-setting-up-multiple-config-files-in-one-pipeline/190643)
**Author:** [@pgw5540](https://discuss.elastic.co/u/pgw5540)\ **Replies:** 4\ **Last updated:** [July 17, 2019, 4:13am UTC](https://discuss.elastic.co/t/logstash-setting-up-multiple-config-files-in-one-pipeline/190643 "2019-07-17T04:13:23Z")
Hi. Logstash I want to configure multiple configs on one pipeline. I want to read dozens of csv files and send data to the same number of different indexes. So, I created as many config files as there are, We want to… --- ## [Illegal Character Authority Error Elasticsearch Filter](https://discuss.elastic.co/t/illegal-character-authority-error-elasticsearch-filter/190825)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 2\ **Last updated:** [July 16, 2019, 10:17pm UTC](https://discuss.elastic.co/t/illegal-character-authority-error-elasticsearch-filter/190825 "2019-07-16T22:17:31Z")
When performing an Elasticsearch filter query, I get the below error (I've replaced sensitive values with an equivilent number of Xs. \[WARN \]\[logstash.filters.elasticsearch\] Failed to query elasticsearch for previous ev… --- ## [Cannot search for \_source.fields in Kibana](https://discuss.elastic.co/t/cannot-search-for-source-fields-in-kibana/190551)
**Author:** [@Vladpov](https://discuss.elastic.co/u/Vladpov)\ **Replies:** 4\ **Last updated:** [July 16, 2019, 8:10pm UTC](https://discuss.elastic.co/t/cannot-search-for-source-fields-in-kibana/190551 "2019-07-16T20:10:26Z")
Hello everyone, I have a problem with searching for fields that I specified in Logstash filter. In particular I cannot search for syslog\_timestamp that I created in filter.conf file in Logstash. I defined the syslog\_ti… --- ## [Error in sending data from kafka to elasticsearch via logstash](https://discuss.elastic.co/t/error-in-sending-data-from-kafka-to-elasticsearch-via-logstash/189655)
**Author:** [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Replies:** 5\ **Last updated:** [July 16, 2019, 7:33pm UTC](https://discuss.elastic.co/t/error-in-sending-data-from-kafka-to-elasticsearch-via-logstash/189655 "2019-07-16T19:33:46Z")
Hi All, I am trying to send metricbeat data to beats input via graylog. Data is coming to streams and storing it in Elasticsearch index (metricbeat\_0). From the streams I am trying to send data to kafka via Manage Outp… --- ## [Logstash daily indexing?](https://discuss.elastic.co/t/logstash-daily-indexing/190812)
**Author:** [@userR](https://discuss.elastic.co/u/userR)\ **Replies:** 3\ **Last updated:** [July 16, 2019, 6:28pm UTC](https://discuss.elastic.co/t/logstash-daily-indexing/190812 "2019-07-16T18:28:46Z")
Hi, I am planning on using Filebeat for log rotation on a server. From my understanding, Filebeat tails the inode so log rotation shouldn't be an issue when it comes to reading new log data. However, I have a question r… --- ## [Logstash Dead Letter Queue, send raw message to ElasticSearch](https://discuss.elastic.co/t/logstash-dead-letter-queue-send-raw-message-to-elasticsearch/190781)
**Author:** [@Joe\_Still](https://discuss.elastic.co/u/Joe_Still)\ **Replies:** 6\ **Last updated:** [July 16, 2019, 5:28pm UTC](https://discuss.elastic.co/t/logstash-dead-letter-queue-send-raw-message-to-elasticsearch/190781 "2019-07-16T17:28:56Z")
Hello, I am running into an issue where I am sending incorrectly formatted JSON to my logstash such that logstash throws errors about being unable to parse it. I learned about Dead Letter Queue, and now I am hoping some… --- ## [Indexing 100 Million records from postgresdb](https://discuss.elastic.co/t/indexing-100-million-records-from-postgresdb/190778)
**Author:** [@Kevin\_Juliano](https://discuss.elastic.co/u/Kevin_Juliano)\ **Replies:** 2\ **Last updated:** [July 16, 2019, 5:17pm UTC](https://discuss.elastic.co/t/indexing-100-million-records-from-postgresdb/190778 "2019-07-16T17:17:49Z")
I just want to know what specs and configuration do I need to index 100Million+ records, Right now I have free memory of: total used free shared buff/cache available Mem: 64410 2… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=507) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=509)