# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=227 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 228 --- ## [Parse subdomain from fqdn in logstash](https://discuss.elastic.co/t/parse-subdomain-from-fqdn-in-logstash/273535)
**Author:** [@tushar.bansal](https://discuss.elastic.co/u/tushar.bansal)\ **Replies:** 6\ **Last updated:** [May 22, 2021, 11:45pm UTC](https://discuss.elastic.co/t/parse-subdomain-from-fqdn-in-logstash/273535 "2021-05-22T23:45:16Z")
Hi, I have several events coming in logstash with info of web link accessed. example: DNS query is called for the name crl.sectigo.com, type 28 DNS query is called for the name repo.maven.apache.org, type 2 DNS quer… --- ## [How to use if condition to filter spider event gracefully in logstash](https://discuss.elastic.co/t/how-to-use-if-condition-to-filter-spider-event-gracefully-in-logstash/273606)
**Author:** [@stille](https://discuss.elastic.co/u/stille)\ **Replies:** 6\ **Last updated:** [May 22, 2021, 11:39pm UTC](https://discuss.elastic.co/t/how-to-use-if-condition-to-filter-spider-event-gracefully-in-logstash/273606 "2021-05-22T23:39:29Z")
I can use drop event in filebeat to filter spider logs , just like following: - drop\_event: when: or: - contains: message: FacebookBot - contains: message… --- ## [Logstash input/ouput elasticsearch plugin capped performances](https://discuss.elastic.co/t/logstash-input-ouput-elasticsearch-plugin-capped-performances/273686)
**Author:** [@rschirin](https://discuss.elastic.co/u/rschirin)\ **Replies:** 5\ **Last updated:** [May 22, 2021, 11:55am UTC](https://discuss.elastic.co/t/logstash-input-ouput-elasticsearch-plugin-capped-performances/273686 "2021-05-22T11:55:22Z")
Hey There, inside a Logstash pipeline I was using both input and output elasticsearch plugin. Taking a look to Stack monitoring I saw that Event Received Rate (/s) and Event Emitted Rate (/s) are capped to 4000 events/… --- ## [Gradually introducing a portion of traffic to a new logstash configuration](https://discuss.elastic.co/t/gradually-introducing-a-portion-of-traffic-to-a-new-logstash-configuration/273603)
**Author:** [@cknz](https://discuss.elastic.co/u/cknz)\ **Replies:** 2\ **Last updated:** [May 22, 2021, 10:35am UTC](https://discuss.elastic.co/t/gradually-introducing-a-portion-of-traffic-to-a-new-logstash-configuration/273603 "2021-05-22T10:35:04Z")
Hi all, my colleague and I have been working on using the memcached filter plugin. We've got memcached fed with a inventory data about our various network equipment. The goal: I want to label all syslog traffic from our… --- ## [LogStash conf file help](https://discuss.elastic.co/t/logstash-conf-file-help/273705)
**Author:** [@jasieltego](https://discuss.elastic.co/u/jasieltego)\ **Replies:** 6\ **Last updated:** [May 22, 2021, 1:19am UTC](https://discuss.elastic.co/t/logstash-conf-file-help/273705 "2021-05-22T01:19:43Z")
Hi, I have created a logstash conf file. It's just bare bones, trying to get the service up and running so I can practice on my PC. This is what's in the file. \`input { \`stdin {} \`} \`output{ \`elasticsearch { hosts… --- ## [Unable to escape special character ""](https://discuss.elastic.co/t/unable-to-escape-special-character/273474)
**Author:** [@ravi-shanker](https://discuss.elastic.co/u/ravi-shanker)\ **Replies:** 9\ **Last updated:** [May 22, 2021, 12:03am UTC](https://discuss.elastic.co/t/unable-to-escape-special-character/273474 "2021-05-22T00:03:32Z")
I am trying to index my data into elasticsearch cluster through logstash. Currently my data is in aws postgres and i have to move data from postgres to elasticsearch. Below is my logstash config file: input { jdbc { … --- ## [Parse logs](https://discuss.elastic.co/t/parse-logs/273706)
**Author:** [@sergio\_junior](https://discuss.elastic.co/u/sergio_junior)\ **Replies:** 1\ **Last updated:** [May 21, 2021, 10:06pm UTC](https://discuss.elastic.co/t/parse-logs/273706 "2021-05-21T22:06:12Z")
hello guys, i need help. I am trying to separate these values ​​from within the message {"message": "File accessed: \\" / Photos / Readme.md \\ "", "@ version": "1", "tags": \["beats\_input\_codec\_plain\_applied", "\_ jsonpar… --- ## [CSV Input - Some fields are multiline - LR vs CRLF](https://discuss.elastic.co/t/csv-input-some-fields-are-multiline-lr-vs-crlf/273691)
**Author:** [@uklipse](https://discuss.elastic.co/u/uklipse)\ **Replies:** 0\ **Last updated:** [May 21, 2021, 7:28pm UTC](https://discuss.elastic.co/t/csv-input-some-fields-are-multiline-lr-vs-crlf/273691 "2021-05-21T19:28:29Z")
I am trying input a csv file that certain fields can be multiline. When I paste the sample data into Notepad++ and display end of line characters, I can see the actual csv line separates using CRLF but the new lines with… --- ## [Querying API On Input With Variables](https://discuss.elastic.co/t/querying-api-on-input-with-variables/273688)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 1\ **Last updated:** [May 21, 2021, 5:30pm UTC](https://discuss.elastic.co/t/querying-api-on-input-with-variables/273688 "2021-05-21T17:30:33Z")
I am trying to hit a HTTP API with time constraints included. The API allows you to pull results since a specified time in the format of yyyy-MM-dd'T'HH:mm:ss:SSS z. Is there an input that would allow me to specify thi… --- ## [Csv header skip\_header =\> "true" not working](https://discuss.elastic.co/t/csv-header-skip-header-true-not-working/273100)
**Author:** [@Bugsbee](https://discuss.elastic.co/u/Bugsbee)\ **Replies:** 2\ **Last updated:** [May 21, 2021, 4:50pm UTC](https://discuss.elastic.co/t/csv-header-skip-header-true-not-working/273100 "2021-05-21T16:50:31Z")
hello all, Im trying to use skip\_header =\> "true", this is because in kibana/discover the header is also coming out as a value. so i tried to add this parameter and it does not seem to working. In my case, this header… --- ## [Logstash keystore error after upgrade](https://discuss.elastic.co/t/logstash-keystore-error-after-upgrade/273683)
**Author:** [@gutierrezfj](https://discuss.elastic.co/u/gutierrezfj)\ **Replies:** 0\ **Last updated:** [May 21, 2021, 3:38pm UTC](https://discuss.elastic.co/t/logstash-keystore-error-after-upgrade/273683 "2021-05-21T15:38:47Z")
Hi, I did a deployment and upgrade, from logstash 7.6.2 deployed binary to logstash 7.10.2 dockerized. I used the same logstash.keystore file (copy and paste .keystore from 7.6.2 to /usr/share/logstash/config inside co… --- ## [Same input (beats) in multiples config files](https://discuss.elastic.co/t/same-input-beats-in-multiples-config-files/273675)
**Author:** [@mar-ro](https://discuss.elastic.co/u/mar-ro)\ **Replies:** 0\ **Last updated:** [May 21, 2021, 2:48pm UTC](https://discuss.elastic.co/t/same-input-beats-in-multiples-config-files/273675 "2021-05-21T14:48:06Z")
Good afternoon, I understand that in logstash you cannot have more than one file in /etc/logstash/config.d with the same input (filebeat) ...in this case, if you want to have different files that will receive data from f… --- ## [GROK URIPATHPARAM SPLIT](https://discuss.elastic.co/t/grok-uripathparam-split/273662)
**Author:** [@Rahul\_Aggarwal](https://discuss.elastic.co/u/Rahul_Aggarwal)\ **Replies:** 0\ **Last updated:** [May 21, 2021, 12:00pm UTC](https://discuss.elastic.co/t/grok-uripathparam-split/273662 "2021-05-21T12:00:57Z")
I've been searching and reading the documentation but I can't seem to find further split my URIPATH into multiple parts. How this will achieve? My Sample URIPATH's are below: /article/india/13-naxals-were-neutralized-… --- ## [Grok - Value could appear or not. multiple rules?](https://discuss.elastic.co/t/grok-value-could-appear-or-not-multiple-rules/273548)
**Author:** [@legolas\_bilbao](https://discuss.elastic.co/u/legolas_bilbao)\ **Replies:** 2\ **Last updated:** [May 21, 2021, 12:45pm UTC](https://discuss.elastic.co/t/grok-value-could-appear-or-not-multiple-rules/273548 "2021-05-21T12:45:16Z")
I'm working with some logs from squid but one value could appear or not 1621232309.575 1 172.99.6.74 TCP\_MEM\_HIT/206 3096 GET http://storage.googleapis.com/ - HIER\_NONE/- application/octet-stream 1621232309.575 … --- ## [Log file doesnt read](https://discuss.elastic.co/t/log-file-doesnt-read/273665)
**Author:** [@legolas\_bilbao](https://discuss.elastic.co/u/legolas_bilbao)\ **Replies:** 0\ **Last updated:** [May 21, 2021, 12:33pm UTC](https://discuss.elastic.co/t/log-file-doesnt-read/273665 "2021-05-21T12:33:19Z")
HI everybody, I wanna read and deploy data from file logs but when i started the logstash service appears the following data Logstash.log \[2021-05-21T13:50:38,097\]\[INFO \]\[logstash.inputs.file \]\[squid\] No sincedb\_… --- ## [How can i schedule logstash every two hours for jdbc input plugin?](https://discuss.elastic.co/t/how-can-i-schedule-logstash-every-two-hours-for-jdbc-input-plugin/272973)
**Author:** [@sreedhar1](https://discuss.elastic.co/u/sreedhar1)\ **Replies:** 5\ **Last updated:** [May 21, 2021, 10:40am UTC](https://discuss.elastic.co/t/how-can-i-schedule-logstash-every-two-hours-for-jdbc-input-plugin/272973 "2021-05-21T10:40:01Z")
How can i schedule logstash every two hours for jdbc input plugin? --- ## [Logstash input s3 Time difference in data acceptance](https://discuss.elastic.co/t/logstash-input-s3-time-difference-in-data-acceptance/273616)
**Author:** [@xiaoloutingfengyu](https://discuss.elastic.co/u/xiaoloutingfengyu)\ **Replies:** 0\ **Last updated:** [May 21, 2021, 6:26am UTC](https://discuss.elastic.co/t/logstash-input-s3-time-difference-in-data-acceptance/273616 "2021-05-21T06:26:37Z")
Logstash receives data from aws-s3, but kibana data is dozens of minutes later than the data on S3. How to modify it? --- ## [Good practices with Beats and Logstash](https://discuss.elastic.co/t/good-practices-with-beats-and-logstash/273296)
**Author:** [@mar-ro](https://discuss.elastic.co/u/mar-ro)\ **Replies:** 2\ **Last updated:** [May 21, 2021, 6:15am UTC](https://discuss.elastic.co/t/good-practices-with-beats-and-logstash/273296 "2021-05-21T06:15:48Z")
Hello to all of you! I have the following question about which option is better to be able to register two types of logs in elastic search. Let's say for example that I want to log the nginx and crontab logs. In addit… --- ## [Collecting syslog from several different vendors](https://discuss.elastic.co/t/collecting-syslog-from-several-different-vendors/273497)
**Author:** [@Christer\_Palmen](https://discuss.elastic.co/u/Christer_Palmen)\ **Replies:** 2\ **Last updated:** [May 21, 2021, 3:39am UTC](https://discuss.elastic.co/t/collecting-syslog-from-several-different-vendors/273497 "2021-05-21T03:39:44Z")
Hello. We have got an task of setting up an Elasticstack server that collects as a start syslog from different vendors in their environment, visualize it in Kibana and forward the syslogs to a CERT department As a sta… --- ## [Logstash reads input files but does not write any output](https://discuss.elastic.co/t/logstash-reads-input-files-but-does-not-write-any-output/273575)
**Author:** [@fmanfredi](https://discuss.elastic.co/u/fmanfredi)\ **Replies:** 2\ **Last updated:** [May 20, 2021, 7:14pm UTC](https://discuss.elastic.co/t/logstash-reads-input-files-but-does-not-write-any-output/273575 "2021-05-20T19:14:20Z")
Hi, I'm having problems with logstash. Looks like it is capable of reaching and reading the files I want it to read but there is no sign of processing and output. I'm trying to write both to elasticsearch and stdout bu… --- ## [Logstash: input file plugin and tcp plugins](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058)
**Author:** [@hsalim](https://discuss.elastic.co/u/hsalim)\ **Replies:** 31\ **Last updated:** [May 20, 2021, 7:13pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058 "2021-05-20T19:13:40Z")
I have 4 pipelines. 3 using tcp plugin and 1 using file plugin. pipeline with input file plugin works fine independently, but when i add 3 other tcp plugin pipeline, the input file plugin pipeline stops running (after s… --- ## [Load more than 1 million row from SQL Server to ES with Logstash JDBC](https://discuss.elastic.co/t/load-more-than-1-million-row-from-sql-server-to-es-with-logstash-jdbc/273563)
**Author:** [@Dai\_Thai\_Hoa\_Vo](https://discuss.elastic.co/u/Dai_Thai_Hoa_Vo)\ **Replies:** 1\ **Last updated:** [May 20, 2021, 6:30pm UTC](https://discuss.elastic.co/t/load-more-than-1-million-row-from-sql-server-to-es-with-logstash-jdbc/273563 "2021-05-20T18:30:56Z")
Hi every one. I have one case that my table has more 1 million rows in SQL Server and I used logstash JDBC to load them to Elastic Search. It work fine but run very slow. This is my file config. input { jdbc { jdbc… --- ## [Logstash input plugin](https://discuss.elastic.co/t/logstash-input-plugin/273574)
**Author:** [@hsalim](https://discuss.elastic.co/u/hsalim)\ **Replies:** 0\ **Last updated:** [May 20, 2021, 6:29pm UTC](https://discuss.elastic.co/t/logstash-input-plugin/273574 "2021-05-20T18:29:56Z")
Has anyone experienced a behavior where logstash is not discovering new files and incorrectly marking new discovery as false? \[2021-05-20T18:20:02,794\]\[TRACE\]\[filewatch.discoverer \] discover\_files {"count"=\>2254} \[2… --- ## [Logstash mutate remove field name 'JSON'](https://discuss.elastic.co/t/logstash-mutate-remove-field-name-json/273572)
**Author:** [@raghav\_krishna\_agarw](https://discuss.elastic.co/u/raghav_krishna_agarw)\ **Replies:** 0\ **Last updated:** [May 20, 2021, 6:19pm UTC](https://discuss.elastic.co/t/logstash-mutate-remove-field-name-json/273572 "2021-05-20T18:19:11Z")
I am trying to remove a field name "JSON" in my mongoDB record, while dumping in ELK. But it is not working, may be the case of reserve keyword. Below is the document structure: "date" : \, "name" : "xyz", "JSON" … --- ## [Saving a specific array element in Logstash](https://discuss.elastic.co/t/saving-a-specific-array-element-in-logstash/273556)
**Author:** [@tumbledwyer](https://discuss.elastic.co/u/tumbledwyer)\ **Replies:** 2\ **Last updated:** [May 20, 2021, 4:47pm UTC](https://discuss.elastic.co/t/saving-a-specific-array-element-in-logstash/273556 "2021-05-20T16:47:12Z")
Hi I am receiving a JSON object with an array property. I would like to search the array and save only the element that matches my criteria. My input looks like this: { "identifier": \[ { "system" : "Source1", "va… --- ## [LogStash::ConvergeResult::FailedAction while tryting to use pipeline output](https://discuss.elastic.co/t/logstash-failedaction-while-tryting-to-use-pipeline-output/273564)
**Author:** [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Replies:** 1\ **Last updated:** [May 20, 2021, 4:30pm UTC](https://discuss.elastic.co/t/logstash-failedaction-while-tryting-to-use-pipeline-output/273564 "2021-05-20T16:30:00Z")
Hi folks, this is my test.conf: input { beats { port =\> 6150 } } output { if \[agent\]\[type\] == "metricbeat" { elasticsearch { ... } } else if \[fields\]\[log\_type\] == "audit" { elasticsearch { … --- ## [There is insufficient memory for the Java Runtime Environment to continue while using --config.test\_and\_exit](https://discuss.elastic.co/t/there-is-insufficient-memory-for-the-java-runtime-environment-to-continue-while-using-config-test-and-exit/273561)
**Author:** [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Replies:** 0\ **Last updated:** [May 20, 2021, 3:42pm UTC](https://discuss.elastic.co/t/there-is-insufficient-memory-for-the-java-runtime-environment-to-continue-while-using-config-test-and-exit/273561 "2021-05-20T15:42:59Z")
When I try to test the config of my logstash pipeline I run into this error: \[bash\]$ logstash --config.test\_and\_exit -f /path/to/config.conf OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in … --- ## [Multiple Logstash outputs vs pipeline-to-pipeline communcations](https://discuss.elastic.co/t/multiple-logstash-outputs-vs-pipeline-to-pipeline-communcations/273546)
**Author:** [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Replies:** 0\ **Last updated:** [May 20, 2021, 1:24pm UTC](https://discuss.elastic.co/t/multiple-logstash-outputs-vs-pipeline-to-pipeline-communcations/273546 "2021-05-20T13:24:25Z")
Hi folks, I have following use case: I want to create one index for each kubernetes namespace (and its own lifecycle policy) and I want to use the ILM. As far as I understand I have two approaches: I have one pipelin… --- ## [From SQL Server Management Studio to Elastic Search: config file](https://discuss.elastic.co/t/from-sql-server-management-studio-to-elastic-search-config-file/273544)
**Author:** [@qttv](https://discuss.elastic.co/u/qttv)\ **Replies:** 0\ **Last updated:** [May 20, 2021, 1:09pm UTC](https://discuss.elastic.co/t/from-sql-server-management-studio-to-elastic-search-config-file/273544 "2021-05-20T13:09:03Z")
Good Evening. I'm new on the ELK stack and for the first time I'm trying to import some data from SQL Server Manamenet Studio database to the ElasticSearch. I know there is the necessity to download a driver to use the… --- ## [Logstash pipeline problem (logstash not appearing in kibana)](https://discuss.elastic.co/t/logstash-pipeline-problem-logstash-not-appearing-in-kibana/273429)
**Author:** [@farciarz121](https://discuss.elastic.co/u/farciarz121)\ **Replies:** 5\ **Last updated:** [May 20, 2021, 11:59am UTC](https://discuss.elastic.co/t/logstash-pipeline-problem-logstash-not-appearing-in-kibana/273429 "2021-05-20T11:59:38Z")
Hi guys, thank you for having me on this forum. I am trying to set up ELK for the first time. I have followed installation instructions from elastic.co with numbers of videos on yt and I have finally installed ELK. The… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=226) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=228)