# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=204 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 205 --- ## [Pulling more than 2000 records from ServiceNow api using http\_poller](https://discuss.elastic.co/t/pulling-more-than-2000-records-from-servicenow-api-using-http-poller/280840)
**Author:** [@Purushottam22](https://discuss.elastic.co/u/Purushottam22)\ **Replies:** 2\ **Last updated:** [August 10, 2021, 3:08am UTC](https://discuss.elastic.co/t/pulling-more-than-2000-records-from-servicenow-api-using-http-poller/280840 "2021-08-10T03:08:46Z")
I have created the pipeline using http\_poller to pull the serviceNow CI records. To pull more than 2000 records, i am hitting the ServiceNow api multiple time with setting up start and count values using http\_poller in… --- ## [Is Logstash JDBC tracking\_column meant to work with BIGINT?](https://discuss.elastic.co/t/is-logstash-jdbc-tracking-column-meant-to-work-with-bigint/280856)
**Author:** [@catmanjan](https://discuss.elastic.co/u/catmanjan)\ **Replies:** 1\ **Last updated:** [August 10, 2021, 1:50am UTC](https://discuss.elastic.co/t/is-logstash-jdbc-tracking-column-meant-to-work-with-bigint/280856 "2021-08-10T01:50:14Z")
I have a Logstash JDBC configuration with tracking\_column set to a BIGINT of the format YYYYMMDDhhmmsss This is obviously a very big number and the value set in last\_run\_metadata\_path is always set to zero. If I use a … --- ## [Logstash Http Poller Method Head](https://discuss.elastic.co/t/logstash-http-poller-method-head/280855)
**Author:** [@gutierrezfj](https://discuss.elastic.co/u/gutierrezfj)\ **Replies:** 1\ **Last updated:** [August 10, 2021, 1:18am UTC](https://discuss.elastic.co/t/logstash-http-poller-method-head/280855 "2021-08-10T01:18:59Z")
I'm trying to retrieve data from an CUSTOM API which only allows HEAD requests, consequently the result is an empty body. where could the header response data be? is it possible to capture the data from a response head… --- ## [Logstash secuirty](https://discuss.elastic.co/t/logstash-secuirty/280854)
**Author:** [@Rickenson\_Robert](https://discuss.elastic.co/u/Rickenson_Robert)\ **Replies:** 1\ **Last updated:** [August 9, 2021, 11:42pm UTC](https://discuss.elastic.co/t/logstash-secuirty/280854 "2021-08-09T23:42:17Z")
After enabling xpack in the elasticsearch yml and setting passwords for elasticsearch I get the following error when trying to index into my elasticsearch cluster using logstash: BadResponseCodeError, :message=\>"Got res… --- ## [Logstash - grokfilter is successful using grokdebug, but fails in live](https://discuss.elastic.co/t/logstash-grokfilter-is-successful-using-grokdebug-but-fails-in-live/280829)
**Author:** [@One\_Punch](https://discuss.elastic.co/u/One_Punch)\ **Replies:** 13\ **Last updated:** [August 9, 2021, 7:50pm UTC](https://discuss.elastic.co/t/logstash-grokfilter-is-successful-using-grokdebug-but-fails-in-live/280829 "2021-08-09T19:50:05Z")
Hello, im integratin filebeats and logstash, sample grok pattern works in grokdebug but throwing grokfailure in logstash Message: \[2m2021-08-09 15:50:07.850 \[0;39m \[32m INFO \[0;39m \[35m1 \[0;39m \[2m--- \[0;39m \[2m\[ni… --- ## [Logstash pipeline error at line 1 column 1](https://discuss.elastic.co/t/logstash-pipeline-error-at-line-1-column-1/280746)
**Author:** [@panagiss](https://discuss.elastic.co/u/panagiss)\ **Replies:** 4\ **Last updated:** [August 9, 2021, 4:05pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-at-line-1-column-1/280746 "2021-08-09T16:05:57Z")
I get this error: \[2021-08-08T20:15:12,553\]\[INFO \]\[logstash.agent \] Successfully started Logstash API endpoint {:port=\>9600} \[2021-08-08T20:15:13,441\]\[ERROR\]\[logstash.agent \] Failed to execute action… --- ## [Size of an index is very less than the raw document size when ingested via logstash](https://discuss.elastic.co/t/size-of-an-index-is-very-less-than-the-raw-document-size-when-ingested-via-logstash/280812)
**Author:** [@madhanbaskar](https://discuss.elastic.co/u/madhanbaskar)\ **Replies:** 3\ **Last updated:** [August 9, 2021, 2:23pm UTC](https://discuss.elastic.co/t/size-of-an-index-is-very-less-than-the-raw-document-size-when-ingested-via-logstash/280812 "2021-08-09T14:23:23Z")
I have a JSON file which has 93k records in it. The size of the JSON file is 413MB. When Indexed via logstash it is just 88MB in the index. I could see all the records ingested. Index setting : I have 50 fields for ind… --- ## [Gsub doesn't replace the given char](https://discuss.elastic.co/t/gsub-doesnt-replace-the-given-char/280783)
**Author:** [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)\ **Replies:** 1\ **Last updated:** [August 9, 2021, 1:02pm UTC](https://discuss.elastic.co/t/gsub-doesnt-replace-the-given-char/280783 "2021-08-09T13:02:43Z")
Hi Team, I'm trying to replace a char with another char in a string field, but it doesn't looks like it is doing the job, please is there anyone to help me in correcting this. Here is an example of what I'm doing , "s… --- ## [How to scrape a page with logstash](https://discuss.elastic.co/t/how-to-scrape-a-page-with-logstash/279892)
**Author:** [@francieliton\_araujo](https://discuss.elastic.co/u/francieliton_araujo)\ **Replies:** 3\ **Last updated:** [August 9, 2021, 11:26am UTC](https://discuss.elastic.co/t/how-to-scrape-a-page-with-logstash/279892 "2021-08-09T11:26:11Z")
We have an application that does not save to the database in real time, however it has a page that has some information in real time, I would like to access and scrape this information and put it inside elasticseach. ob… --- ## [Gem/Bundler errors while installing a new logstash filter](https://discuss.elastic.co/t/gem-bundler-errors-while-installing-a-new-logstash-filter/280797)
**Author:** [@lionaneesh](https://discuss.elastic.co/u/lionaneesh)\ **Replies:** 0\ **Last updated:** [August 9, 2021, 11:23am UTC](https://discuss.elastic.co/t/gem-bundler-errors-while-installing-a-new-logstash-filter/280797 "2021-08-09T11:23:01Z")
I am trying to install the logstash-filter-goaudit plugin on my kubernetes pod running logstash image: docker.elastic.co/logstash/logstash-oss:6.4.1 . We are still on ES 6.6 and am currently trying to onboard goaudit log… --- ## [Load balancing data from Filebeat to Logstash using Nginx](https://discuss.elastic.co/t/load-balancing-data-from-filebeat-to-logstash-using-nginx/179473)
**Author:** [@Nikhil04](https://discuss.elastic.co/u/Nikhil04)\ **Replies:** 8\ **Last updated:** [August 9, 2021, 9:35am UTC](https://discuss.elastic.co/t/load-balancing-data-from-filebeat-to-logstash-using-nginx/179473 "2021-08-09T09:35:16Z")
Hello, I have ELK cluster with 3 nodes for elasticsearch(master+data) , 2 nodes for logstash(Active and Passive) and 2 for kibana.I have to utilize both the logstash nodes. I am using filebeat to parse logs and send da… --- ## [Setup logstash module netflow](https://discuss.elastic.co/t/setup-logstash-module-netflow/279528)
**Author:** [@Muhammed\_Ibrahim](https://discuss.elastic.co/u/Muhammed_Ibrahim)\ **Replies:** 1\ **Last updated:** [August 9, 2021, 8:16am UTC](https://discuss.elastic.co/t/setup-logstash-module-netflow/279528 "2021-08-09T08:16:46Z")
Dears, I need to install netflow dashboard as it's shown in elastic guide by this command : sudo bin/logstash --modules netflow --setup --path.settings /etc/logstash/ but appears this error below : Using bundled JDK… --- ## [Synchronize whole files (mostly PDFs) to ES from directory and e-mails](https://discuss.elastic.co/t/synchronize-whole-files-mostly-pdfs-to-es-from-directory-and-e-mails/279962)
**Author:** [@jporzelt](https://discuss.elastic.co/u/jporzelt)\ **Replies:** 3\ **Last updated:** [August 9, 2021, 8:08am UTC](https://discuss.elastic.co/t/synchronize-whole-files-mostly-pdfs-to-es-from-directory-and-e-mails/279962 "2021-08-09T08:08:54Z")
Hi all, we would like to setup a document synchronization from a file directory and from a e-mail imap folder to elasticsearch. For the e-mails we are interested in the attachments. Here the imap input plugin looks pro… --- ## [Logstash stopped to treat the incoming logs](https://discuss.elastic.co/t/logstash-stopped-to-treat-the-incoming-logs/280402)
**Author:** [@tarekilani](https://discuss.elastic.co/u/tarekilani)\ **Replies:** 1\ **Last updated:** [August 9, 2021, 7:59am UTC](https://discuss.elastic.co/t/logstash-stopped-to-treat-the-incoming-logs/280402 "2021-08-09T07:59:15Z")
Hello, I'm using ELK stack to correlate logs of network devices. The stack was perfectly working for a long period of time. But today logstash stopped treating the incoming logs. When i did an investigation i found thi… --- ## [Logstash geoip databasemanager with outgoing proxy](https://discuss.elastic.co/t/logstash-geoip-databasemanager-with-outgoing-proxy/280751)
**Author:** [@cknz](https://discuss.elastic.co/u/cknz)\ **Replies:** 2\ **Last updated:** [August 9, 2021, 7:47am UTC](https://discuss.elastic.co/t/logstash-geoip-databasemanager-with-outgoing-proxy/280751 "2021-08-09T07:47:32Z")
Hi all, I've upgraded to Logstash 7.14 from 7.11, and now I'm having to deal with the geoip databasemanager. My servers do not have direct database access and need to go out via proxy. Current observed behaviour is that… --- ## [How logstash uses add\_field to add a JSON array](https://discuss.elastic.co/t/how-logstash-uses-add-field-to-add-a-json-array/280640)
**Author:** [@ktpktr0](https://discuss.elastic.co/u/ktpktr0)\ **Replies:** 2\ **Last updated:** [August 9, 2021, 7:27am UTC](https://discuss.elastic.co/t/how-logstash-uses-add-field-to-add-a-json-array/280640 "2021-08-09T07:27:11Z")
The raw data are as follows "money快照" =\> \[ \[0\] "money:14683689,value:15105589,time:2021-07-27 23:50:54", \[1\] "money:14683689,value:15105589,time:2021-07-27 23:50:56" \], The confi… --- ## [Logstash not listening on beats input port](https://discuss.elastic.co/t/logstash-not-listening-on-beats-input-port/280559)
**Author:** [@r.saigiridhar](https://discuss.elastic.co/u/r.saigiridhar)\ **Replies:** 1\ **Last updated:** [August 9, 2021, 5:02am UTC](https://discuss.elastic.co/t/logstash-not-listening-on-beats-input-port/280559 "2021-08-09T05:02:27Z")
Logstash server is not listening on the beats input port specified. I have installed beats input plugin in logstash server, created a conf file with input port specified and pipelines file that has an entry for the conf. … --- ## [Parse a CSV file with grok logstash](https://discuss.elastic.co/t/parse-a-csv-file-with-grok-logstash/280726)
**Author:** [@Hind](https://discuss.elastic.co/u/Hind)\ **Replies:** 3\ **Last updated:** [August 8, 2021, 10:32pm UTC](https://discuss.elastic.co/t/parse-a-csv-file-with-grok-logstash/280726 "2021-08-08T22:32:19Z")
Hello! How could I parse a string that is between double quotes with grok logstash. "Kaathadimattam, Balacola Post, NEAR Siva Tea Factory, Ooty, 643203 Ooty, India – Great location -",ooty,India,..... thank you. --- ## [Array of objects - LogStash Conf](https://discuss.elastic.co/t/array-of-objects-logstash-conf/280750)
**Author:** [@me.mohammed](https://discuss.elastic.co/u/me.mohammed)\ **Replies:** 1\ **Last updated:** [August 8, 2021, 10:08pm UTC](https://discuss.elastic.co/t/array-of-objects-logstash-conf/280750 "2021-08-08T22:08:31Z")
I'm currently working on fetching my data from MySQL and pushing it to ElasticSearch through LogStash, although I'm kind of having an issue with the logstash configuration I basically have Products and each of those pro… --- ## [Parsing a Raw JSON file "Invalid Setting for json filter"](https://discuss.elastic.co/t/parsing-a-raw-json-file-invalid-setting-for-json-filter/280735)
**Author:** [@Wilks](https://discuss.elastic.co/u/Wilks)\ **Replies:** 4\ **Last updated:** [August 8, 2021, 7:38pm UTC](https://discuss.elastic.co/t/parsing-a-raw-json-file-invalid-setting-for-json-filter/280735 "2021-08-08T19:38:58Z")
Hi, I am trying to parse a simple raw JSON file but I am getting an error about invalid setting for JSON filter. I am wondering if its the way my filter is setup and the fact that its nested json? \[ERROR\] 2021-08-08 1… --- ## [Http\_poller skip SSL validation](https://discuss.elastic.co/t/http-poller-skip-ssl-validation/280713)
**Author:** [@panagiss](https://discuss.elastic.co/u/panagiss)\ **Replies:** 3\ **Last updated:** [August 8, 2021, 6:41pm UTC](https://discuss.elastic.co/t/http-poller-skip-ssl-validation/280713 "2021-08-08T18:41:22Z")
Is there a way to skip SSL server certificate validation? I'm trying to hit a local server with a self-signed cert but i get errors of course. I saw for other plugins that this was added as an option but nothing for ht… --- ## [Logstash crashes when I run .bat file from powershell](https://discuss.elastic.co/t/logstash-crashes-when-i-run-bat-file-from-powershell/280245)
**Author:** [@realj4ke](https://discuss.elastic.co/u/realj4ke)\ **Replies:** 4\ **Last updated:** [August 8, 2021, 4:59pm UTC](https://discuss.elastic.co/t/logstash-crashes-when-i-run-bat-file-from-powershell/280245 "2021-08-08T16:59:05Z")
Below is the error that I get when I try to run logstash after downloading the .zip file from the official website. .\\logstash.bat -f logstash.conf Using JAVA\_HOME defined java: C:\\P… --- ## [Logstash not seeing updates in logs by spring boot](https://discuss.elastic.co/t/logstash-not-seeing-updates-in-logs-by-spring-boot/280717)
**Author:** [@Kapil\_Soni](https://discuss.elastic.co/u/Kapil_Soni)\ **Replies:** 0\ **Last updated:** [August 7, 2021, 8:58pm UTC](https://discuss.elastic.co/t/logstash-not-seeing-updates-in-logs-by-spring-boot/280717 "2021-08-07T20:58:33Z")
New Logs generated by Spring Boot is not updating in Kibana. Logstash =\> ES =\> Kibana I'm using spring logback.xml with RollingFileAppender & SizeAndTimeBasedRollingPolicy to generate log file. Here is my logstash.con… --- ## [How to give hidden file path in input.conf for file input plugin](https://discuss.elastic.co/t/how-to-give-hidden-file-path-in-input-conf-for-file-input-plugin/280617)
**Author:** [@priyankab](https://discuss.elastic.co/u/priyankab)\ **Replies:** 2\ **Last updated:** [August 7, 2021, 3:02am UTC](https://discuss.elastic.co/t/how-to-give-hidden-file-path-in-input-conf-for-file-input-plugin/280617 "2021-08-07T03:02:00Z")
input{ file { exclude =\> "\*.gz" path =\> "/home/abc/work/.productLogs/.auditLogs/.auditLog" type =\> "audit" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" stat\_interval =\> 5 ignore\_older =\> 0 } } --- ## [How to enrich a set of linked documents with a field value from one document?](https://discuss.elastic.co/t/how-to-enrich-a-set-of-linked-documents-with-a-field-value-from-one-document/280673)
**Author:** [@Micah\_Hunsberger](https://discuss.elastic.co/u/Micah_Hunsberger)\ **Replies:** 0\ **Last updated:** [August 6, 2021, 5:12pm UTC](https://discuss.elastic.co/t/how-to-enrich-a-set-of-linked-documents-with-a-field-value-from-one-document/280673 "2021-08-06T17:12:06Z")
I have several log sources (files) that are being ingested by filebeat. There will be several (say, 5-20) log lines that are all linked by a single field (I'll call it the transaction id). In just one of the many logs th… --- ## [Can I pass an array into a plugin through a field](https://discuss.elastic.co/t/can-i-pass-an-array-into-a-plugin-through-a-field/280672)
**Author:** [@R.L](https://discuss.elastic.co/u/R.L)\ **Replies:** 0\ **Last updated:** [August 6, 2021, 4:50pm UTC](https://discuss.elastic.co/t/can-i-pass-an-array-into-a-plugin-through-a-field/280672 "2021-08-06T16:50:48Z")
I am currently writing a custom java plugin for Logstash. One of the input settings is an array. This array might be set in another field within the logstash event. so basically I want to be able to do this: #array\_f… --- ## [Trying to Get Grok Pattern for Error.log](https://discuss.elastic.co/t/trying-to-get-grok-pattern-for-error-log/280670)
**Author:** [@shailesh](https://discuss.elastic.co/u/shailesh)\ **Replies:** 0\ **Last updated:** [August 6, 2021, 4:37pm UTC](https://discuss.elastic.co/t/trying-to-get-grok-pattern-for-error-log/280670 "2021-08-06T16:37:33Z")
Hello Guts - i am trying to write a Grok Pattern for the below log file 20210621.00h02m29s RESPONSE: sent 127.0.0.1 status 502 (Tunnel Connection Failed) for 'xxxx.com:443/ for the above log line i am trying to parse … --- ## [Logstash cannot compile the .conf file](https://discuss.elastic.co/t/logstash-cannot-compile-the-conf-file/279735)
**Author:** [@Adriann](https://discuss.elastic.co/u/Adriann)\ **Replies:** 7\ **Last updated:** [August 6, 2021, 2:21pm UTC](https://discuss.elastic.co/t/logstash-cannot-compile-the-conf-file/279735 "2021-08-06T14:21:52Z")
Hello, so I have this weird situation that all my logstash config files just stopped working. I have noticed it started probably like a few weeks ago after something happened. Only one thing I recognize I have done that… --- ## [Unused visualizations](https://discuss.elastic.co/t/unused-visualizations/280624)
**Author:** [@artobstrel](https://discuss.elastic.co/u/artobstrel)\ **Replies:** 0\ **Last updated:** [August 6, 2021, 8:37am UTC](https://discuss.elastic.co/t/unused-visualizations/280624 "2021-08-06T08:37:20Z")
I want to remove all visualizations that are not used on dashboards. How can I get a list of all such visualizations through api or in some other way? Example Used visualization has dashboard parent: Unused visual… --- ## [JDBC streaming plugin, variable statement](https://discuss.elastic.co/t/jdbc-streaming-plugin-variable-statement/280630)
**Author:** [@JFO\_90](https://discuss.elastic.co/u/JFO_90)\ **Replies:** 0\ **Last updated:** [August 6, 2021, 9:06am UTC](https://discuss.elastic.co/t/jdbc-streaming-plugin-variable-statement/280630 "2021-08-06T09:06:18Z")
Hi, Im using elasticsearch input plugin to obtain info from elastic, with this extracted info i use the JDBC streaming plugin to get information from BD with some fields that i extracted from elastic. The problem that … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=203) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=205)