# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=125 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 126 --- ## [Dynamic URL input in Logstash](https://discuss.elastic.co/t/dynamic-url-input-in-logstash/310605)
**Author:** [@Elie](https://discuss.elastic.co/u/Elie)\ **Replies:** 0\ **Last updated:** [July 26, 2022, 7:55am UTC](https://discuss.elastic.co/t/dynamic-url-input-in-logstash/310605 "2022-07-26T07:55:46Z")
Hello, I am trying to insert a URL as an input in Logstash, but the content of the URL is too large for Logstash to take in all at once with http\_poller (memory error), so what I can do is add arguments such as "PageNo"… --- ## [Ingest Logstash Pipelines for Multiple Applications](https://discuss.elastic.co/t/ingest-logstash-pipelines-for-multiple-applications/310601)
**Author:** [@amseshadri](https://discuss.elastic.co/u/amseshadri)\ **Replies:** 0\ **Last updated:** [July 26, 2022, 6:17am UTC](https://discuss.elastic.co/t/ingest-logstash-pipelines-for-multiple-applications/310601 "2022-07-26T06:17:08Z")
Hi, I started exploring at how to separate multiple applications using Ingest Logstash Pipelines.. so that better manage the access to users or spaces based on the application. Could you provide me some steps or links … --- ## [Relation input jdbc POSTGRES logstash](https://discuss.elastic.co/t/relation-input-jdbc-postgres-logstash/310458)
**Author:** [@5k\_pwc](https://discuss.elastic.co/u/5k_pwc)\ **Replies:** 0\ **Last updated:** [July 23, 2022, 4:18pm UTC](https://discuss.elastic.co/t/relation-input-jdbc-postgres-logstash/310458 "2022-07-23T16:18:27Z")
Hello everyone, It seems im missing something, but couldn't found anything related to this. Im using jdbc input plugin to collect some data from postgresql db. e Elastic Common Schema. Once ECS v8 and an updated releas… --- ## [Output to rabbitMQ does not honor reject-publish](https://discuss.elastic.co/t/output-to-rabbitmq-does-not-honor-reject-publish/310565)
**Author:** [@sylvain.mazet](https://discuss.elastic.co/u/sylvain.mazet)\ **Replies:** 0\ **Last updated:** [July 25, 2022, 5:48pm UTC](https://discuss.elastic.co/t/output-to-rabbitmq-does-not-honor-reject-publish/310565 "2022-07-25T17:48:37Z")
Hi all, I need help on rabbit output plugin. We are pushing to rabbit with logstash. For data integrity reasons, we want rabbit to apply back pressure on logstash. When rabbit is filling up (because our platform is un… --- ## [Logstash how to handle null value with jdbc\_streaming](https://discuss.elastic.co/t/logstash-how-to-handle-null-value-with-jdbc-streaming/310533)
**Author:** [@jsauvebois](https://discuss.elastic.co/u/jsauvebois)\ **Replies:** 0\ **Last updated:** [July 25, 2022, 12:09pm UTC](https://discuss.elastic.co/t/logstash-how-to-handle-null-value-with-jdbc-streaming/310533 "2022-07-25T12:09:24Z")
Hi All, I use filter jdbc\_streaming because I've to extract data from two distinct database based on one column here o\_id (similar to left outer join in SQL) input { jdbc{ statement =\> "select col1,o\_id from tabl… --- ## [When the logstash instance is closed (exit from terminal by Ctrl+C) and is restarted again,the logs gets duplicated in the elasticsearch](https://discuss.elastic.co/t/when-the-logstash-instance-is-closed-exit-from-terminal-by-ctrl-c-and-is-restarted-again-the-logs-gets-duplicated-in-the-elasticsearch/310479)
**Author:** [@bbkunbi](https://discuss.elastic.co/u/bbkunbi)\ **Replies:** 4\ **Last updated:** [July 25, 2022, 11:38am UTC](https://discuss.elastic.co/t/when-the-logstash-instance-is-closed-exit-from-terminal-by-ctrl-c-and-is-restarted-again-the-logs-gets-duplicated-in-the-elasticsearch/310479 "2022-07-25T11:38:52Z")
What to add in .conf file to avoid duplicate entry problem? --- ## [Dissect file pattern that is not always available](https://discuss.elastic.co/t/dissect-file-pattern-that-is-not-always-available/310387)
**Author:** [@mikhatanu](https://discuss.elastic.co/u/mikhatanu)\ **Replies:** 6\ **Last updated:** [July 25, 2022, 5:51am UTC](https://discuss.elastic.co/t/dissect-file-pattern-that-is-not-always-available/310387 "2022-07-25T05:51:00Z")
Hello, i have a txt that looks like this: ltm classification application app\_name { application-id 7974 category Entertainment description "this app is popular. (tcp/http/ssl)" predefined yes risk 3 } ltm classification… --- ## [JDBC multiple inputs asynchronous?](https://discuss.elastic.co/t/jdbc-multiple-inputs-asynchronous/310294)
**Author:** [@bhumikadave](https://discuss.elastic.co/u/bhumikadave)\ **Replies:** 1\ **Last updated:** [July 25, 2022, 4:49am UTC](https://discuss.elastic.co/t/jdbc-multiple-inputs-asynchronous/310294 "2022-07-25T04:49:46Z")
I am wondering if using multiple JDBC inputs in one logstash is advisable. Does it execute the inputs asynchronously or sequentially? Thanks! --- ## [Error Ignoring the 'pipelines.yml' file because modules or command line options are specified](https://discuss.elastic.co/t/error-ignoring-the-pipelines-yml-file-because-modules-or-command-line-options-are-specified/310322)
**Author:** [@Khadija\_BOUDINAR1](https://discuss.elastic.co/u/Khadija_BOUDINAR1)\ **Replies:** 1\ **Last updated:** [July 24, 2022, 2:30am UTC](https://discuss.elastic.co/t/error-ignoring-the-pipelines-yml-file-because-modules-or-command-line-options-are-specified/310322 "2022-07-24T02:30:09Z")
when i chargin my postgres in elastic with my file.conf error like:Ignoring the 'pipelines.yml' file because modules or command line options are specified any suggestion please ? input { jdbc { jdbc\_connection\_string… --- ## [AGPL v3 license in logstash dependency](https://discuss.elastic.co/t/agpl-v3-license-in-logstash-dependency/310372)
**Author:** [@daniele8](https://discuss.elastic.co/u/daniele8)\ **Replies:** 1\ **Last updated:** [July 23, 2022, 5:49pm UTC](https://discuss.elastic.co/t/agpl-v3-license-in-logstash-dependency/310372 "2022-07-23T17:49:38Z")
We would like to include logstash in our commercial projects but we realized it comes with a dependency (flores 0.0.7) that has an AGPL v3 license. We think the presence of a package with this license will prevent us fro… --- ## [Output filter, Elasticsearch & cloud instance](https://discuss.elastic.co/t/output-filter-elasticsearch-cloud-instance/310376)
**Author:** [@Mark\_Rodman](https://discuss.elastic.co/u/Mark_Rodman)\ **Replies:** 24\ **Last updated:** [July 23, 2022, 9:51am UTC](https://discuss.elastic.co/t/output-filter-elasticsearch-cloud-instance/310376 "2022-07-23T09:51:15Z")
Hi, Hope you are all well. I've been building a new Logstash pipeline and testing against a local instance of Elasticsearch on my network which is running fine without any security. The process has been pretty straigh… --- ## [Multiline codec and csv filter performance](https://discuss.elastic.co/t/multiline-codec-and-csv-filter-performance/310383)
**Author:** [@INS](https://discuss.elastic.co/u/INS)\ **Replies:** 3\ **Last updated:** [July 22, 2022, 7:53pm UTC](https://discuss.elastic.co/t/multiline-codec-and-csv-filter-performance/310383 "2022-07-22T19:53:49Z")
Hi Could someone please explain to me how the multicodec works (with pattern ) along with the CSV filter? In my case I have a problem to handle 26,473,906 lines for one of pattern even though I cranked up the configur… --- ## [Mutate add\_field attribute accidently become duplicate when using multiple logstash](https://discuss.elastic.co/t/mutate-add-field-attribute-accidently-become-duplicate-when-using-multiple-logstash/309536)
**Author:** [@mahendrapratitos](https://discuss.elastic.co/u/mahendrapratitos)\ **Replies:** 2\ **Last updated:** [July 22, 2022, 11:23am UTC](https://discuss.elastic.co/t/mutate-add-field-attribute-accidently-become-duplicate-when-using-multiple-logstash/309536 "2022-07-22T11:23:48Z")
I tried to execute 8 logstash that accept different port and different filebeat. I run this logstash on Linux environment. when I run the logstash, turns out some attribute that I create by using script mutate add\_fiel… --- ## [Elasticsearch cluster security permission for logstash](https://discuss.elastic.co/t/elasticsearch-cluster-security-permission-for-logstash/310388)
**Author:** [@ciaroda](https://discuss.elastic.co/u/ciaroda)\ **Replies:** 0\ **Last updated:** [July 22, 2022, 11:16am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-security-permission-for-logstash/310388 "2022-07-22T11:16:06Z")
Hi all, playing further with elasticsearch and logstash security (view this post), I received other errors regarding cluster permissions. The configuration is the same as per the previous post. I configured the logsta… --- ## [Logstash exec input and sdtout](https://discuss.elastic.co/t/logstash-exec-input-and-sdtout/310369)
**Author:** [@Silver137](https://discuss.elastic.co/u/Silver137)\ **Replies:** 0\ **Last updated:** [July 22, 2022, 7:30am UTC](https://discuss.elastic.co/t/logstash-exec-input-and-sdtout/310369 "2022-07-22T07:30:44Z")
I'm trying to execute a custom 5 minutes duration bash script in logstash using exec input plugin. but imposible to get output. Trying things, i realized that redirecting the script stdout to stderr the script makes ou… --- ## [Prevent nested json from appearing in elasticsaerch field](https://discuss.elastic.co/t/prevent-nested-json-from-appearing-in-elasticsaerch-field/310328)
**Author:** [@mikhatanu](https://discuss.elastic.co/u/mikhatanu)\ **Replies:** 6\ **Last updated:** [July 22, 2022, 4:04am UTC](https://discuss.elastic.co/t/prevent-nested-json-from-appearing-in-elasticsaerch-field/310328 "2022-07-22T04:04:11Z")
Hello, i'm using http filter to do api call to a rest api server. The data returned is json. it looks like this example: { a:{ b:{ c:{} d:{} e:{} } } } I want… --- ## [Logstash filter not working, when all the filters are applied at once, but working when applied only one filter](https://discuss.elastic.co/t/logstash-filter-not-working-when-all-the-filters-are-applied-at-once-but-working-when-applied-only-one-filter/310186)
**Author:** [@djrshn2346](https://discuss.elastic.co/u/djrshn2346)\ **Replies:** 7\ **Last updated:** [July 22, 2022, 5:15am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-when-all-the-filters-are-applied-at-once-but-working-when-applied-only-one-filter/310186 "2022-07-22T05:15:23Z")
I have added 3 filters in Logstash but at a time only 2 are working, but all the three are not working at the same time. One of the filter is throwing error. When I applied single filter that is working fine, when 2 that… --- ## [Logstash Elapsed filter not working](https://discuss.elastic.co/t/logstash-elapsed-filter-not-working/310220)
**Author:** [@nakula](https://discuss.elastic.co/u/nakula)\ **Replies:** 8\ **Last updated:** [July 21, 2022, 8:02pm UTC](https://discuss.elastic.co/t/logstash-elapsed-filter-not-working/310220 "2022-07-21T20:02:37Z")
I am trying to find the different between the start time and end time of a transaction. I am using the sample log file as below.. 2022-07-17T12:36:30.3081415Z Info 38 \[xxx.frame.logging.serviceCols\] \["Level3"\] : Wires-… --- ## [Manipulating data in logstash](https://discuss.elastic.co/t/manipulating-data-in-logstash/310304)
**Author:** [@KeithL](https://discuss.elastic.co/u/KeithL)\ **Replies:** 1\ **Last updated:** [July 21, 2022, 2:47pm UTC](https://discuss.elastic.co/t/manipulating-data-in-logstash/310304 "2022-07-21T14:47:17Z")
Beginner's question; pardon my potentially poor terminology. I'd like to manipulate ingested data to include geo\_point data to facilitate visualisation in elastic/kibana e.g given this: { "Entities" =\> \[ \[… --- ## [Input beats - output syslog, what in the middle](https://discuss.elastic.co/t/input-beats-output-syslog-what-in-the-middle/310310)
**Author:** [@Kakos](https://discuss.elastic.co/u/Kakos)\ **Replies:** 0\ **Last updated:** [July 21, 2022, 2:40pm UTC](https://discuss.elastic.co/t/input-beats-output-syslog-what-in-the-middle/310310 "2022-07-21T14:40:43Z")
Hi, The messages i get from beats are completely unstructured. The most important of all they include characters like \\t and \\n and actually don't recognize the new lines and tabs which exist in the raw data. So they lo… --- ## [Modular pipeline construction with same input plugin (Kafka) but different topic name](https://discuss.elastic.co/t/modular-pipeline-construction-with-same-input-plugin-kafka-but-different-topic-name/310300)
**Author:** [@ibrahim.ramadan](https://discuss.elastic.co/u/ibrahim.ramadan)\ **Replies:** 0\ **Last updated:** [July 21, 2022, 1:26pm UTC](https://discuss.elastic.co/t/modular-pipeline-construction-with-same-input-plugin-kafka-but-different-topic-name/310300 "2022-07-21T13:26:37Z")
Hi, I am trying to construct modular multiple pipeline in logstash. I have the same input plugin from (kafka) in all pipelines but at each one some attributes like topic name and broker address different from each input… --- ## [Logstash date filter error](https://discuss.elastic.co/t/logstash-date-filter-error/310268)
**Author:** [@yugeeklab](https://discuss.elastic.co/u/yugeeklab)\ **Replies:** 1\ **Last updated:** [July 21, 2022, 12:48pm UTC](https://discuss.elastic.co/t/logstash-date-filter-error/310268 "2022-07-21T12:48:35Z")
filter { if "client" in \[tags\] { grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:timestamp} MysqlClient:SYSTEM\_USER:%{QS:mysql\_client.system\_user}, MYSQL\_USER:%{QS:mysql\_client.mysql\_user}… --- ## [Elasticsearch security permission for logstash](https://discuss.elastic.co/t/elasticsearch-security-permission-for-logstash/310015)
**Author:** [@ciaroda](https://discuss.elastic.co/u/ciaroda)\ **Replies:** 1\ **Last updated:** [July 21, 2022, 12:27pm UTC](https://discuss.elastic.co/t/elasticsearch-security-permission-for-logstash/310015 "2022-07-21T12:27:21Z")
Hi all, I played with ELK stack for a week or so... After installing elasticsearch and kibana (7.17 version), I enabled simple security following this guide. Later I installed logstash (version 7.17) with security con… --- ## [Csv plugin cooperating with multiplying pattern](https://discuss.elastic.co/t/csv-plugin-cooperating-with-multiplying-pattern/309865)
**Author:** [@INS](https://discuss.elastic.co/u/INS)\ **Replies:** 13\ **Last updated:** [July 21, 2022, 12:01pm UTC](https://discuss.elastic.co/t/csv-plugin-cooperating-with-multiplying-pattern/309865 "2022-07-21T12:01:01Z")
Hi @Badger I need to continue below topic: referring to above I have a question how I can mark in this code pattern, also count of records is not regular (once it's more once it's less) a place with different kind o… --- ## [Json file not parsing getting error unexpected character (':'' (code 58))](https://discuss.elastic.co/t/json-file-not-parsing-getting-error-unexpected-character-code-58/309323)
**Author:** [@Mary2022](https://discuss.elastic.co/u/Mary2022)\ **Replies:** 5\ **Last updated:** [July 21, 2022, 10:58am UTC](https://discuss.elastic.co/t/json-file-not-parsing-getting-error-unexpected-character-code-58/309323 "2022-07-21T10:58:09Z")
I have json files coming in the following format: \[ { "user": "Beta", "percent": 28, "startTime": "2022-07-07T11:31:45", "type": "CPU", "total": 1072987793, "Location": "locationB", "desk": "MAC" }, { … --- ## [Adding data after csv filter from rest api using http filter](https://discuss.elastic.co/t/adding-data-after-csv-filter-from-rest-api-using-http-filter/310241)
**Author:** [@mikhatanu](https://discuss.elastic.co/u/mikhatanu)\ **Replies:** 0\ **Last updated:** [July 21, 2022, 7:05am UTC](https://discuss.elastic.co/t/adding-data-after-csv-filter-from-rest-api-using-http-filter/310241 "2022-07-21T07:05:56Z")
Hello, i have 2 file: csv and json. i wanted to read the json file to get access token. here is my config: input { # Open csv from path file{ path =\>\["/some/path/log/\*.csv"\] } } filter{ # Parse csv input … --- ## [Logstash cannot send output to elasticsearch when elasticsearch not restarted first](https://discuss.elastic.co/t/logstash-cannot-send-output-to-elasticsearch-when-elasticsearch-not-restarted-first/310230)
**Author:** [@RichYaNa](https://discuss.elastic.co/u/RichYaNa)\ **Replies:** 0\ **Last updated:** [July 21, 2022, 4:57am UTC](https://discuss.elastic.co/t/logstash-cannot-send-output-to-elasticsearch-when-elasticsearch-not-restarted-first/310230 "2022-07-21T04:57:36Z")
Hi Everyone, I have a problem here when I run EL (elasticsearch and logstash) as a service in linux the problem is, logstash cannot output to the elasticsearch. But if i restart the elasticsearch service, suddenly all … --- ## [Change Logstash Reindex Time](https://discuss.elastic.co/t/change-logstash-reindex-time/310223)
**Author:** [@Greg12](https://discuss.elastic.co/u/Greg12)\ **Replies:** 1\ **Last updated:** [July 21, 2022, 1:39am UTC](https://discuss.elastic.co/t/change-logstash-reindex-time/310223 "2022-07-21T01:39:14Z")
Logstash is rotating and creating new indexes every day at 7pm, I want to shift this to 2am. Is there a way to do this? --- ## [Space in field reference not working as intended](https://discuss.elastic.co/t/space-in-field-reference-not-working-as-intended/310215)
**Author:** [@BenJeau](https://discuss.elastic.co/u/BenJeau)\ **Replies:** 2\ **Last updated:** [July 21, 2022, 12:20am UTC](https://discuss.elastic.co/t/space-in-field-reference-not-working-as-intended/310215 "2022-07-21T00:20:26Z")
Hi, I'd like to verify if a field (which contains a space) is present or not in a document and to drop it if its present (in the example below I just added a field to debug and see what/which statement works, but turns … --- ## [Replace a value to another value logstash](https://discuss.elastic.co/t/replace-a-value-to-another-value-logstash/310129)
**Author:** [@mikhatanu](https://discuss.elastic.co/u/mikhatanu)\ **Replies:** 1\ **Last updated:** [July 20, 2022, 12:47pm UTC](https://discuss.elastic.co/t/replace-a-value-to-another-value-logstash/310129 "2022-07-20T12:47:42Z")
Hello, i have a log with an id. In another file, i have list file of the id mapping. for example id 1 has a name, description, etc in the list file. I want to replace the id in that log to the data in the list file. Log… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=124) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=126)