# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=126 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 127 --- ## [Ilm policy error](https://discuss.elastic.co/t/ilm-policy-error/310135)
**Author:** [@HeTvaM](https://discuss.elastic.co/u/HeTvaM)\ **Replies:** 0\ **Last updated:** [July 20, 2022, 11:36am UTC](https://discuss.elastic.co/t/ilm-policy-error/310135 "2022-07-20T11:36:58Z")
I have been trying to set up ilm policy for my indexes for several weeks now. I use filebeat in conjunction with logstash and my indexes end up being created every day. An example of my index name: 'filebeat-7.7.1-2022… --- ## [Adding fields based on match](https://discuss.elastic.co/t/adding-fields-based-on-match/310094)
**Author:** [@rusty\_cole](https://discuss.elastic.co/u/rusty_cole)\ **Replies:** 0\ **Last updated:** [July 20, 2022, 6:53am UTC](https://discuss.elastic.co/t/adding-fields-based-on-match/310094 "2022-07-20T06:53:04Z")
Hi All, I have the following scenario: 1 index containing json fields. I need to match Event1\_fieldA to Event2\_fieldB. If there is a match, add additional existing field from Event2 to Event1. How would I achieve t… --- ## [Logstash to Logstash enable security SSL](https://discuss.elastic.co/t/logstash-to-logstash-enable-security-ssl/310103)
**Author:** [@thahgr](https://discuss.elastic.co/u/thahgr)\ **Replies:** 0\ **Last updated:** [July 20, 2022, 7:49am UTC](https://discuss.elastic.co/t/logstash-to-logstash-enable-security-ssl/310103 "2022-07-20T07:49:28Z")
Logstash to logstash, how can we enable SSL communication with http out -\> http in plugins ? for the moment , unsecured, the two instances as below work fine output { http { url =\> "the url" http\_m… --- ## [Logstash pipeline grok issue with regex](https://discuss.elastic.co/t/logstash-pipeline-grok-issue-with-regex/308252)
**Author:** [@Sharon\_Hacham](https://discuss.elastic.co/u/Sharon_Hacham)\ **Replies:** 24\ **Last updated:** [July 20, 2022, 6:49am UTC](https://discuss.elastic.co/t/logstash-pipeline-grok-issue-with-regex/308252 "2022-07-20T06:49:18Z")
Hi , we have a Logstash pipeline , for Kafka on-premise Confluent Platform logs - shipped using Filebeat Kafka module , We are using a grok pattern to extract some of the entries in the data in order to use that data i… --- ## [How logstash extracts specific characters from a path?](https://discuss.elastic.co/t/how-logstash-extracts-specific-characters-from-a-path/310090)
**Author:** [@wajika](https://discuss.elastic.co/u/wajika)\ **Replies:** 2\ **Last updated:** [July 20, 2022, 6:45am UTC](https://discuss.elastic.co/t/how-logstash-extracts-specific-characters-from-a-path/310090 "2022-07-20T06:45:21Z")
I want to extract "acpro" as the service name from the path below, can anyone offer some advice? /api/acpro/filter/dowork/ayc /api/acpro/filter/toko /api/acpro/filter/toko/user/passwd /api/acpro/account/getsales/b77dd0b… --- ## [Logstash Dissect against \\e control character](https://discuss.elastic.co/t/logstash-dissect-against-e-control-character/309993)
**Author:** [@Silver137](https://discuss.elastic.co/u/Silver137)\ **Replies:** 2\ **Last updated:** [July 20, 2022, 6:01am UTC](https://discuss.elastic.co/t/logstash-dissect-against-e-control-character/309993 "2022-07-20T06:01:38Z")
My line to dissect begins with \\e\[92mContent , I need extract Content avoiding \\e\[92 I have already try: dissect { mapping =\> { "message" =\> "\\e\[92m%{content}"}} But not works --- ## [Logstash running without logstash.yml](https://discuss.elastic.co/t/logstash-running-without-logstash-yml/307338)
**Author:** [@vikram\_singh](https://discuss.elastic.co/u/vikram_singh)\ **Replies:** 5\ **Last updated:** [July 20, 2022, 5:47am UTC](https://discuss.elastic.co/t/logstash-running-without-logstash-yml/307338 "2022-07-20T05:47:40Z")
Hi, I am running logstash for a single file by sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/myfile.conf it is running correctly. But when I start logstash as a service… --- ## [Http input plugins is not behaving as expected](https://discuss.elastic.co/t/http-input-plugins-is-not-behaving-as-expected/309982)
**Author:** [@priyankamondalhcl](https://discuss.elastic.co/u/priyankamondalhcl)\ **Replies:** 2\ **Last updated:** [July 20, 2022, 5:20am UTC](https://discuss.elastic.co/t/http-input-plugins-is-not-behaving-as-expected/309982 "2022-07-20T05:20:22Z")
Hi, I am using http input plugins, which should be capable to expect json events, and the amount of data is huge (around 7,00,000 in 30 mins). after processing some data it stopped working and throwing error: \[13004ae0f… --- ## [Copy contents of one field to same field of another row if certain condition matches](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339)
**Author:** [@Ashutosh\_Vaidya](https://discuss.elastic.co/u/Ashutosh_Vaidya)\ **Replies:** 14\ **Last updated:** [July 20, 2022, 5:20am UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339 "2022-07-20T05:20:07Z")
I need to copy contents of “Service” field from No. 15256 into the :path field of No.15257 if “Stream\_Identifier” of both lines in 15256 and 15257 are matching. Kindly suggest how to get this done in the conf file of lo… --- ## [How to create filter for strings with separator and calculate string?](https://discuss.elastic.co/t/how-to-create-filter-for-strings-with-separator-and-calculate-string/310073)
**Author:** [@Bohdan\_Repetskyi](https://discuss.elastic.co/u/Bohdan_Repetskyi)\ **Replies:** 0\ **Last updated:** [July 19, 2022, 9:57pm UTC](https://discuss.elastic.co/t/how-to-create-filter-for-strings-with-separator-and-calculate-string/310073 "2022-07-19T21:57:26Z")
I have a simple config file: input { file{ path =\> "/tmp/data\_for\_logstach" start\_position =\> "beginning" } } filter { } output { file{ path =\> "/tmp/logstash\_convert" action =\> "update" } } H… --- ## [Logstash kafka](https://discuss.elastic.co/t/logstash-kafka/310064)
**Author:** [@emmanuel\_stevens\_LED](https://discuss.elastic.co/u/emmanuel_stevens_LED)\ **Replies:** 0\ **Last updated:** [July 19, 2022, 7:55pm UTC](https://discuss.elastic.co/t/logstash-kafka/310064 "2022-07-19T19:55:03Z")
Hello, I need some help to understand someting. I am new in elk stack so i will ask a lot of questions for the next weeks. With th persistant queue in logstash and the Filebeat backpressure-sensitive protocol when send… --- ## [Not able to recognize rsyslogs in SIEM](https://discuss.elastic.co/t/not-able-to-recognize-rsyslogs-in-siem/309750)
**Author:** [@Rao\_Nelakurti](https://discuss.elastic.co/u/Rao_Nelakurti)\ **Replies:** 5\ **Last updated:** [July 18, 2022, 2:59pm UTC](https://discuss.elastic.co/t/not-able-to-recognize-rsyslogs-in-siem/309750 "2022-07-18T14:59:08Z")
I'm trying to send rsyslogs to SIEM (rsyslog-\> filebeat-\> logstash-\> siem) filebeat.conf, filebeat.spool\_size: 2048 filebeat.idle\_timeout: 5s output.logstash: hosts: \['10.x.x.5:6045'\] - type: log paths: - /var/l… --- ## [1 huge pipelines vs 2 medium ones](https://discuss.elastic.co/t/1-huge-pipelines-vs-2-medium-ones/309925)
**Author:** [@anon90868141](https://discuss.elastic.co/u/anon90868141)\ **Replies:** 5\ **Last updated:** [July 19, 2022, 12:28pm UTC](https://discuss.elastic.co/t/1-huge-pipelines-vs-2-medium-ones/309925 "2022-07-19T12:28:41Z")
Hello, let's say I've got a cluster with 10 machines a 12 cores and I see data not being collected fast enough, so I need to raise my threads which is already at 12. Is better to go with the same pipeline and raise the… --- ## [Logstash JDBC connection for 3000 databases](https://discuss.elastic.co/t/logstash-jdbc-connection-for-3000-databases/310004)
**Author:** [@Poojan](https://discuss.elastic.co/u/Poojan)\ **Replies:** 0\ **Last updated:** [July 19, 2022, 10:17am UTC](https://discuss.elastic.co/t/logstash-jdbc-connection-for-3000-databases/310004 "2022-07-19T10:17:54Z")
Hello guys, I want to connect to 3000 databases which are on the same host using logstash. how can I connect those dbs and push data to elasticsearch? --- ## [How to make logstash filter multiple grok pattern](https://discuss.elastic.co/t/how-to-make-logstash-filter-multiple-grok-pattern/309810)
**Author:** [@lzold\_z](https://discuss.elastic.co/u/lzold_z)\ **Replies:** 2\ **Last updated:** [July 19, 2022, 9:17am UTC](https://discuss.elastic.co/t/how-to-make-logstash-filter-multiple-grok-pattern/309810 "2022-07-19T09:17:56Z")
Hello, im new to the logstash and i want to know how to make logstash filter multiple grok pattern for example from httpd error logs and access logs, im already have those pattern but logstash seem didnt catch the logs, … --- ## [JSON file parse error](https://discuss.elastic.co/t/json-file-parse-error/309869)
**Author:** [@ichasco\_heytrade](https://discuss.elastic.co/u/ichasco_heytrade)\ **Replies:** 2\ **Last updated:** [July 19, 2022, 8:58am UTC](https://discuss.elastic.co/t/json-file-parse-error/309869 "2022-07-19T08:58:44Z")
Hi, I am trying to parse a JSON file with this structure: \[ { "plugin": "cloudtrailBucketAccessLogging", "category": "CloudTrail", "title": "CloudTrail Bucket Access Logging", "description": "Ensures C… --- ## [Collect logs from external deployments exposing a domain](https://discuss.elastic.co/t/collect-logs-from-external-deployments-exposing-a-domain/309746)
**Author:** [@thahgr](https://discuss.elastic.co/u/thahgr)\ **Replies:** 4\ **Last updated:** [July 19, 2022, 7:33am UTC](https://discuss.elastic.co/t/collect-logs-from-external-deployments-exposing-a-domain/309746 "2022-07-19T07:33:48Z")
TL;DR I want to ship logs from remote deployments via filebeat or logstash to a central Elastic/Elastic stack via HTTP exposing just domains I have considered two solutions, cant get any to work The setup is as follow… --- ## [Logstash stuck at stage pipelines running](https://discuss.elastic.co/t/logstash-stuck-at-stage-pipelines-running/309951)
**Author:** [@sridharnetha](https://discuss.elastic.co/u/sridharnetha)\ **Replies:** 4\ **Last updated:** [July 19, 2022, 4:46am UTC](https://discuss.elastic.co/t/logstash-stuck-at-stage-pipelines-running/309951 "2022-07-19T04:46:14Z")
I am trying to ingest a csv file to include data on the elasticsearch server on the existing index. The csv file contains whitespaces in header column names. Here is an example "MOBILE NO" should be renamed to "MOBILE\_N… --- ## [Logstash syncing from Mongodb to Elasticsearch - Null values in array data objects](https://discuss.elastic.co/t/logstash-syncing-from-mongodb-to-elasticsearch-null-values-in-array-data-objects/309965)
**Author:** [@bindu1](https://discuss.elastic.co/u/bindu1)\ **Replies:** 0\ **Last updated:** [July 19, 2022, 1:44am UTC](https://discuss.elastic.co/t/logstash-syncing-from-mongodb-to-elasticsearch-null-values-in-array-data-objects/309965 "2022-07-19T01:44:44Z")
Hello Community, I'm using logstash to sync data from Mongdb to Elasticsearch & Kibana using JDBC input plugin. But array data fields are being synced as NULL values in ES/Kibana. Below is logstash.conf: input { … --- ## [Logstash fillter](https://discuss.elastic.co/t/logstash-fillter/309846)
**Author:** [@Jathurshan\_Sumandira](https://discuss.elastic.co/u/Jathurshan_Sumandira)\ **Replies:** 0\ **Last updated:** [July 18, 2022, 6:16am UTC](https://discuss.elastic.co/t/logstash-fillter/309846 "2022-07-18T06:16:13Z")
Hi Team, We are using 2 filters. First one is to get the token and by getting the token we are going to call the next url with bearer token First http getting the response but second http method is not giving data ht… --- ## [Best practices for dynamic expiration](https://discuss.elastic.co/t/best-practices-for-dynamic-expiration/309957)
**Author:** [@terramar](https://discuss.elastic.co/u/terramar)\ **Replies:** 1\ **Last updated:** [July 19, 2022, 12:24am UTC](https://discuss.elastic.co/t/best-practices-for-dynamic-expiration/309957 "2022-07-19T00:24:18Z")
I'm not sure if that title makes much sense. Right now, I have a fair amount of data coming in through logstash - about 7-10GB/day, and it all needs to stick around for 60 days. I currently write it to an index ("index-… --- ## [Logstash pipeline DLQ issue](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920)
**Author:** [@Harper\_S1](https://discuss.elastic.co/u/Harper_S1)\ **Replies:** 9\ **Last updated:** [July 18, 2022, 7:31pm UTC](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920 "2022-07-18T19:31:37Z")
Hi, I am upgrading my logstash from 7.x.x to 8.2.0. Problem is, logstash is not able to run the pipeline, showing the following error. \[2022-07-18T11:08:52,606\]\[ERROR\]\[org.logstash.common.io.DeadLetterQueueWriter\]\[main… --- ## [Logstash syslog input filter default grok patterns](https://discuss.elastic.co/t/logstash-syslog-input-filter-default-grok-patterns/309938)
**Author:** [@rugenl](https://discuss.elastic.co/u/rugenl)\ **Replies:** 1\ **Last updated:** [July 18, 2022, 6:54pm UTC](https://discuss.elastic.co/t/logstash-syslog-input-filter-default-grok-patterns/309938 "2022-07-18T18:54:47Z")
What are the default grok patterns for the syslog input filter? The data is from old snare, failing messages are formatted something like this: \<135\> 07/18/2022:18:14:52 GMT HOSTNAME syslog\_message. Thanks --- ## [Logstash cannot parse user\_agent field of nginx](https://discuss.elastic.co/t/logstash-cannot-parse-user-agent-field-of-nginx/309934)
**Author:** [@kosmylo](https://discuss.elastic.co/u/kosmylo)\ **Replies:** 3\ **Last updated:** [July 18, 2022, 6:51pm UTC](https://discuss.elastic.co/t/logstash-cannot-parse-user-agent-field-of-nginx/309934 "2022-07-18T18:51:43Z")
I have nginx logs with the following format: 192.168.0.1 - - \[18/Jul/2022:11:20:28 +0000\] "GET / HTTP/1.1" 200 15 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Sa… --- ## [Logstash Filter JSON syslog message field is not getting parsed](https://discuss.elastic.co/t/logstash-filter-json-syslog-message-field-is-not-getting-parsed/309864)
**Author:** [@Giridharan\_C](https://discuss.elastic.co/u/Giridharan_C)\ **Replies:** 3\ **Last updated:** [July 18, 2022, 6:12pm UTC](https://discuss.elastic.co/t/logstash-filter-json-syslog-message-field-is-not-getting-parsed/309864 "2022-07-18T18:12:22Z")
I cannot parse the incoming Syslog by JSON. The message field is not getting parsed. I tried JSON filter using addfield and also with mutate but no luck. I used GROK to parse specific fields but the message field has key… --- ## [Logstash grok pattern for custom nginx access log](https://discuss.elastic.co/t/logstash-grok-pattern-for-custom-nginx-access-log/309933)
**Author:** [@kosmylo](https://discuss.elastic.co/u/kosmylo)\ **Replies:** 0\ **Last updated:** [July 18, 2022, 5:46pm UTC](https://discuss.elastic.co/t/logstash-grok-pattern-for-custom-nginx-access-log/309933 "2022-07-18T17:46:09Z")
I have a custom format of nginx access.log and I want to parse it with logstash. The format of the access log is the following: log\_format main '$remote\_addr - $remote\_user \[$time\_local\] "$request" ' … --- ## [Removing special characters](https://discuss.elastic.co/t/removing-special-characters/309766)
**Author:** [@JeremyP](https://discuss.elastic.co/u/JeremyP)\ **Replies:** 6\ **Last updated:** [July 18, 2022, 4:40pm UTC](https://discuss.elastic.co/t/removing-special-characters/309766 "2022-07-18T16:40:07Z")
Hello, I have some special characters in one of my fields which I'm having issues removing. They may be reserved characters so my logstash instance is failing. Here is the event in question... "proof" =\> \[ … --- ## [Receiving partial messages in elastic](https://discuss.elastic.co/t/receiving-partial-messages-in-elastic/309730)
**Author:** [@ytld](https://discuss.elastic.co/u/ytld)\ **Replies:** 4\ **Last updated:** [July 18, 2022, 2:56pm UTC](https://discuss.elastic.co/t/receiving-partial-messages-in-elastic/309730 "2022-07-18T14:56:03Z")
We have a problem in our ELK-stack of which I'm unsure how to solve. We're running 8.3 version of the ELK-stack on Centos 7 machines and whereas everything worked fine before, since this week we're only seeing partial m… --- ## [Overriding logstash access "remote\_host" with client IP - Spring Boot](https://discuss.elastic.co/t/overriding-logstash-access-remote-host-with-client-ip-spring-boot/309887)
**Author:** [@Wes1](https://discuss.elastic.co/u/Wes1)\ **Replies:** 0\ **Last updated:** [July 18, 2022, 10:57am UTC](https://discuss.elastic.co/t/overriding-logstash-access-remote-host-with-client-ip-spring-boot/309887 "2022-07-18T10:57:59Z")
Good day, Does anyone know how to override "remote\_host" field within logstash access. Currently we are logging the load balancer's IP. I would like to use the "X-Forwarded-For" header. Thank you. --- ## [Logstash http filter with JWT](https://discuss.elastic.co/t/logstash-http-filter-with-jwt/309856)
**Author:** [@pszemesy](https://discuss.elastic.co/u/pszemesy)\ **Replies:** 0\ **Last updated:** [July 18, 2022, 7:42am UTC](https://discuss.elastic.co/t/logstash-http-filter-with-jwt/309856 "2022-07-18T07:42:10Z")
Hi, I have to creat a Logstash pipeline where I have to enrich the data from Remedy. But Remedy requires JWT based authentication. Are there any way to use JWT token based authentication in http filter? Thanks. [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=125) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=127)