# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=232 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 233 --- ## [Getting single line from multiline](https://discuss.elastic.co/t/getting-single-line-from-multiline/272292)
**Author:** [@Learn\_Mulesoft\_With](https://discuss.elastic.co/u/Learn_Mulesoft_With)\ **Replies:** 0\ **Last updated:** [May 6, 2021, 12:35pm UTC](https://discuss.elastic.co/t/getting-single-line-from-multiline/272292 "2021-05-06T12:35:32Z")
hi , my message is like below Error type : VALIDATION:INVALID\_NUMBER FlowStack : at db-error-logFlow(db-error-logFlow/processors/0 @ db-error-log:db-error-log.xml:18 i want just single line " E… --- ## [Java plugin - Irrelevant JARs are loaded and cause conflicts](https://discuss.elastic.co/t/java-plugin-irrelevant-jars-are-loaded-and-cause-conflicts/272276)
**Author:** [@MosheElisha](https://discuss.elastic.co/u/MosheElisha)\ **Replies:** 0\ **Last updated:** [May 6, 2021, 10:20am UTC](https://discuss.elastic.co/t/java-plugin-irrelevant-jars-are-loaded-and-cause-conflicts/272276 "2021-05-06T10:20:55Z")
Hi, I am creating a Java input plugin that uses 'software.amazon.awssdk:s3:2.16.26' and libraries from manticore-0.7.0-java are loaded into my plugin and cause conflicts. After successful installation, when I try to us… --- ## [Kibana refuse connection on docker-compose config](https://discuss.elastic.co/t/kibana-refuse-connection-on-docker-compose-config/272272)
**Author:** [@danyxs](https://discuss.elastic.co/u/danyxs)\ **Replies:** 0\ **Last updated:** [May 6, 2021, 10:03am UTC](https://discuss.elastic.co/t/kibana-refuse-connection-on-docker-compose-config/272272 "2021-05-06T10:03:33Z")
Hi, I have a problem with the following docker-compose.yaml file. I used a default file, but when i want to run the containers the kibana refuse connection. I attached the docker-compose config version: '3' services:… --- ## [Try to convert UTC timestamp to IST in logstash](https://discuss.elastic.co/t/try-to-convert-utc-timestamp-to-ist-in-logstash/272268)
**Author:** [@sudheendra](https://discuss.elastic.co/u/sudheendra)\ **Replies:** 0\ **Last updated:** [May 6, 2021, 9:36am UTC](https://discuss.elastic.co/t/try-to-convert-utc-timestamp-to-ist-in-logstash/272268 "2021-05-06T09:36:21Z")
Hi , I need to convert the log UTC to my local time in IST . Used ruby Filter to convert to local time . My filter in logstash.yaml file filter { ruby { code =\> "event.set('new\_date', event.get('\[@timestamp\]').time… --- ## [Can't parse after optionnal field has been set](https://discuss.elastic.co/t/cant-parse-after-optionnal-field-has-been-set/272253)
**Author:** [@Travis](https://discuss.elastic.co/u/Travis)\ **Replies:** 2\ **Last updated:** [May 6, 2021, 8:40am UTC](https://discuss.elastic.co/t/cant-parse-after-optionnal-field-has-been-set/272253 "2021-05-06T08:40:42Z")
Hello ! I'm working on Cisco WSA logs. Some fields are optionnal. In particular, the filed which contains username infomation: sometimes we have "DOMAIN\\smith@AD" and sometimes we have just - When username is present, … --- ## [Events are lost when elasticsearch output is temporary unavailable](https://discuss.elastic.co/t/events-are-lost-when-elasticsearch-output-is-temporary-unavailable/272176)
**Author:** [@mabato](https://discuss.elastic.co/u/mabato)\ **Replies:** 3\ **Last updated:** [May 6, 2021, 8:23am UTC](https://discuss.elastic.co/t/events-are-lost-when-elasticsearch-output-is-temporary-unavailable/272176 "2021-05-06T08:23:44Z")
Hi all, Newbie here. I have kubernetes cluster in a location with patchy internet connection. I have metric beats on the cluster and I want to send metrics to ES which runs in cloud. I thought that when I put logstash … --- ## [I want to further split messages into json and message after splitting them with grok](https://discuss.elastic.co/t/i-want-to-further-split-messages-into-json-and-message-after-splitting-them-with-grok/272243)
**Author:** [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Replies:** 4\ **Last updated:** [May 6, 2021, 7:23am UTC](https://discuss.elastic.co/t/i-want-to-further-split-messages-into-json-and-message-after-splitting-them-with-grok/272243 "2021-05-06T07:23:10Z")
I want to further split messages into json and message after splitting them with grok. I am using grok to split messages. I'm getting a log in json format in a field, and I found a case where the field contains both js… --- ## [Process never end when I run a single pipeline inside a docker container](https://discuss.elastic.co/t/process-never-end-when-i-run-a-single-pipeline-inside-a-docker-container/272231)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 1\ **Last updated:** [May 6, 2021, 1:41am UTC](https://discuss.elastic.co/t/process-never-end-when-i-run-a-single-pipeline-inside-a-docker-container/272231 "2021-05-06T01:41:39Z")
with a python script Im running logstash via command inside a docker container, the normal behavior (with logstash installed in the server) is that after the pipeline get the data that pipeline shuts down, but the proces… --- ## [How to run multiple logstash config in parallel](https://discuss.elastic.co/t/how-to-run-multiple-logstash-config-in-parallel/272198)
**Author:** [@ullasrao](https://discuss.elastic.co/u/ullasrao)\ **Replies:** 1\ **Last updated:** [May 6, 2021, 12:20am UTC](https://discuss.elastic.co/t/how-to-run-multiple-logstash-config-in-parallel/272198 "2021-05-06T00:20:05Z")
hi , i am new to ELK and I am setting up the pipelines for the data. i have a requirement where i have to read data from 2 different kafka topics and load them to 2 different elastic search tables respectively. I hav… --- ## [Date Plugin doc not visible](https://discuss.elastic.co/t/date-plugin-doc-not-visible/272202)
**Author:** [@Soren\_vdc](https://discuss.elastic.co/u/Soren_vdc)\ **Replies:** 0\ **Last updated:** [May 5, 2021, 2:03pm UTC](https://discuss.elastic.co/t/date-plugin-doc-not-visible/272202 "2021-05-05T14:03:45Z")
Hi, I have this time format in logfile: time\_stamp=\[30/Apr/2021:15:23:46+0200\] When I use the date plugin: kv { source =\> "message" field\_split =\> "|" } date { match =\> \[ "time\_stamp", ,… --- ## [Understand "reconnect" and "retries" settings in Kafka output](https://discuss.elastic.co/t/understand-reconnect-and-retries-settings-in-kafka-output/272157)
**Author:** [@Travis](https://discuss.elastic.co/u/Travis)\ **Replies:** 2\ **Last updated:** [May 5, 2021, 9:52am UTC](https://discuss.elastic.co/t/understand-reconnect-and-retries-settings-in-kafka-output/272157 "2021-05-05T09:52:25Z")
Hello all, With this config : input { tcp { type =\> "NETWORK\_DEVICE" port =\> 1602 } filter{ } output { kafka { topic\_id =\> \["network.device"\] codec =\> json bootstrap\_servers =\> "kafka1:9092,kafka2:9092,k… --- ## [Mutate Convert not working as I expect](https://discuss.elastic.co/t/mutate-convert-not-working-as-i-expect/272133)
**Author:** [@lachlan.simpson](https://discuss.elastic.co/u/lachlan.simpson)\ **Replies:** 3\ **Last updated:** [May 5, 2021, 9:29am UTC](https://discuss.elastic.co/t/mutate-convert-not-working-as-i-expect/272133 "2021-05-05T09:29:15Z")
RHEL8.3, ELK 7.12.1, I am using convert as documented, but I just can't see what I've done wrong. Here's the filter, per the docs filter { if \[pbs\_accounting\] { csv { separator =\> ";" columns =\> \["da… --- ## [Pipeline to pipeline communication](https://discuss.elastic.co/t/pipeline-to-pipeline-communication/272140)
**Author:** [@bdn](https://discuss.elastic.co/u/bdn)\ **Replies:** 2\ **Last updated:** [May 5, 2021, 9:12am UTC](https://discuss.elastic.co/t/pipeline-to-pipeline-communication/272140 "2021-05-05T09:12:21Z")
I have a requirement. Events will be listening on some TCP ports in Logstash. using a filter plugin I will extract the username field from the event. After that, I have to query another API server providing that username… --- ## [Remove a key that starts with 0 to 9](https://discuss.elastic.co/t/remove-a-key-that-starts-with-0-to-9/272142)
**Author:** [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)\ **Replies:** 0\ **Last updated:** [May 5, 2021, 7:49am UTC](https://discuss.elastic.co/t/remove-a-key-that-starts-with-0-to-9/272142 "2021-05-05T07:49:54Z")
Hello team, I need to Remove a field that starts with an integer. I am using logstash & kv filter to parse this logs. Can you please help me. PFB sample log line: 2021-01-07 05:32:31,761 Method=get call SoapAction="Ri… --- ## [Directly accessing data from multiline log data](https://discuss.elastic.co/t/directly-accessing-data-from-multiline-log-data/272135)
**Author:** [@Learn\_Mulesoft\_With](https://discuss.elastic.co/u/Learn_Mulesoft_With)\ **Replies:** 0\ **Last updated:** [May 5, 2021, 5:50am UTC](https://discuss.elastic.co/t/directly-accessing-data-from-multiline-log-data/272135 "2021-05-05T05:50:52Z")
Hi Below is my sample plain text input to http input , is there any way in filter to directly jump to " Trace: " and then directly jump to "Error type" and retrieve value the part denoted by "XXX" is dynamic it will … --- ## [Logstash filter for muliple values not working](https://discuss.elastic.co/t/logstash-filter-for-muliple-values-not-working/272087)
**Author:** [@rohitarorait82](https://discuss.elastic.co/u/rohitarorait82)\ **Replies:** 2\ **Last updated:** [May 5, 2021, 5:04am UTC](https://discuss.elastic.co/t/logstash-filter-for-muliple-values-not-working/272087 "2021-05-05T05:04:01Z")
Hi All, I am trying to remove message for few service tags, but I am facing below issue. Below is not working if \[SERVICE\] in \["GET", "UPDATE"\] { mutate { remove\_field =\> \["message"\] } } Below is working if "GET"… --- ## [Parsing Palo Alto System Logs with Date Filter](https://discuss.elastic.co/t/parsing-palo-alto-system-logs-with-date-filter/271836)
**Author:** [@6igwig](https://discuss.elastic.co/u/6igwig)\ **Replies:** 7\ **Last updated:** [May 4, 2021, 7:23pm UTC](https://discuss.elastic.co/t/parsing-palo-alto-system-logs-with-date-filter/271836 "2021-05-04T19:23:24Z")
I am trying to parse the Palo Alto System logs with logstash using the following dissect pattern inside a logstash filter: dissect { mapping =\> { "message" =\> '%{?date} %{?date} %{?date} %{?network} - %{?version},%{@… --- ## [Logstash date filter, kibana index template warning in logs](https://discuss.elastic.co/t/logstash-date-filter-kibana-index-template-warning-in-logs/272094)
**Author:** [@wmei](https://discuss.elastic.co/u/wmei)\ **Replies:** 2\ **Last updated:** [May 4, 2021, 5:43pm UTC](https://discuss.elastic.co/t/logstash-date-filter-kibana-index-template-warning-in-logs/272094 "2021-05-04T17:43:23Z")
Hi, Original log, these dates are in the format 20210504-15:40:02.601 and I have a date filter to convert that in my logstash.yml. I am doing the same filter for both the SendingTime and the TransactTime. if \[layers\]\[f… --- ## [What version of TLS used by Logstash output](https://discuss.elastic.co/t/what-version-of-tls-used-by-logstash-output/272086)
**Author:** [@Sunil\_Chadha](https://discuss.elastic.co/u/Sunil_Chadha)\ **Replies:** 1\ **Last updated:** [May 4, 2021, 4:05pm UTC](https://discuss.elastic.co/t/what-version-of-tls-used-by-logstash-output/272086 "2021-05-04T16:05:05Z")
Hi, I was looking at Logstash documentation for output plugin, but could not find the TLS version used by it's outputs-elasticsearch output-syslog output-lumberjack If anyone knows about the TLS protocol version supp… --- ## [Logstash conf file, difference between multiple input/output vs consolidated input/output](https://discuss.elastic.co/t/logstash-conf-file-difference-between-multiple-input-output-vs-consolidated-input-output/272066)
**Author:** [@Baygon](https://discuss.elastic.co/u/Baygon)\ **Replies:** 1\ **Last updated:** [May 4, 2021, 4:09pm UTC](https://discuss.elastic.co/t/logstash-conf-file-difference-between-multiple-input-output-vs-consolidated-input-output/272066 "2021-05-04T16:09:03Z")
Hi, I'm having about 25 pipelines to configure. I used to have all my jdbc sources inside one input block, then in the output I would have a if condition on the 'tags' value. That would mean on input with 25 jdbbc sec… --- ## [Logstash with docker compose and grafana](https://discuss.elastic.co/t/logstash-with-docker-compose-and-grafana/272075)
**Author:** [@XXXPVSZYON](https://discuss.elastic.co/u/XXXPVSZYON)\ **Replies:** 0\ **Last updated:** [May 4, 2021, 2:07pm UTC](https://discuss.elastic.co/t/logstash-with-docker-compose-and-grafana/272075 "2021-05-04T14:07:26Z")
Hi, I'm working on my final degree work and i'm stuck uploading a .cvs from logstash to elasticsearch with Docker Compose. This is my Docker-Compose.yml: version: '3' services: grafana: image: grafana/grafana ports… --- ## [How many indices we can create at a time using logstash.conf?](https://discuss.elastic.co/t/how-many-indices-we-can-create-at-a-time-using-logstash-conf/272059)
**Author:** [@Learn\_Mulesoft\_With](https://discuss.elastic.co/u/Learn_Mulesoft_With)\ **Replies:** 3\ **Last updated:** [May 4, 2021, 1:50pm UTC](https://discuss.elastic.co/t/how-many-indices-we-can-create-at-a-time-using-logstash-conf/272059 "2021-05-04T13:50:22Z")
consider my logstash.conf input{ http{ host =\>"localhost" port =\>"5044" response\_code =\>201 type =\> "log\_error" } http{ host =\>"localhost" port =\>"5045" response\_code =\>201 type =\> "log\_message" } http{ ho… --- ## [Output Elasticsearch index name error](https://discuss.elastic.co/t/output-elasticsearch-index-name-error/271979)
**Author:** [@Frack](https://discuss.elastic.co/u/Frack)\ **Replies:** 3\ **Last updated:** [May 4, 2021, 9:25am UTC](https://discuss.elastic.co/t/output-elasticsearch-index-name-error/271979 "2021-05-04T09:25:45Z")
I have a "logstash" index with documents while I have hardcoded the name of the output index by the tags. when I check in the index "logstash" my logs have the right tag "windows" or "iptable" ... I added a final "else… --- ## [How to configure GREDDYDATA in logstash to take mutiline input](https://discuss.elastic.co/t/how-to-configure-greddydata-in-logstash-to-take-mutiline-input/272046)
**Author:** [@Learn\_Mulesoft\_With](https://discuss.elastic.co/u/Learn_Mulesoft_With)\ **Replies:** 0\ **Last updated:** [May 4, 2021, 7:46am UTC](https://discuss.elastic.co/t/how-to-configure-greddydata-in-logstash-to-take-mutiline-input/272046 "2021-05-04T07:46:33Z")
INFO 2021-04-17 21:40:25,689 \[\[MuleRuntime\].uber.01: \[testproject\].testprojectFlow.CPU\_LITE @4d779882\] \[processor: testprojectFlow/processors/0; event: 6b78a980-9f97-11eb-a8e6-f8ac6500be0e\] org.mule.runtime.core.interna… --- ## [Grok regex, How should I interpret the hour pattern?](https://discuss.elastic.co/t/grok-regex-how-should-i-interpret-the-hour-pattern/272030)
**Author:** [@111420](https://discuss.elastic.co/u/111420)\ **Replies:** 2\ **Last updated:** [May 4, 2021, 5:33am UTC](https://discuss.elastic.co/t/grok-regex-how-should-i-interpret-the-hour-pattern/272030 "2021-05-04T05:33:59Z")
In Logstash, does HOUR's regular expression (?:2\[0123\]|\[01\])?\[0-9\]) I wonder what the number after the colon means and how to interpret the whole thing. --- ## [Load data from sql server to elasticsearch aws](https://discuss.elastic.co/t/load-data-from-sql-server-to-elasticsearch-aws/272023)
**Author:** [@hiba](https://discuss.elastic.co/u/hiba)\ **Replies:** 2\ **Last updated:** [May 4, 2021, 2:32am UTC](https://discuss.elastic.co/t/load-data-from-sql-server-to-elasticsearch-aws/272023 "2021-05-04T02:32:53Z")
Hi all, How can I load data from sql server to elasticsearch (aws version 6.5) using logstash ( version 7.9). I can't download the output plugin es-amzon on Windows. Can someone help me please and tell me thé steps … --- ## [Multiple indices for single logstash pipeline](https://discuss.elastic.co/t/multiple-indices-for-single-logstash-pipeline/272015)
**Author:** [@srk1](https://discuss.elastic.co/u/srk1)\ **Replies:** 1\ **Last updated:** [May 4, 2021, 12:12am UTC](https://discuss.elastic.co/t/multiple-indices-for-single-logstash-pipeline/272015 "2021-05-04T00:12:31Z")
Hello ES users, i have filebeat instance sending one source log to ES via logstash pipeline. Now, i want to add another file on filebeat input list and use the same logstash pipeline but write to a different new index. C… --- ## [Force logstash to restart after "Pipeline aborted due to error"](https://discuss.elastic.co/t/force-logstash-to-restart-after-pipeline-aborted-due-to-error/272007)
**Author:** [@wmd](https://discuss.elastic.co/u/wmd)\ **Replies:** 4\ **Last updated:** [May 3, 2021, 9:45pm UTC](https://discuss.elastic.co/t/force-logstash-to-restart-after-pipeline-aborted-due-to-error/272007 "2021-05-03T21:45:51Z")
Logstash v6.3.1 is logging this at startup and then getting stuck forever: Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties \[2021-04-30T13:05:39,258\]\[ERROR\]\[logstash.pipe… --- ## [Auto\_flush\_interval with ordered pipeline](https://discuss.elastic.co/t/auto-flush-interval-with-ordered-pipeline/271924)
**Author:** [@roua.B](https://discuss.elastic.co/u/roua.B)\ **Replies:** 3\ **Last updated:** [May 3, 2021, 7:41pm UTC](https://discuss.elastic.co/t/auto-flush-interval-with-ordered-pipeline/271924 "2021-05-03T19:41:34Z")
Hello, So I was facing an issue with logstash not parsing the last line of my xml log file. I found that the answer would be to add auto\_flush\_interval =\> 1 to my multiline codec. I would like my pipeline to be ordered … --- ## [Logstash mongodb output plugin fails to authenticate](https://discuss.elastic.co/t/logstash-mongodb-output-plugin-fails-to-authenticate/272014)
**Author:** [@aaron-dsouza](https://discuss.elastic.co/u/aaron-dsouza)\ **Replies:** 0\ **Last updated:** [May 3, 2021, 6:09pm UTC](https://discuss.elastic.co/t/logstash-mongodb-output-plugin-fails-to-authenticate/272014 "2021-05-03T18:09:43Z")
I'm using the logstash mongodb output plugin to write documents to Azure cosmos DB. I am facing the exact same issue as reported here Logstash MongoDB Output Plugin - command insert requires authentication Here is my co… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=231) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=233)