# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=891 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 892 --- ## [What are the default data types when Logstash reads from SQL DB?](https://discuss.elastic.co/t/what-are-the-default-data-types-when-logstash-reads-from-sql-db/103996)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 5\ **Last updated:** [October 17, 2017, 4:04am UTC](https://discuss.elastic.co/t/what-are-the-default-data-types-when-logstash-reads-from-sql-db/103996 "2017-10-17T04:04:50Z")
Hi, Can someone please say what are the default data types when Logstash reads data from SQL DB. Will it be the same as declared in the Sql server Database? And when I want to send it to Elasticsearch do I need to perfo… --- ## [S3 output requires PutObject on \* Resources](https://discuss.elastic.co/t/s3-output-requires-putobject-on-resources/103457)
**Author:** [@Theodore\_Cowan](https://discuss.elastic.co/u/Theodore_Cowan)\ **Replies:** 6\ **Last updated:** [October 16, 2017, 10:55pm UTC](https://discuss.elastic.co/t/s3-output-requires-putobject-on-resources/103457 "2017-10-16T22:55:48Z")
Posting this so somebody else may avoid spending hours trying to debug the s3 output plugin. The plugin will throw permissions errors if you do not grant PutObject permissions on bucket resources. \[2017-10-11T02:53:41,… --- ## [Parse multiline xml config file into one event, not many](https://discuss.elastic.co/t/parse-multiline-xml-config-file-into-one-event-not-many/104146)
**Author:** [@JSkier](https://discuss.elastic.co/u/JSkier)\ **Replies:** 0\ **Last updated:** [October 16, 2017, 9:30pm UTC](https://discuss.elastic.co/t/parse-multiline-xml-config-file-into-one-event-not-many/104146 "2017-10-16T21:30:10Z")
Hi, I'm monitoring a configuration xml file for changes using filebeat, and sending it over to logstash. I would like to take the key value pair into one event in elasticsearch. It seems to do fine parsing the XML line … --- ## [Different input source with same information problem](https://discuss.elastic.co/t/different-input-source-with-same-information-problem/103980)
**Author:** [@kiwi0217](https://discuss.elastic.co/u/kiwi0217)\ **Replies:** 1\ **Last updated:** [October 16, 2017, 7:52pm UTC](https://discuss.elastic.co/t/different-input-source-with-same-information-problem/103980 "2017-10-16T19:52:51Z")
I want to use one filter to fetch same information from multiple input logfile like： userA's log: "git clone" and userB's log: "git clone" can I gork this and make difference output information to elasticsearch like： … --- ## [How do I create an index per log file that I read?](https://discuss.elastic.co/t/how-do-i-create-an-index-per-log-file-that-i-read/104134)
**Author:** [@magdieldominguez](https://discuss.elastic.co/u/magdieldominguez)\ **Replies:** 1\ **Last updated:** [October 16, 2017, 7:46pm UTC](https://discuss.elastic.co/t/how-do-i-create-an-index-per-log-file-that-i-read/104134 "2017-10-16T19:46:36Z")
Hi I have 3 types of log files that I am reading from. My setup is as follows: Filebeat-Logstash-Elasticsearch-Kibana. The 3 types of logs are logs such as: Processing logs, Spike logs, Error logs. I would like to be a… --- ## [Send json message only from a log file to elastic seach](https://discuss.elastic.co/t/send-json-message-only-from-a-log-file-to-elastic-seach/103791)
**Author:** [@rainbow1\_rainbow](https://discuss.elastic.co/u/rainbow1_rainbow)\ **Replies:** 6\ **Last updated:** [October 16, 2017, 6:05pm UTC](https://discuss.elastic.co/t/send-json-message-only-from-a-log-file-to-elastic-seach/103791 "2017-10-16T18:05:44Z")
Hi, Below is the output of a log file, I will need to send json message only from a log file, and add a filter geoip for the visitorip cat visitor\_cluster01server02.log \[ECOM\] \[2017-10-11 14:50:06,810\] \[ECOM\] \[DEBUG\] … --- ## [Grokparsefailure for auth.log](https://discuss.elastic.co/t/grokparsefailure-for-auth-log/104127)
**Author:** [@pdna](https://discuss.elastic.co/u/pdna)\ **Replies:** 1\ **Last updated:** [October 16, 2017, 5:36pm UTC](https://discuss.elastic.co/t/grokparsefailure-for-auth-log/104127 "2017-10-16T17:36:53Z")
I have the following test.conf file input { beats { port =\> "5043" } } filter { grok { match =\> \["message", "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} sshd(?:… --- ## ["RegEx Interrupted" and "Interrupted waiting for mutex" crashes](https://discuss.elastic.co/t/regex-interrupted-and-interrupted-waiting-for-mutex-crashes/104128)
**Author:** [@Daniel314](https://discuss.elastic.co/u/Daniel314)\ **Replies:** 0\ **Last updated:** [October 16, 2017, 5:11pm UTC](https://discuss.elastic.co/t/regex-interrupted-and-interrupted-waiting-for-mutex-crashes/104128 "2017-10-16T17:11:27Z")
This is an update to the other topics I created about this issue (RegExp interrupted and Interrupted waiting for mutex) In my environment I am using Kafka, and it is pulling from a firehose of data and pushing to a sore… --- ## [Logstash XML parse with meta-data](https://discuss.elastic.co/t/logstash-xml-parse-with-meta-data/103687)
**Author:** [@sawri](https://discuss.elastic.co/u/sawri)\ **Replies:** 6\ **Last updated:** [October 16, 2017, 4:58pm UTC](https://discuss.elastic.co/t/logstash-xml-parse-with-meta-data/103687 "2017-10-16T16:58:00Z")
I don't like to needlessly bother people with long questions, so I'll keep this short and searchable. I need some logstash-xml expertise. Problem: I have a large directory of xml files that need to be inserted into ela… --- ## [Logstash not sending events to es with this template](https://discuss.elastic.co/t/logstash-not-sending-events-to-es-with-this-template/104117)
**Author:** [@Badger](https://discuss.elastic.co/u/Badger)\ **Replies:** 0\ **Last updated:** [October 16, 2017, 4:18pm UTC](https://discuss.elastic.co/t/logstash-not-sending-events-to-es-with-this-template/104117 "2017-10-16T16:18:38Z")
I had a working 5.6.3 environment which I upgraded to 6.0.0-rc1. metricbeat requests the Apache server-status page and filebeat is capturing both the apache logs and some GC logs. After a trip through Kafka, the data hit… --- ## [Logstash does not create index in es](https://discuss.elastic.co/t/logstash-does-not-create-index-in-es/103995)
**Author:** [@Jean-Paul\_Da\_Cunha](https://discuss.elastic.co/u/Jean-Paul_Da_Cunha)\ **Replies:** 4\ **Last updated:** [October 16, 2017, 2:36pm UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-in-es/103995 "2017-10-16T14:36:39Z")
Hello, Can someone help me solving this index creation issue. I'm currently using logstash and filebeat to send tomcat logs to elasticsearch. Here is my logstash configuration input { beats { port =\> 5044 … --- ## [@timestamp not updating with my logs file timestamp](https://discuss.elastic.co/t/timestamp-not-updating-with-my-logs-file-timestamp/104005)
**Author:** [@Megha\_Gupta](https://discuss.elastic.co/u/Megha_Gupta)\ **Replies:** 1\ **Last updated:** [October 16, 2017, 2:35pm UTC](https://discuss.elastic.co/t/timestamp-not-updating-with-my-logs-file-timestamp/104005 "2017-10-16T14:35:38Z")
Hi, My Log messages are kind of json object : {"clientToken":"GMTALEOAPPLY","event":"uid\_needs\_to\_map","initialReferral":"https://www.oscarplusmcmaster.ca/myAccount/internships-coop/engineering/postings.htm","ipAddres… --- ## [Logstash.service can not read nginx access.log](https://discuss.elastic.co/t/logstash-service-can-not-read-nginx-access-log/104066)
**Author:** [@bo8888](https://discuss.elastic.co/u/bo8888)\ **Replies:** 5\ **Last updated:** [October 16, 2017, 2:00pm UTC](https://discuss.elastic.co/t/logstash-service-can-not-read-nginx-access-log/104066 "2017-10-16T14:00:54Z")
vim /etc/systemd/system/logstash.service User=logstash Group=logstash /etc/logstash/conf.d/nginx.conf it is not work well but vim /etc/systemd/system/logstash.service User=root Group=root it is work well access… --- ## [Failed to Import JSON to elastic via logstash](https://discuss.elastic.co/t/failed-to-import-json-to-elastic-via-logstash/104093)
**Author:** [@Mickey](https://discuss.elastic.co/u/Mickey)\ **Replies:** 0\ **Last updated:** [October 16, 2017, 1:35pm UTC](https://discuss.elastic.co/t/failed-to-import-json-to-elastic-via-logstash/104093 "2017-10-16T13:35:44Z")
i'm new to ELK stack and i am breaking my head trying to import single line json files to elasticsearch using logstach. nothing shows up at Elasticsearch (10.10.20.13:9200/monitor/\_search?q=\*) or kibana. my json looks l… --- ## [Loading incremental data into Elasticsearch from Oracle database](https://discuss.elastic.co/t/loading-incremental-data-into-elasticsearch-from-oracle-database/102007)
**Author:** [@Newuser](https://discuss.elastic.co/u/Newuser)\ **Replies:** 17\ **Last updated:** [October 16, 2017, 11:53am UTC](https://discuss.elastic.co/t/loading-incremental-data-into-elasticsearch-from-oracle-database/102007 "2017-10-16T11:53:59Z")
Hi all, I need to Load incremental data into Elasticsearch from Oracle database, after the first load ( via Logstash), i need to load only the data updated. Could you advise on this please? Regrads, --- ## [Populating logtime for @timestamp](https://discuss.elastic.co/t/populating-logtime-for-timestamp/103877)
**Author:** [@akash.01](https://discuss.elastic.co/u/akash.01)\ **Replies:** 14\ **Last updated:** [October 16, 2017, 11:44am UTC](https://discuss.elastic.co/t/populating-logtime-for-timestamp/103877 "2017-10-16T11:44:13Z")
Hi mates, I am parsing the logs of Bluecoat proxies using filebeat and indexing the logs using logstash. When I search in Elasticsearch for the indexed data, I am getting the timestamp as the indexed time. I want to tak… --- ## [Getting two inputs (from beats and from a file)](https://discuss.elastic.co/t/getting-two-inputs-from-beats-and-from-a-file/104060)
**Author:** [@tomer](https://discuss.elastic.co/u/tomer)\ **Replies:** 1\ **Last updated:** [October 16, 2017, 8:40am UTC](https://discuss.elastic.co/t/getting-two-inputs-from-beats-and-from-a-file/104060 "2017-10-16T08:40:04Z")
Hi, how can I get input from two sources, from some file, and from beats? Here is what I did input { file{ path =\> "/var/example/exampleFile.txt" } beats { port =\> 5044 } } --- ## [Date parsing, logstash does not take slovene locale](https://discuss.elastic.co/t/date-parsing-logstash-does-not-take-slovene-locale/103830)
**Author:** [@Mojster](https://discuss.elastic.co/u/Mojster)\ **Replies:** 2\ **Last updated:** [October 16, 2017, 8:39am UTC](https://discuss.elastic.co/t/date-parsing-logstash-does-not-take-slovene-locale/103830 "2017-10-16T08:39:17Z")
Log entry: SEARCH#Android#SLO#sikmb#BIB#sikmb#02-okt-2017 07:04:00.721#ADV##0#0#0#2#8#((šola AND luči)) NOT (DT=D) NOT RS=D and (DP=08 OR DP=XX) NOT NH=1 filter part from pipeline: grok { match =\> {"message" =\> "%{… --- ## [ERROR Null object returned for RollingFile in Appenders](https://discuss.elastic.co/t/error-null-object-returned-for-rollingfile-in-appenders/103990)
**Author:** [@bo8888](https://discuss.elastic.co/u/bo8888)\ **Replies:** 1\ **Last updated:** [October 16, 2017, 2:55am UTC](https://discuss.elastic.co/t/error-null-object-returned-for-rollingfile-in-appenders/103990 "2017-10-16T02:55:23Z")
hi, i need help logstash-5.6.2 \[root@bogon ~\]# java -version java version "1.8.0\_121" Java(TM) SE Runtime Environment (build 1.8.0\_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) \[root@b… --- ## [Applying XML plugin on Kafka output message](https://discuss.elastic.co/t/applying-xml-plugin-on-kafka-output-message/103730)
**Author:** [@rm1890](https://discuss.elastic.co/u/rm1890)\ **Replies:** 1\ **Last updated:** [October 15, 2017, 7:52pm UTC](https://discuss.elastic.co/t/applying-xml-plugin-on-kafka-output-message/103730 "2017-10-15T19:52:03Z")
hi all, I'm trying to apply XML plugin on kafka messages in logstash. So, the message itself is an xml but in string format, which makes impossible to apply xpath directly without preprocessing the message. Ideally, it … --- ## [Access json fields from string](https://discuss.elastic.co/t/access-json-fields-from-string/103538)
**Author:** [@zuokuok](https://discuss.elastic.co/u/zuokuok)\ **Replies:** 1\ **Last updated:** [October 15, 2017, 7:49pm UTC](https://discuss.elastic.co/t/access-json-fields-from-string/103538 "2017-10-15T19:49:34Z")
I have a valid string within my initial message. I extract this string in a variable "payload" and remove the end of the string which is not part of the json. At this point, my string is a valid JSON. … --- ## [Logstash Configuration doubts](https://discuss.elastic.co/t/logstash-configuration-doubts/103514)
**Author:** [@Balamurali](https://discuss.elastic.co/u/Balamurali)\ **Replies:** 1\ **Last updated:** [October 15, 2017, 7:47pm UTC](https://discuss.elastic.co/t/logstash-configuration-doubts/103514 "2017-10-15T19:47:22Z")
Hi All, We are working on Indexing system logs ( nagios log ) to Elasticsearch using logstash. We have two basic doubts on this . We are going to parse nagios log to logstash for every minute, For this every minut… --- ## [Logstash csv ingestion with single row updates frequently](https://discuss.elastic.co/t/logstash-csv-ingestion-with-single-row-updates-frequently/103416)
**Author:** [@cp\_el](https://discuss.elastic.co/u/cp_el)\ **Replies:** 1\ **Last updated:** [October 15, 2017, 7:43pm UTC](https://discuss.elastic.co/t/logstash-csv-ingestion-with-single-row-updates-frequently/103416 "2017-10-15T19:43:32Z")
I ingested csv in elasticsearch using logstash and also put that ingesting script running in background for continues ingestion. It's ingest if new row added to csv but if I have only one row in csv and it's changes fre… --- ## [Logstash ingest data after any updates in CSV input](https://discuss.elastic.co/t/logstash-ingest-data-after-any-updates-in-csv-input/103415)
**Author:** [@cp\_el](https://discuss.elastic.co/u/cp_el)\ **Replies:** 1\ **Last updated:** [October 15, 2017, 7:42pm UTC](https://discuss.elastic.co/t/logstash-ingest-data-after-any-updates-in-csv-input/103415 "2017-10-15T19:42:54Z")
I ingested csv in elasticsearch using logstash and also put that ingesting script running in background for continues ingestion. It's ingest if new row added to csv but if I have any row values update then it's not upda… --- ## [How to check the count for value within 30 sec frame in logstash](https://discuss.elastic.co/t/how-to-check-the-count-for-value-within-30-sec-frame-in-logstash/104015)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 4\ **Last updated:** [October 15, 2017, 2:26pm UTC](https://discuss.elastic.co/t/how-to-check-the-count-for-value-within-30-sec-frame-in-logstash/104015 "2017-10-15T14:26:42Z")
HI Team, I working in new scenario in that i want to count the field value in the time frame of 30 sec. For example: if i have field produce action value more than 5 times in the timeframe of 30 sec i need to send the… --- ## [Logstash fails to establish output connection with AWS service domain?](https://discuss.elastic.co/t/logstash-fails-to-establish-output-connection-with-aws-service-domain/104013)
**Author:** [@inian](https://discuss.elastic.co/u/inian)\ **Replies:** 1\ **Last updated:** [October 15, 2017, 2:03pm UTC](https://discuss.elastic.co/t/logstash-fails-to-establish-output-connection-with-aws-service-domain/104013 "2017-10-15T14:03:10Z")
I setup a Elasticsearch enabled service with AWS. I setup the conf file in logstash local as specified I ran logstash using following command logstash -f logstash-simple.conf Connections gets established for inpu… --- ## [Lumberjack output contains no data](https://discuss.elastic.co/t/lumberjack-output-contains-no-data/103818)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 1\ **Last updated:** [October 13, 2017, 11:47pm UTC](https://discuss.elastic.co/t/lumberjack-output-contains-no-data/103818 "2017-10-13T23:47:09Z")
Hi, I want to ship data from Logstash A (client) to Logstash B (server). Logtash A will need to decode netflow traffic, add and remove some fields and then send it to Logstash B for further processing and finally send … --- ## [Csv output plugin prints all fields though it should print fields which are specified in parameter 'fields'](https://discuss.elastic.co/t/csv-output-plugin-prints-all-fields-though-it-should-print-fields-which-are-specified-in-parameter-fields/103957)
**Author:** [@ankur.aggarwal](https://discuss.elastic.co/u/ankur.aggarwal)\ **Replies:** 0\ **Last updated:** [October 13, 2017, 8:42pm UTC](https://discuss.elastic.co/t/csv-output-plugin-prints-all-fields-though-it-should-print-fields-which-are-specified-in-parameter-fields/103957 "2017-10-13T20:42:41Z")
Logstash (5.2.0) logstash-input-elasticsearch (4.0.0) logstash-output-csv (3.0.2) logstash-input-elasticsearch is used to take data from a particular index and query results 10+ fields. Below output should print only … --- ## [Logstash getting auto-restarted very frequently](https://discuss.elastic.co/t/logstash-getting-auto-restarted-very-frequently/103812)
**Author:** [@max\_dodo](https://discuss.elastic.co/u/max_dodo)\ **Replies:** 1\ **Last updated:** [October 13, 2017, 6:02pm UTC](https://discuss.elastic.co/t/logstash-getting-auto-restarted-very-frequently/103812 "2017-10-13T18:02:16Z")
I am using UDP input & output plugin and codec =\> json\_lines . Since few days I am observing that my logstash process is getting restarted every few minutes and below error is noted in the logstash logs. Any one else fac… --- ## [Logstash service keeps dying - no logs](https://discuss.elastic.co/t/logstash-service-keeps-dying-no-logs/103925)
**Author:** [@carlfriedrichgauss](https://discuss.elastic.co/u/carlfriedrichgauss)\ **Replies:** 2\ **Last updated:** [October 13, 2017, 4:58pm UTC](https://discuss.elastic.co/t/logstash-service-keeps-dying-no-logs/103925 "2017-10-13T16:58:31Z")
Trying a new ELK stack install against RHEL 7. Java on openjdk version "1.8.0\_144" I'm installing an older version (this is an absolute requirement) and used the logstash-2.1.3-1.noarch.rpm package. ES is up and running… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=890) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=892)