(?<=scheduledActionList\s\[)[\d,\s]+)"
]
}
}
if "" in [scheduledActionList] {
mutate {
gsub => ["scheduledActionList", " ", ""]
split => {"scheduledActionList" => ","}
}
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
ilm_enabled => false
index => "%{[fields][log_type]}-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}
```
And this is the filebeat.yml on server-
```auto
#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
#- C:/Users/Administrator/Downloads/filebeat-7.5.1-windows-x86_64/filebeat-7.5.1-windows-x86_64/access.2020-01-09.log
- C:\Program Files (x86)\ESQ SST\DataEdgev1.2\ngta-distribution-web-3.2.0.0-bin\logs
#- c:\programdata\elasticsearch\logs\*
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["192.168.xx.xx:5044"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
```
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 22, 2020, 9:56pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/5 "2020-01-22T21:56:00Z")
@yaaule, I have also done port forwarding so kibana can be seen on my IP:5601 port with host Ip is 192.168.x.x and guest Ip 10.0.x.x and both ports set to 5601.
---
### Author:  [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [January 22, 2020, 9:59pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/6 "2020-01-22T21:59:42Z")
10.x.x.x IPs are [Private IPs](https://tools.ietf.org/html/rfc1918#section-3), which means that the host is not necessarily reachable by hosts outside the private network.
Does the host on which Filebeat is running have a route to the VM running on your local machine (or even just to your local machine)?
If not, you will likely need to rely on port forwarding here too (e.g., forwarding a port on the Filebeat server's loopback interface to your VM by means of an ssh tunnel)
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 22, 2020, 10:04pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/7 "2020-01-22T22:04:54Z")
@yaauie, The 10.x.x.x is my Ubuntu's Ip and 192.168.x.x my host's.
FIlebeat is running on a server with ip 192.168.x.x and it doesnt have a route to the VM on my local machine yet. How can that be done?
> [@yaauie](#):
>
> forwarding a port on the Filebeat server's loopback interface to your VM by means of an ssh tunnel
Here, filebeat on the server's Ip will be host Ip and guest Ip will be my VM's?
Is there a way I can send logs from my filebeat on server to my local machine? I want to take this step first! And then i will try to send from machine to Vm. I f this wont work as a different machine, then i will know my logstass conf has issues.
---
### Author:  [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [January 22, 2020, 10:21pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/8 "2020-01-22T22:21:58Z")
if you have credentials to log into the host on which Filebeat is running, you can ssh tunnel from your local machine (or the VM), and use that tunnel to bind to a port on the remote machine back to your local machine. This is called [Remote Forwarding](https://www.ssh.com/ssh/tunneling/example#remote-forwarding) and can be done using the `-R` flag:
```auto
ssh -R '5044:localhost:5044' username@filebeat_host
```
The above uses the `port:host:hostport` form of the argument for `-R`, where:
- `port`: the port on the server that should listen for new connections
- `host`: the host that should receive connections that were sent to the listening port on the server (`localhost` here is your local machine)
- `hostport`: the port on the host that is receiving connections
So, if you have Logstash running on a VM with IP `10.0.0.7`, I believe you could do something like the following on your VM host to forward inbound requests directly to it:
```auto
ssh -R '5044:10.0.0.7:5044' username@filebeat_host
```
By default, SSH will bind the forwarded port _only_ to the loopback interface, which means that to send to the port that has been forwarded, you would configure Filebeat to send to `127.0.0.1:5044`.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 22, 2020, 10:28pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/9 "2020-01-22T22:28:30Z")
I do have credentials to log into the server on which filebeat is installed. So in the command-
> [@yaauie](#):
>
> ```auto
> ssh -R '5044:localhost:5044' username@filebeat_host
>
> ```
username would be username and filebeat\_host would be the 192.168.x.x server IP, right?
> [@yaauie](#):
>
> By default, SSH will bind the forwarded port _only_ to the loopback interface, which means that to send to the port that has been forwarded, you would configure Filebeat to send to `127.0.0.1:5044` .
So why 127.0.0.1:5044 in filebeat.yml? and this will be changed in the output block in filebeat.yml, right? And would this mean that when filebeat on server runs, it would get logs and send to logstash on the VM?
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 22, 2020, 10:34pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/10 "2020-01-22T22:34:47Z")
After trying the command, I get logged into server and this terminal comes up-
Not sure if this means that the remote forwarding has started?
And I have changed the filebeat hosts to 127.0.0.1:5044. Should I run filebeat and see if logstash takes it in?
---
### Author:  [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [January 22, 2020, 10:36pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/11 "2020-01-22T22:36:56Z")
> username would be username and filebeat\_host would be the 192.168.x.x server IP, right?
yes
> So why 127.0.0.1:5044 in filebeat.yml?
Because once you have opened a Remote Forwarding tunnel, port `5044` on the Filebeat server's loopback interface will be forwarded through the tunnel.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 22, 2020, 10:42pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/12 "2020-01-22T22:42:08Z")
@yaauie, sorry I realize we shouldnt send screenshots but i wasnt sure how to explain this terminal otherwise!
After the remote forwarding, I tried starting filebeat on server but it isnt starting up. How do I know the tunnel is working?
---
### Author:  [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [January 22, 2020, 11:25pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/13 "2020-01-22T23:25:38Z")
> I tried starting filebeat on server but it isnt starting up.
You will need to look at Filebeat's logs to figure out why it is not starting.
* * *
ssh isn't good at logging the remote forwarding, and it just kindof happens silently. It looks like you can also pass `-N` (for **N** o remote command), which will cause the ssh command to just _hang_ there as long as the remote tunnel is open.
> ```auto
> -N Do not execute a remote command. This is useful for just for-
> warding ports.
>
> ```
And `-v` will make the connection log verbosely (in my case, it includes a message `Remote connections from LOCALHOST:5055 forwarded to local address 10.0.0.7:5044` and a relevant success message for the below examples).
* * *
It's easy to get things a bit confused when all the port numbers are the same. In the below example, we will end up starting a logstash listening for Beats on port `5044` on a VM that is accessible to our localhost via IP `10.0.0.7` (substitute your VM's IP as necessary)
Then, on our local machine, we create the Remote Forwarding tunnel:
```auto
ssh -v -N -R '5055:10.0.0.7:5044' username@192.168.1.1
```
This would:
- connect to `192.168.1.1` (e.g, filebeat server's IP)
- cause the server to bind to port `5055` on its own loopback interface, forwarding inbound connections back through our tunnel
- cause our local machine to forward connections it receives through the tunnel to`10.0.0.7`'s port `5044`
Finally, we would configure Filebeat on the server to point to port `5055` in its own loopback, thereby using the tunnel we just created.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 23, 2020, 12:45am UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/14 "2020-01-23T00:45:31Z")
@yaauie Please let me know how to revert back the ssh tunneling My logstash was listening before and now its stopped. I want to reverse back the forwarding! Please let me know.
---
### Author:  [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [January 23, 2020, 1:10am UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/15 "2020-01-23T01:10:28Z")
When you close the SSH session that is doing the tunneling, the tunnel will also close.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 23, 2020, 6:10pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/16 "2020-01-23T18:10:00Z")
@yaauie I tried this step and it almost works in a way that tunnel is there and collects logs, but I think it didnt get shipped since no new indexes were made.
In the meantime, I did figure out how to send logs from my machine to the VM. Just added port forwarding rule for logstash and through that logs were being shipped from filebeat on my machine to logstash in my VM. And now the logs are shipped from server to VM also.
But I want to undertstand one thing- filebeat is running from server and the filebeat.yml file goes to pipeline.conf on my ubuntu machine and takes that pipeline's configuration, right?
I am concerned because the index name in my logstash configuration file, called pipelines.conf, is in [fields][log\_type]-date format whereas the index being made right now is "logstash" which means ILM is in action. how do I fix it? what logstash configuration file needs to get changed?
---
### Author:  [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [January 24, 2020, 10:01pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/17 "2020-01-24T22:01:19Z")
ILM doesn't work along-side index patterns that reference field values, because ILM configuration needs to know about _specific_ index aliases at startup and cannot know all possible values that the pipeline will use to populate the index pattern. So if you specify `index` and rely on the value of fields (e.g., using `%{...}` format placeholders), you should set `ilm_enabled => false`.
If you're still seeing new data flowing into `logstash-*` indices, this indicates that you have an Elasticsearch output plugin that is using the default value for `index`.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 24, 2020, 10:55pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/18 "2020-01-24T22:55:37Z")
@yaauie, Thanks for replying! I do have ILM set to false as you mentioned. Now my index name is stuck at "%{[fields][tag]}". I know what mistake I made which is that in logstash output block, I should have written
```auto
index => "%{[fields][tags]"
```
But now, when i try to delete in dev tools of kibana under
```auto
DELETE /%{[fields][tag]}
```
the index isnt deleting. I tried DELETE /%25{[fields][tag]} as well and that isnt working either.
I do notice that when I select this DELETE command, the grey area that usually is one line selection, currently the grey area selected is taking the commands above it too.
> [@yaauie](#):
>
> Elasticsearch output plugin that is using the default value for `index` .
Where does this exist?
My index from logstash moved onto this new one after I made sure my pipeline.conf was the right configuration being read.
Thanks!
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 25, 2020, 1:19am UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/20 "2020-01-25T01:19:12Z")
Fixed this. Deleted it under Index Management.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 22, 2020, 1:19am UTC](https://discuss.elastic.co/t/how-to-send-logs-from-server-to-local-machine-to-vm-where-elk-is-running/216120/21 "2020-02-22T01:19:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.