# 2 conf sending data to the same index
**URL:** https://discuss.elastic.co/t/2-conf-sending-data-to-the-same-index/333888
**Category:** Logstash
**Created:** [May 19, 2023, 8:40pm UTC](https://discuss.elastic.co/t/2-conf-sending-data-to-the-same-index/333888 "2023-05-19T20:40:45Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@jefin\_dark](https://discuss.elastic.co/u/jefin_dark)
#### Post date: [May 19, 2023, 8:40pm UTC](https://discuss.elastic.co/t/2-conf-sending-data-to-the-same-index/333888/1 "2023-05-19T20:40:45Z")
Hello,
I have 2 conf files and they are sending data at the same time to the 2 index (when I would like each conf to send the information to the specific index)
If you can help me, I can provide more information if needed.
Below is my configuration.
Firts.
```auto
input {
beats {
port => 5044
}
}
filter {
grok {
match => {
"[event_data][Data]" => [
"Subject:\s*Security ID:\s*%{DATA:security_id}",
"Subject:\s*Account Name:\s*%{DATA:account_name}",
"Subject:\s*Account Domain:\s*%{DATA:account_domain}",
"Subject:\s*Logon ID:\s*%{DATA:logon_id}",
"Object:\s*Object Server:\s*%{DATA:object_server}",
"Object:\s*Object Type:\s*%{DATA:object_type}",
"Object:\s*Object Name:\s*%{DATA:object_name}",
"Object:\s*Handle ID:\s*%{DATA:handle_id}",
"Access:\s*Accesses:\s*%{DATA:accesses}",
"Access:\s*Access Mask:\s*%{DATA:access_mask}",
"Access:\s*Privileges:\s*%{DATA:privileges}"
]
}
}
}
output {
elasticsearch {
hosts => ["https://192.168.12.109:9200"]
index => "srvvmfs01_log-%{+YYYY.MM.dd}"
user => "elastic"
password => "XXXXX"
ssl => true
cacert => "/etc/logstash/http_ca.crt"
}
}
```
Second
```auto
input {
udp {
port => 5514
codec => plain
}
}
filter {
grok {
match => {
"message" => "<%{POSINT:priority}>%{MONTH:month} %{MONTHDAY:day} %{TIME:time} %{DATA:hostname} %{WORD:event_type} %{GREEDYDATA:message_data}"
}
}
mutate {
convert => { "priority" => "integer" }
}
if [event_type] == "user" {
grok {
match => {
"message_data" => "admin logged in from %{IP:source_ip} via %{WORD:login_method}"
}
}
}
if [event_type] == "filter" {
grok {
match => {
"message_data" => "rule %{WORD:rule_action} by %{DATA:rule_modifier}"
}
}
}
}
output {
elasticsearch {
hosts => ["https://192.168.12.109:9200"]
index => "mikrotik_log-%{+YYYY.MM.dd}"
user => "elastic"
password => "XXXXXXXX"
ssl => true
cacert => "/etc/logstash/http_ca.crt"
}
}
```
pipeline.yml (locate on /etc/logstash/)
```auto
- pipeline.id: mikrotik
path.config: "/etc/logstash/conf.d/mikrotik-log.conf"
- pipeline.id: srvvmfs01
path.config: "/etc/logstash/conf.d/srvvmfs01-log.cof"
```
logstash conf service
```auto
[Unit]
Description=logstash
[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
#EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.config /etc/logstash/conf.d/*.conf
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384
# When stopping, how long to wait before giving up and sending SIGKILL?
# Keep in mind that SIGKILL on a process can cause data loss.
TimeoutStopSec=infinity
[Install]
WantedBy=multi-user.target
```
when some information is sent , by any of the conf, it duplicates in the 2 index
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [May 19, 2023, 10:55pm UTC](https://discuss.elastic.co/t/2-conf-sending-data-to-the-same-index/333888/2 "2023-05-19T22:55:21Z")
@jefin_dark Thanks for all the detail
Your pipelines.yml looks correct but
> [@jefin\_dark](#):
>
> ```auto
> --path.config /etc/logstash/conf.d/*.conf
>
> ```
Curious did you create / edit the service file yourself or is that the default?
Seems to me the line above is the problem... that will concatonate the 2 files together which would explain what you are seeing ... Or your pipelines.yml not being read and therefore the files and the directory are being concatenated... Either way, pretty sure that's the issue
Seems like that should be
`--path.settings /etc/logstash`
I will need to triple check, but I would test with that first.
Yup I checked mine
```auto
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
```
If you intentionally want to concatenate them together which I don't think you want to, but if you do we can just put in logic so that the input will get a tag and then the output will be conditional on that tag
---
### Author:  [@jefin\_dark](https://discuss.elastic.co/u/jefin_dark)
#### Post date: [May 20, 2023, 12:55am UTC](https://discuss.elastic.co/t/2-conf-sending-data-to-the-same-index/333888/3 "2023-05-20T00:55:56Z")
@stephenb , thanks again for your attention
yes, i made intentionaly change, because this way, charge every file on conf.d, but i dont know that is correct. 🙂
when i put the service conf like you say
```auto
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
```
only one conf is up, but works correctly, just only one index has created and process
How can i made these TAGS for work correct.
1 Index for 1 Conf.
Thank you in advance
Regards.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [May 20, 2023, 1:00am UTC](https://discuss.elastic.co/t/2-conf-sending-data-to-the-same-index/333888/4 "2023-05-20T01:00:57Z")
Hi @jefin_dark
Let's take a moment and debug why you don't now have 2 independent pipelines... because Your instincts are correct ...
Ahhh found it... typo... in pipelines.yml
```auto
- pipeline.id: srvvmfs01
path.config: "/etc/logstash/conf.d/srvvmfs01-log.cof"
```
Missing the `n` should be
```auto
- pipeline.id: srvvmfs01
path.config: "/etc/logstash/conf.d/srvvmfs01-log.conf"
.....................................................^
```
This is why "Pair Programming" is good!!
---
### Author:  [@jefin\_dark](https://discuss.elastic.co/u/jefin_dark)
#### Post date: [May 20, 2023, 1:14am UTC](https://discuss.elastic.co/t/2-conf-sending-data-to-the-same-index/333888/5 "2023-05-20T01:14:18Z")
it was a typo on my part. I'm sorry
On the pipeline.yml is correct ☹
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [May 20, 2023, 1:29am UTC](https://discuss.elastic.co/t/2-conf-sending-data-to-the-same-index/333888/6 "2023-05-20T01:29:59Z")
Hi @jefin_dark
Apologies I am not clear what you are saying...
a) Your pipeline.yml has the correct path and it is still not working?
b) Or now it is fixed and it is working?
if a) That would indicate that If the path is correct and it is still not working there is a different issue... like logstash can not find / access the .conf file or there is an error in the conf file. You need to show the logstash startup logs...
**I would comment out the mikrotik pipeline in the pipeline.yml so we can focus on the other one...**
Things to check check list the actual file ... is is there... are the permissions correct?
`ls -l /etc/logstash/conf.d/srvvmfs01-log.conf`
You can also just start logstash in the foreground with the following command and watch the logs....
```auto
/usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.config /etc/logstash
```
There is something simple at this point... most likely has to do with the pipelines.yml or the path... because when you used \*.conf 2 pipelines ran which indicates there is not an error in the actual .conf file.
Minor I don't think you need this on the beats
```auto
codec => plain
```
Do you have a line feed? You know blank line after the last line in the pipeline.yml
---
### Author:  [@jefin\_dark](https://discuss.elastic.co/u/jefin_dark)
#### Post date: [May 20, 2023, 10:30pm UTC](https://discuss.elastic.co/t/2-conf-sending-data-to-the-same-index/333888/7 "2023-05-20T22:30:38Z")
Hi @stephenb , sorry for my english.
I'm a Brazilian who speaks very bad English, so sometimes I express myself wrong 🤣
A) My pipeline.yml file was not ok.
B) I made the correction, as you pointed out and now everything is working perfectly.
The first was that I changed the logstash service file incorrectly.
The second was from so much editing the config pipeline.yml I ended up forgetting the N
Now everything is working as it should.
Follow the prints.
Thank you immensely for your help. 🤝
I'm sorry for the beginner's mistakes, but it's really a new environment for me.
I hope I can evolve here in my analysis.
A good rest.
Regards
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [May 20, 2023, 10:41pm UTC](https://discuss.elastic.co/t/2-conf-sending-data-to-the-same-index/333888/8 "2023-05-20T22:41:47Z")
@jefin_dark
You speak/ write great English...
My Portuguese is "_inexistente_" 🙂
Glad you got is solved
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 17, 2023, 10:41pm UTC](https://discuss.elastic.co/t/2-conf-sending-data-to-the-same-index/333888/9 "2023-06-17T22:41:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.