Custom index with field from xpath

Elastic Stack Logstash
1

Filebeat is sending xml files to logstash.

input {
        beats {
                port => "5044"
        }
}

filter {
        ## interpret the message as XML
        if [type] == "nessus-report" {
                xml {
                        source => "message"
                        store_xml => false

                        xpath =>[
                                "/NessusClientData_v2/Report/@name","report_name",
                                "/NessusClientData_v2/Report/ReportHost","report_host"
                        ]
                }
                split {
                        field => "report_host"
                }

                xml {
                        source => "report_host"
                        store_xml => false

                        xpath =>[
                                "/ReportHost/ReportItem","report_item",
                                "/ReportHost/@name","report_host_name",
                                "/ReportHost/HostProperties/tag[@name='HOST_START']/text()","report_host_start",
                                "/ReportHost/HostProperties/tag[@name='HOST_END']/text()","report_host_end"
                           
                        ]
                }
                split {
                        field => "report_item"
                }
                xml {
                        source => "report_item"
                        store_xml => false

                        xpath =>
                        [
                                "/ReportItem/@port","report_item_port"                              
                        ]
                }


                mutate  {
                        remove_field => [ "message","report_host","report_item" ]
                        replace => { "report_host_start" => "%{report_host_start[0]}" }
                        replace => { "report_host_end" => "%{report_host_end[0]}" }
                        convert => { "report_item_severity" => "integer" }

                }
                date {
                        match => [ "report_host_start", "EEE MMM dd HH:mm:ss yyyy" ]
                        target => "report_host_start"
                        locale => "en_US"
                }
                date {
                        match => [ "report_host_end", "EEE MMM dd HH:mm:ss yyyy" ]
                        target => "report_host_end"
                        locale => "en_US"
                }


        }
}

output {
        if [type]=="nessus-report" {
                elasticsearch {
                        hosts => ["10.99.40.16:9200"]
                        manage_template => false
                        index => "nessus-report-%{report_name}-%{+YYYY.MM.dd}"
                        document_type => "nessus-report"
                }
        }
}

I'm trying to insert field report_name into index but the result is:

_index= nessus-report-2017-04-24
type =  %{[@metadata][type]}
report_name = correct-name

can someone help me?
tnks !!

2

You mean the type of the resulting document is "%{[@metadata][type]}"? That's impossible with the configuration you posted.

3

Ok, but what about empty %{report_name} at index? The sintax is correct?
Thank you

4

Ok, but what about empty %{report_name} at index?

I'm not sure what you mean. The index name is "nessus-report-2017-04-24", but that string can't possibly have been produced by the pattern "nessus-report-%{report_name}-%{+YYYY.MM.dd}". As I said, the result you claim you get is impossible with the configuration you've shown us.

5

but that string can't possibly have been produced by the pattern "nessus-report-%{report_name}-%{+YYYY.MM.dd}"

but it is and i don't understand why.

result doc example:
report_name: CTB-AdvNetworkScan
index: nessus-report-2017-04-24

as you see, the field report_name has a value.
more then that, if it was an %{report_name} problem, the index should be "nessus-report--2017-04-24" (with 1 more "-" between report and date)

i'm trying to understand that

this conf file is named nessus-xml.conf.
i have another conf file with an output that doesn't have conditional dependency, named odin.conf.

elasticsearch {
                                hosts => ["10.99.40.16:9200"]
                                manage_template => false
                                index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
                                document_type => "%{[@metadata][type]}"
                        }

can be that?

6

Yeah, that might be the culprit. Logstash sends all events to all outputs unless you wrap them in conditionals.

7

i think that was the problem.

now i'm trying to lowercase and remove whitespaces from the report_name field

 mutate  {
                        add_field => { "index_report_name" => "%{report_name}" }
                        lowercase => [ "index_report_name" ]
                        gsub => [
                                #remove all whitespaces
                                "index_report_name", " ", ""
                        ]                          
                }

but that still not working

[2017-04-25T16:43:02,678][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"nessus-report-VA vsware-2017.04.25", :_type=>"nessus-report", :_routing=>nil}, 2017-04-25T19:41:54.197Z CSB-OdinProxy %{message}], :response=>{"index"=>{"_index"=>"nessus-report-VA vsware-2017.04.25", "_type"=>"nessus-report", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [nessus-report-VA vsware-2017.04.25], must not contain the following characters [ , \", *, \\, <, |, ,, >, /, ?]", "index_uuid"=>"_na_", "index"=>"nessus-report-VA vsware-2017.04.25"}}}}
8

ok, i had to do this:

mutate  {
                        add_field => { "index_report_name" => "%{report_name}" }
                      
                }
mutate{
                        lowercase => [ "index_report_name" ]
              }
mutate{
                        gsub => [
                                #remove all whitespaces
                                "index_report_name", " ", ""
                        ]                          
}

now i have indexes like:

nessus-report-a-2017.04.25
nessus-report-b-2017.04.25
nessus-report-c-2017.04.25
nessus-report-2017.04.25 (this one is index a + b + c. this is normal in elasticsearch?)

9

nessus-report-2017.04.25 (this one is index a + b + c. this is normal in elasticsearch?)

No. You must have an elasticsearch output somewhere that sets index to nessus-report-2017.04.25.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.