Fields, type mapping

Elastic Stack Logstash
1

Hi,
We are having one system where error and info logs are created in different files as below

/var/log/sales/salesbackedERROR.log,
/var/log/sales/salesbackendINFO.log,
/var/log/sales/crmERROR.log,
/var/log/sales/crmINFO.log

**Note:**Above are examples from salesbackend and crm only. There are other applications as well.

When I configure LSF, I create fields as below:
Sub-System, Application, and type
And I am doing indexing on the basis of Application name. "%Application%_YYYY_MM_DD"

  "files": [
    {
      "paths": [ "/var/log/sales/salesbackendERROR.log"  ],
      "fields": {"Application":"salesbackend","Sub-System":"Servicepoint", "type": "saleserror" }
    },
    {
      "paths": [ "/var/log/sales/salesbackendINFO.log" ],
      "fields": { "Application": "salesbackend", "Sub-System":"servicepoint","type":"salesinfo" }
    }, {
      "paths": [ "/var/log/sales/crmERROR.log"  ],
      "fields": {"Application":"crm","Sub-System":"servicepoint", "type": "crmerror" }
    },
    {
      "paths": [ "/var/log/sales/crmINFO.log" ],
      "fields": { "Application": "crm", "Sub-System":"servicepoint","type":"crminfo" }
    }

User wants only one category for error and info logs so that they can see all errors in sales and crm (there are other 10 application also). He wants simple naming convetion so that queries would be simple.
Application: salesbackend and type:error
They don't want type as "saleserror" or "crmerror"
I hope if I do that, it wont store all errors from sales or crm under same table(type mapping).

I want to know, how these fields will be mapped in DB.
I understand there will be separate database (Index) for each application (sales, crm) and under each index, there will be separate table (type-mapping) for error, info. Is this correct?

How will this impact (positive or negative) on performance? Is there any good suggestion to optimize this?

Please throw some light on third field Sub-system. is it wise to create 3 fields. Sub-System can help me to get error logs from all application in simple query: Sub-System:"servicepoint" AND type:"error"

br,
Sunil

2

Hello,
Anybody has any good suggestion?

br,
Sunil

3

You really need to move off LSF, it is deprecated and will be completely unsupported some point in the near future.

4

Hi,
I will surely do that. But current timelines and task list doesn't allow me to do that.
I have to get things running someo how in this current pipeline. However, I have to plan replacement of LSF with filebeat and timebox this activity.

br,
Sunil

5

They don't want type as "saleserror" or "crmerror"

Sure, no problem.

I understand there will be separate database (Index) for each application (sales, crm) and under each index, there will be separate table (type-mapping) for error, info. Is this correct?

Whether you have different indexes for different applications depends on your Logstash configuration.

Please throw some light on third field Sub-system. is it wise to create 3 fields. Sub-System can help me to get error logs from all application in simple query: Sub-System:"servicepoint" AND type:"error"

You'll be fine with these fields.

6

Thanks a lot Magnus. :slight_smile: