Querying only logs coming from source x

tomer · June 18, 2017, 1:39pm

Hi,

I am trying to search only logs that come from a certain source.

Here is what I tried to do:

GET filebeat-*/log/_search
{
"query": {
"bool":{
"must": [
{ "match": { "source": "/opt/modulo/smsc/cdr/traffic.cdr" } }
]
}
}
}

From some reason this retrieves me all logs also logs that doent come from this exact path.

How can I retrieve only logs from this path?

thiago · June 19, 2017, 1:52am

I could not reproduce this using a vanilla setup. Maybe the template is wrong? Please post the output of GET _template/filebeat.

tomer · June 20, 2017, 7:33am

Hi I get an empty reply, what is this query?

thiago · June 20, 2017, 1:50pm

There should have been a template for your index. Probably the absence of it is the reason why you are querying and more is coming. What about GET _template? Can you post the output?

tomer · June 22, 2017, 10:16am

Ummm... from some reason also this gives me a blank message

thiago · June 22, 2017, 10:52am

I think I know what's going on, but I need one last confirmation. Expand the filebeat-* index pattern into an existing index, for instance, filebeat-2017.06.22 (you can get a list of existing indices with GET _cat/indices) and then get the output of GET filebeat-2027.06.22/_mapping. Post this output here please.

tomer · June 25, 2017, 6:50am

Hi,

Sorry about the delays,

The query is:

GET filebeat-2017.06.14/_mapping

The output is pretty long. Since the the reply is limited to 7000 chars I will split the output:

{
  "filebeat-2017.06.14": {
    "mappings": {
      "log": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "Jms_Lai_TS": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "apiMedTimeGot": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "apiMedTimeSent": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "authCBGot": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "authCBSent": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "beat": {
            "properties": {
              "hostname": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "name": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "version": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "billable": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "blackListCBGot": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "blackListCBSent": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },

tomer · June 25, 2017, 6:51am

Second part (probably not an useful part:

      "calledImsi": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "calledMsc": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "calledMscNpi": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "calledMscTon": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "calledNumber": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "callingImsi": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "callingMsc": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "callingNumber": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "campignId": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "cdrType": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "channel": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "clientId": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "concatenated": {
        "type": "boolean"
      },
      "concatenatedFrom": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "deferred": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "destinationProtocol": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "gmt1": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "gmt2": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "host": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "input_type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },

thiago · June 25, 2017, 12:35pm

The issue here is that the proper filebeat template was not created. If you are not indexing directly in elasticsearch from filebeat then you need to load the index template before indexing any data.

tomer · June 27, 2017, 7:32am

Hi,

Thanks, I didnt completely understand what does this mean, but I tried to fix any way.

since I am sending from Filebeat to Logstash to ES I understood I have to do this manually. To this manually in Centos I understood I wrote something like this:

curl -H 'Content-Type: application/json' -XPUT 'http://192.168.1.116:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json

Is this right?

I still get for the query:

GET filebeat-*/log/_search
{
"query": {
"bool":{ 
"must": [
{ "match": { "source": "\/path_to_files" } }
]
}
}
}

all the logs and not only the logs in the path I wanted.

On the other I dont get now for:
GET _template

an empty answer.

tomer · June 27, 2017, 7:51am

Maybe there is another way all together to obtain logs only from a certain source?
By the way I have control to put the logs on different directories if this can help

thiago · June 27, 2017, 10:10am

The command is wrong, there is a missing space, it should be:

curl -H 'Content-Type: application/json' -XPUT 'http://192.168.1.116:9200/_template/filebeat' -d @/etc/filebeat/filebeat.template.json

You need to get it working, this is crucial. You may verify it worked by running GET _template and verifying that it is not an empty response.

Lastly, after you properly create the template, for the template to be effective, you will need to delete all filebeat-* indices and reindex everything again.

Note: if you can not reindex externally, then let me know, since there is a way to ask Elasticsearch to reindex internally

system · July 25, 2017, 10:10am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.