ElasticSearch.net AND NEST - Indexing without storing file content

Elastic Stack Elasticsearch
1

I am using the NEST API with Elasticsearch.NET, and Visual Basic. With the latest version of Elasticsearch how can I create an index so that the when a file is indexed the entire file content is not stored on the server?

Below are my current calls to Indices.Create and PutPipeline. My understanding is that the AllField constructor is now obsolete and ignored - but I think this is what stopped the file content from being stored in in previous versions with the CreateIndex method.

            Dim result = client.Indices.Create(index_name,
                                                Function(ms) ms.Map(Of IndexFile)(
                                                    Function(m) m.AutoMap().AllField(
                                                        Function(al) al.Enabled(False)).Properties(
                                                            Function(ps) ps.Object(Of Nest.Attachment)(
                                                                Function(a) a.Name(
                                                                    Function(n) n.Attachment).AutoMap()))
                                        ).Settings(
                                            Function(a) a.Analysis(
                                                Function(aa) aa.Analyzers(
                                                    Function(am) am.Custom("filename_analyzier",
                                                        Function(ca) ca.Filters("lowercase", "stop").Tokenizer("whitespace"))))))

           Dim putPipelineResponse = client.Ingest.PutPipeline("attachments",
                                                Function(p) p.Description("Document Attachment Pipeline").Processors(
                                                    Function(pr) pr.Attachment(Of IndexFile)(
                                                        Function(a) a.Field(Function(f) f.FileContent).TargetField(
                                                            Function(f) f.Attachment)).Remove(Of IndexFile)(
                                                                Function(r) r.Field(
                                                                    Function(f) f.Field(Of String)(
                                                                        Function(ff) ff.FileContent)))))
2

Can I just revise my class something like?:

Public Class IndexFile

        Public Property Attachment As Attachment
        <ElasticProperty(Store:=False)>
        Public Property FileContent As String
        Public Property FileName As String
        Public Property Description As String

End Class
3

You can use

Public Class IndexFile

        Public Property Attachment As Attachment
        <Text(Store:=False, Index:=false)>
        Public Property FileContent As String
        Public Property FileName As String
        Public Property Description As String

End Class

so that the file content is not stored or indexed in Elasticsearch.

What might be better though is to remove the file content field in the ingest pipeline with a remove processor, after running the attachment processor, like this blog post does.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.