How to define custom analyzer to do global search with hibernate-search and elasticsearch

Lauren-D Source

I have an implementation of hibernate-search-orm (5.9.0.Final) with hibernate-search-elasticsearch (5.9.0.Final).

I defined a custom analyzer on an entity (see beelow) and I indexed two entities :

id: "1"
title: "Médiatiques : récit et société"
abstract:...

id: "2"
title: "Mediatique Com'7"
abstract:...

The search works fine when I search on title field :

"title:médiatique" => 2 results.
"title:mediatique" => 2 results.

My problem is when I do a global search with accents (or not) :

search on "médiatique => 1 result (id:1)
search on "mediatique => 1 result (id:2)

Is there a way to resolve this?

Thanks.

Entity definition:

@Entity
@Table(name="bibliographic")
@DynamicUpdate
@DynamicInsert
@Indexed(index = "bibliographic")
@FullTextFilterDefs({
    @FullTextFilterDef(name = "fieldsElasticsearchFilter",
        impl = FieldsElasticsearchFilter.class)
})
@AnalyzerDef(name = "customAnalyzer",
tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
filters = {
    @TokenFilterDef(factory = LowerCaseFilterFactory.class),
    @TokenFilterDef(factory = ASCIIFoldingFilterFactory.class),
})

@Analyzer(definition = "customAnalyzer")
public class BibliographicHibernate implements Bibliographic {
  ...
  @Column(name="title", updatable = false)
  @Fields( {
    @Field,
    @Field(name = "titleSort", analyze = Analyze.NO, store = Store.YES)
  })
  @SortableField(forField = "titleSort")
  private String title;
  ...
}

Search method :

FullTextEntityManager ftem = Search.getFullTextEntityManager(entityManager); 
QueryBuilder qb = ftem.getSearchFactory().buildQueryBuilder().forEntity(Bibliographic.class).get();   
QueryDescriptor q = ElasticsearchQueries.fromQueryString(queryString);
FullTextQuery query = ftem.createFullTextQuery(q, Bibliographic.class).setFirstResult(start).setMaxResults(rows);

if (filters!=null){
  filters.stream().map((filter) -> filter.split(":")).forEach((f) -> {
    query.enableFullTextFilter("fieldsElasticsearchFilter")
      .setParameter("field", f[0])
      .setParameter("value", f[1]);
    }
  );
}
if (facetFields!=null){
  facetFields.stream().map((facet) -> facet.split(":")).forEach((f) ->{
    query.getFacetManager()
      .enableFaceting(qb.facet()
      .name(f[0])
      .onField(f[0])
      .discrete()
      .orderedBy(FacetSortOrder.COUNT_DESC)
      .includeZeroCounts(false)
      .maxFacetCount(10)
      .createFacetingRequest() );
    }
  );
}
List<Bibliographic> bibs = query.getResultList();
javaelasticsearchhibernate-search

Answers

answered 7 months ago yrodiere #1

To be honest I'm more surprised document 1 would match at all, since there's a trailing "s" on "Médiatiques" and you don't use any stemmer.

You are in a special case here: you are using a query string and passing it directly to Elasticsearch (that's what ElasticsearchQueries.fromQueryString(queryString) does). Hibernate Search has very little impact on the query being run, it only impacts the indexed content and the Elasticsearch mapping here.

When you run a QueryString query on Elasticsearch and you don't specify any field, it uses all fields in the document. I wouldn't bet that the analyzer used when analyzing your query is the same analyzer that you defined on your "title" field. In particular, it may not be removing accents.

An alternative solution would be to build a simple query string query using the QueryBuilder. The syntax of queries is a bit more limited, but is generally enough for end users. The code would look like this:

FullTextEntityManager ftem = Search.getFullTextEntityManager(entityManager); 
QueryBuilder qb = ftem.getSearchFactory().buildQueryBuilder().forEntity(Bibliographic.class).get();   
Query q = qb.simpleQueryString()
    .onFields("title", "abstract")
    .matching(queryString)
    .createQuery();
FullTextQuery query = ftem.createFullTextQuery(q, Bibliographic.class).setFirstResult(start).setMaxResults(rows);

Users would still be able to target specific fields, but only in the list you provided (which, by the way, is probably safer, otherwise they could target sort fields and so on, which you probably don't want to allow). By default, all the fields in that list would be targeted.

This may lead to the exact same result as the query string, but the advantage is, you can override the analyzer being used for the query. For instance:

FullTextEntityManager ftem = Search.getFullTextEntityManager(entityManager);
QueryBuilder qb = ftem.getSearchFactory().buildQueryBuilder().forEntity(Bibliographic.class)
        .overridesForField("title", "customAnalyzer")
        .overridesForField("abstract", "customAnalyzer")
        .get();   
Query q = qb.simpleQueryString()
    .onFields("title", "abstract")
    .matching(queryString)
    .createQuery();
FullTextQuery query = ftem.createFullTextQuery(q, Bibliographic.class).setFirstResult(start).setMaxResults(rows);

... and this will use your analyzer when querying.

As an alternative, you can also use a more advanced JSON query by replacing ElasticsearchQueries.fromQueryString(queryString) with ElasticsearchQueries.fromJsonQuery(json). You will have to craft the JSON yourself, though, taking some precautions to avoid any injection from the user (use Gson to build the Json), and taking care to follow the Elasticsearch query syntax.

You can find more information about simple query string queries in the official documentation.

Note: you may want to add FrenchMinimalStemFilterFactory to your list of token filters in your custom analyzer. It's not the cause of your problem, but once you manage to use your analyzer in search queries, you will very soon find it useful.

comments powered by Disqus