Advanced filtering and full-text search

The JPA Criteria API can provide support for the implementation of optional filter clauses. For long text columns, Hibernate Search can provide a full-text search.

The Movie Manager project shows a combination of Hibernate Search and the JPA Criteria API. The API is used to create a searchable user interface for movies and actors.

Using Hibernate Search and JPA Criteria API

The MovieManager project stores text data for movie overviews and actor biographies. New filter functions for movies and actors include a full-text search for text data. The JPA Criteria API is used to implement additional filter functions to help with optional query components such as age or release date.

backend

MovieController has a new relaxing interface:

@RequestMapping(value = "/filter-criteria", method = RequestMethod.POST, produces = MediaType.APPLICATION_JSON_VALUE, consumes = MediaType.APPLICATION_JSON_VALUE)
public List<MovieDto> getMoviesByCriteria(@RequestHeader(value =  
   HttpHeaders.AUTHORIZATION) String bearerStr,
   @RequestBody MovieFilterCriteriaDto filterCriteria) {
   return this.service.findMoviesByFilterCriteria(bearerStr,  
      filterCriteria).stream().map(m -> this.mapper.convert(m)).toList();
}

Actors controller has a similar rest interface:

@RequestMapping(value = "/filter-criteria", method = RequestMethod.POST,  
   produces = MediaType.APPLICATION_JSON_VALUE, consumes =   
   MediaType.APPLICATION_JSON_VALUE)
public List<ActorDto> getActorsByCriteria(@RequestHeader(value = 
   HttpHeaders.AUTHORIZATION) String bearerStr,
   @RequestBody ActorFilterCriteriaDto filterCriteria) {
      return this.service.findActorsByFilterCriteria(bearerStr,  
         filterCriteria).stream().map(m -> this.mapper.convert(m)).toList();
}

These are the rest endpoints that require a posted JSON, which is mapped with @RequestBody Annotation to FilterCriteria DTO. DTO is used to call the service for filtering.

search services

The Actor service and the Movie service implement the services for filtering. Movie service shown here:

public List<Movie> findMoviesByFilterCriteria(String bearerStr, 
   MovieFilterCriteriaDto filterCriteriaDto) {
   List<Movie> jpaMovies = 
      this.movieRep.findByFilterCriteria(filterCriteriaDto,
         this.auds.getCurrentUser(bearerStr).getId());
   SearchTermDto searchTermDto = new SearchTermDto();	  
   searchTermDto.setSearchPhraseDto(filterCriteriaDto.getSearchPhraseDto());
   List<Movie> ftMovies = this.findMoviesBySearchTerm(bearerStr, 
      searchTermDto);
   List<Movie> results = jpaMovies;
   if (filterCriteriaDto.getSearchPhraseDto() != null && 
      !Objects.isNull(filterCriteriaDto.getSearchPhraseDto().getPhrase()) && 
      filterCriteriaDto.getSearchPhraseDto().getPhrase().length() > 2) {
      Collection<Long> dublicates =  CommonUtils.
         findDublicates(Stream.of(jpaMovies,ftMovies)
         .flatMap(List::stream).toList());
      results = Stream.of(jpaMovies, ftMovies).flatMap(List::stream)
         .filter(myMovie -> CommonUtils.
            filterForDublicates(myMovie, dublicates)).toList();
      // remove dublicates
      results = results.isEmpty() ? ftMovies : 
      List.copyOf(CommonUtils.filterDublicates(results));
   }
   return results.subList(0, results.size() > 50 ? 50 : results.size());
}

public List<Movie> findMoviesBySearchTerm(String bearerStr, 
   SearchTermDto searchTermDto) {
   List<Movie> movies = searchTermDto.getSearchPhraseDto() != null ?   
      this.movieRep.findMoviesByPhrase(searchTermDto.getSearchPhraseDto()) :  
      this.movieRep.
         findMoviesBySearchStrings(searchTermDto.getSearchStringDtos());
   List<Movie> filteredMovies = movies.stream().filter(myMovie -> 
      myMovie.getUsers().stream().anyMatch(myUser -> myUser.getId()
      .equals(this.auds.getCurrentUser(bearerStr).getId()))).toList();
   return filteredMovies;
}

findMoviesByFilterCriteria(...) The method first calls the JPA repository to select the movies. process getCurrentUser(...) Finds the user entity for which the JWT token was issued and returns the ID. Movies contains a database related to the user. Because of this relationship, a movie is stored in the table only once and is used by all users who have imported it.

Then SearchTermDto made to call findMoviesBySearchTerm(...) Method for full-text search. uses the method MovieRep To perform a search in the ‘Overview’ index of movies and to filter the results for the current user’s movies.

Then the results of JPA query and full-text search are merged in 3 steps:

  1. findDublicates(...) Returns the ID found in both search results.
  2. filterForDublicates(...) Returns the entities of ID.
  3. filterDublicates(...) Removes objects with duplicate IDs and returns them. If no normal results are found, full-text search results are returned.

Combined results are limited to 50 entities and are returned.

data repository

MovieRepositoryBean and ActorRepositoryBean implement JPA Criteria searches and HibernateSearch searches. jpa search of MovieRepositoryBean Shown here:

public List<Movie> findByFilterCriteria(MovieFilterCriteriaDto 
   filterCriteriaDto, Long userId) {
   CriteriaQuery<Movie> cq = this.entityManager.getCriteriaBuilder()
      .createQuery(Movie.class);
   Root<Movie> cMovie = cq.from(Movie.class);
   List<Predicate> predicates = new ArrayList<>();
  ...
   if (filterCriteriaDto.getMovieTitle() != null &&   
      filterCriteriaDto.getMovieTitle().trim().length() > 2) {
      predicates.add(this.entityManager.getCriteriaBuilder().like(
         this.entityManager.getCriteriaBuilder()
            .lower(cMovie.get("title")), 
               String.format("%%%s%%", filterCriteriaDto
                  .getMovieTitle().toLowerCase())));
   }
   if (filterCriteriaDto.getMovieActor() != null && 
      filterCriteriaDto.getMovieActor().trim().length() > 2) {
      Metamodel m = this.entityManager.getMetamodel();
      EntityType<Movie> movie_ = m.entity(Movie.class);
      predicates.add(this.entityManager.getCriteriaBuilder()
         .like(this.entityManager.getCriteriaBuilder().lower(
            cMovie.join(movie_.getDeclaredList("cast", Cast.class))
	       .get("characterName")),
               String.format("%%%s%%", filterCriteriaDto.
                  getMovieActor().toLowerCase())));
   }
   ...
   // user check
   Metamodel m = this.entityManager.getMetamodel();
   EntityType<Movie> movie_ = m.entity(Movie.class);
   predicates.add(this.entityManager.getCriteriaBuilder()
      .equal(cMovie.join(movie_.getDeclaredSet("users", User.class))
      .get("id"), userId));
   cq.where(predicates.toArray(new Predicate[0])).distinct(true);
   return this.entityManager.createQuery(cq)
      .setMaxResults(1000).getResultList();
}

this part of the law filterByCriteria(...) Shows criteria for movie title search and actor name with join search.

First, the criteria query and route movie objects are created. The predicate list is created to contain the query criteria for the search.

The title of the film is checked for its existence and its minimum length. The EntityManager is used to create a ‘like’ criteria that has a ‘bottom’ function for the ‘title’ property. The title string is converted to lowercase and surrounded by “%%” to find all headings containing a case-insensitive string. Then the criterion is added to the predicate list.

Actor’s name is checked for its existence and its minimum length. Then the JPA metamodel is created to get the EntityType to join the cast entity. EntityManager is used to create ‘Like’ and ‘Follow’ criteria. The parent entity (‘cMove’) is used to join the cast entity to the query. The ‘charactername’ of the cast entity is used in the ‘like’ criteria. The actor name string for search is converted to lowercase and surrounded by “%%” to find all actor names containing the search string. The full actor criterion is eventually added to the predicate list.

Then the user check criteria is created and added to the predicate list in the same way as the actor name search criteria.

The criterion predicate is added to CriteriaQuery.where(...) And a distinct(true) call is added to remove duplicates.

The query result is limited to 1000 units to protect the server from I/O and memory overloads.

full Text search

Full-text search is implemented in findMoviesBySearchPhrase(...) Method of MovieRepository:

@SuppressWarnings("unchecked")
public List<Movie> findMoviesByPhrase(SearchPhraseDto searchPhraseDto) {
   List<Movie> resultList = List.of();
   if (searchPhraseDto.getPhrase() != null && 
      searchPhraseDto.getPhrase().trim().length() > 2) {
      FullTextEntityManager fullTextEntityManager = 
         Search.getFullTextEntityManager(entityManager);
      QueryBuilder movieQueryBuilder = 
         fullTextEntityManager.getSearchFactory().buildQueryBuilder()
         .forEntity(Movie.class).get();
      Query phraseQuery = movieQueryBuilder.phrase()
         .withSlop(searchPhraseDto.getOtherWordsInPhrase())
         .onField("overview")
         .sentence(searchPhraseDto.getPhrase()).createQuery();
      resultList = fullTextEntityManager
         .createFullTextQuery(phraseQuery, Movie.class)
         .setMaxResults(1000).getResultList();
   }
   return resultList;
}

process findMoviesByPhrase(...) Is SearchPhraseDto as a parameter. Which has properties:

  • otherWordsInPhrase whose default value is 0.
  • ‘Phrase’ containing the search string for movie overview in hibernate search index.

The existence and length of the phrase are checked. Then FullTextEntityManager and QueryBuilder are created. ‘QueryBuilder’ is used to build a full-text query with the search parameter ‘Phrase’ on the Movie entity field ‘Overview’. otherWordsInPhrase is added to FullTextEntityManager with withSlop(…) parameter.

FullTextEntityManager Used to execute full-text queries on the movie ‘Overview’ index with a limit of 1000 results. The limit is set to protect the server from I/O and memory overloads.

keeping hibernate search indices updated

In the CronJobs class the Hibernate search index is checked at application start for necessary updates:

@Async
@EventListener(ApplicationReadyEvent.class)
public void checkHibernateSearchIndexes() throws InterruptedException {
   int movieCount = this.entityManager.createNamedQuery("Movie.count", 
      Long.class).getSingleResult().intValue();
   int actorCount = this.entityManager.createNamedQuery("Actor.count", 
      Long.class).getSingleResult().intValue();
   FullTextEntityManager fullTextEntityManager = 
   Search.getFullTextEntityManager(entityManager);
   int actorResults = checkForActorIndex(fullTextEntityManager);
   int movieResults = checkForMovieIndex(fullTextEntityManager);
   LOG.info(String.format("DbMovies: %d, DbActors: %d, FtMovies: %d, 
      FtActors: %d", movieCount, actorCount, movieResults, actorResults));
   if (actorResults == 0 || movieResults == 0 
      || actorResults != actorCount || movieResults != movieCount) {
      fullTextEntityManager.createIndexer().startAndWait();
      this.indexDone = true;
      LOG.info("Hibernate Search Index ready.");
   } else {
      this.indexDone = true;
      LOG.info("Hibernate Search Index ready.");
   }
}

private int checkForMovieIndex(FullTextEntityManager fullTextEntityManager) {
   org.apache.lucene.search.Query movieQuery = fullTextEntityManager
      .getSearchFactory().buildQueryBuilder()				  
      .forEntity(Movie.class).get().all().createQuery();
   int movieResults = fullTextEntityManager.createFullTextQuery(movieQuery, 
      Movie.class).getResultSize();
   return movieResults;
}

@Async And this @EventListener(ApplicationReadyEvent.class) Execute Spring’s annotations checkHibernateSearchIndexes() method on application startup on its own background thread.

First, the number of Movie and Actor entities in the database are queried with named queries.

Second, the number of movie and actor entities in the Hibernate search index is queried with checkForMovieIndex(...) And checkForActorIndex(...) ways.

Then, the results are compared and the Hibernate search indices are rebuilt with FullTextEntityManager If difference is found. For normal startup time, the method must execute on its own background thread. Hibernate Search indexes files on the file system. They must be created or checked on first startup. By having a local index on each instance of the application, conflicts and inconsistencies are avoided.

Conclusion Backend

There are many optional criteria for queries, and JPA criteria queries support this use case. The code is verbose and some need to get used to it. The alternative is to either build the query string yourself or add a library (with possible code generation) for more support. This project tries to link only the required libraries and the code is maintainable enough. I haven’t found support for executing Hibernate Search and JPA Criteria queries simultaneously. Because of that, the results have to be added to the code. This requires a limit on the result size to protect the server’s I/O and memory resources, which can lead to missed matches in large result sets.

Hibernate search was easy to use and indices could be created/updated on application start up.

Angular Frontend

Movie/actor filters are represented in the lazy loaded Angular modules filter-actors and filter-movies. Modules are lazy loaded to make application startup faster and because they are the only users of the components of the ng-bootstrap library. The template filter-movies.component.html uses the offcanvas component and the datepicker component for the filter criteria:

<ng-template #content let-offcanvas>
  <div class="offcanvas-header">
    <h4 class="offcanvas-title" id="offcanvas-basic-title" 
       i18n="@@filtersAvailiable">Filters availiable</h4>
    <button type="button" class="btn-close" aria-label="Close" 
       (click)="offcanvas.dismiss('Cross click')"></button>
  </div>
  <div class="offcanvas-body">
    <form>
      <div class="select-range">
      <div class="mb-3 me-1">
      <label for="releasedAfter" i18n="@@filterMoviesReleasedAfter">
         Released after</label>
        <div class="input-group">
          <input id="releasedBefore"  class="form-control" 
             [(ngModel)]="ngbReleaseFrom" placeholder="yyyy-mm-dd" 
             name="dpFrom" ngbDatepicker #dpFrom="ngbDatepicker">
          <button class="btn btn-outline-secondary calendar" 
             (click)="dpFrom.toggle()" type="button"></button>
        </div>
      </div>
...
</ng-template>
<div class="container-fluid">
   <div>
      <div class="row">
         <div class="col">
            <button class="btn btn-primary filter-change-btn"
               (click)="showFilterActors()"  
               i18n="@@filterMoviesFilterActors">Filter<br/>Actors</button>
            <button class="btn btn-primary open-filters" 
               (click)="open(content)"  
               i18n="@@filterMoviesOpenFilters">Open<br/>Filters</button>
...

  •  <ng-template ...> The component has a template variable ‘#content’ to identify it.
  • <div class="offcanvas-header"> Includes label and its close button.
  • <div class="offcanvas-body"> Contains datepicker component with input for value and button to open datepicker. template variable #dpFrom="ngbDatepicker" Gets the datepicker object. The button uses this in the ‘click’ action to toggle the datepicker.
  • <div class="container-fluid"> Contains button and result table.
  • switch Filter<br/>Actors executes showFilterActors() Method to navigate to the filter-actors module.
  • switch Open<br/>Filters Executes the ‘open(content)’ method to open the component with Offcanvas <ng-template> Including the template variable ‘#content’.

filter-movies.component.ts shows the offcanvas component, calls filter, and shows the result:

{ this.closeResult = `close with: ${result}`; }, (reason) => { this.closeResult = `dismiss ${this.getDismissReason(reason)}`; , } public ShowFilterActors(): void { this.router.navigate([‘/filter-actors’], } private getDismissReason(reason: unknown): void { //console.log(this.filterCriteria); if (reason === OffcanvasDismissReasons.ESC) { return this. reset filter(); } else { this.filterCriteria.releaseFrom = !this.ngbReleaseFrom ? void: new Date(this.ngbReleaseFrom.year, this.ngbReleaseFrom.month, this.ngbReleaseFrom.day); this.filterCriteria.releaseTo = !this.ngbReleaseTo ? void: new Date(this.ngbReleaseTo.year, this.ngbReleaseTo.month, this.ngbReleaseTo.day); This.[“http://dzone.com/”], , } } }” data-lang = “application/typescript”>

@Component({
  selector: 'app-filter-movies',
  templateUrl: './filter-movies.component.html',
  styleUrls: ['./filter-movies.component.scss']
})
export class FilterMoviesComponent implements OnInit {
  protected filteredMovies: Movie[] = [];
  protected filtering = false;
  protected selectedGeneresStr="";
  protected generes: Genere[] = [];
  protected closeResult="";
  protected filterCriteria = new MovieFilterCriteria();
  protected ngbReleaseFrom: NgbDateStruct;
  protected ngbReleaseTo: NgbDateStruct;
  
  constructor(private offcanvasService: NgbOffcanvas, 
     public ngbRatingConfig: NgbRatingConfig, 
     private movieService: MoviesService, private router: Router) {}
  
...

  public open(content: unknown) {
    this.offcanvasService.open(content, 
       {ariaLabelledBy: 'offcanvas-basic-title'}).result.then((result) =>   
          { this.closeResult = `Closed with: ${result}`;
    }, (reason) => {
      this.closeResult = `Dismissed ${this.getDismissReason(reason)}`;
    });
  }

  public showFilterActors(): void {
     this.router.navigate(['/filter-actors']);
  }

  private getDismissReason(reason: unknown): void {
    //console.log(this.filterCriteria);
    if (reason === OffcanvasDismissReasons.ESC) {
      return this.resetFilters();
    } else {
       this.filterCriteria.releaseFrom = !this.ngbReleaseFrom ? null : 
          new Date(this.ngbReleaseFrom.year, this.ngbReleaseFrom.month, 
             this.ngbReleaseFrom.day);
       this.filterCriteria.releaseTo = !this.ngbReleaseTo ? null : 
	  new Date(this.ngbReleaseTo.year, this.ngbReleaseTo.month, 
             this.ngbReleaseTo.day);
       this.movieService.findMoviesByCriteria(this.filterCriteria)
          .subscribe({next: result => this.filteredMovies = result, 
             error: failed => {
	        console.log(failed);
	        this.router.navigate(["http://dzone.com/"]);
             }
       });
    }
  }
}

FilterMoviesComponent the constructor gets NgbOffcanvas, NgbRatingConfigMoviesService, Router injected.

The open method of ‘offCanvasService’ opens the OffCanvas component and returns a promise to return ‘CloseResult’.

showFilterActors(..) Navigates the path of the lazy loaded filter-actors module.

process getDismissReason(…) also checks for “to escape” Button that resets the filter. FilterCriteria contains dates from ngbDateStruct Datepicker objects and calls findMoviesByCriteria(…) Of ‘MovieService’. subscribe(…) The method stores the result in the ‘filteredMovies’ property.

Conclusion Frontend

The angular frontend required some effort and time until the ng-bootstrap components were simply lazily loaded into the modules filter-actors and filter-movies. This enables faster initial load times. The ng-bootstrap components were easy to use and worked fine. The front end is a UX challenge. For example, how to show the user a full-text search with the operators “and,” “or,” and “not”.

Leave a Comment