A Probabilistic Framework for Multimodal Retrieval using Integrative Indian Buffet Process

Bahadir Ozdemir,  Larry S. Davis


iIBP is an unsupervised integrative multimodal retrieval framework to discover abstract features and finds most relevant images to a given text or image query.

Paper   Supplemental   Code (MATLAB)

Figure 1. Schematic overview of the iIBP algorithm. The flow chart illustrates discovery of abstract features from multimodal data, the retrieval system for cross-view queries and user relevance feedback.

 

Figure 2. Latent abstract feature model. Visual data is a product of Z and Av with some noise; and similarly the textual data is a product of Z and At with some noise.

 

Figure 3. Graphical model for the integrative IBP approach. Circles indicate random variables, shaded circles denote observed values, and the blue square boxes are hyperparameters.

 

Figure 4. Graphical model for the feedback query model. Circles indicate random variables, shaded circles denote observed values. Hyperparameters are omitted for clarity. Note that Z is considered as an observed variable in the retrieval part.