Manling Li

Multimedia Event Knowledge Acquisition: From Object-Centric to Event-Centric Understanding

Human memories can be viewed as repositories of historical events. An event is a semantic structure encapsulating the fundamental questions of Who? What? Where? When? Why?, which are of primary concern to humans. The enormous volume of data requires machines to automatically obtain such factual knowledge including events and their arguments (participants) and perform reasoning by synthesizing a wide variety of unstructured data, such as text, images, and videos. However, current event understanding is text-only, local, and lacking in reasoning. 
This talk focuses on constructing event graphs to deal with real-world events that are multimedia, interconnected, probabilistic, and span a long time period. We discover the local event structures (i.e., who, what, where, and when) by transferring the semantic understanding ability from text to vision during vision-language pretraining. To perform global reasoning across events (i.e., what is likely to occur, and why), we further investigate the interactions between events such as the temporal orders and the evolution patterns along a long period, and leverage historical events to discover such event schema knowledge globally. Our structural event graph modeling is able to represent the global inter-dependencies of events and long-distance interactions via arguments, leading to a comprehensive understanding of events and effective forecasting of future events.