NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generation Models

Xiaokun Feng1,2,3, Haiming Yu3, Meiqi Wu3,4*, Shiyu Hu5, Jintao Chen3,6,
Chen Zhu3,, Jiahong Wu3, Xiangxiang Chu3, Kaiqi Huang1,2,
1 School of Artificial Intelligence, UCAS    2 CASIA    3 AMAP, Alibaba Group   
4 School of Computer Science and Technology, UCAS   
5 School of Physical and Mathematical Sciences, NTU    6 PKU   
Teaser.

Framework of our NarrLV. (a) Our prompt suite is inspired by film narrative theory and identifies three key factors influencing Temporal Narrative Atom (TNA) transitions. Based on these, we construct a prompt generation pipeline capable of producing evaluation prompts with flexibly adjustable TNA counts. (b) Our evaluation models include long video generation models and the foundation models they often rely on. (c) Based on the progressive expression of narrative content, we conduct evaluations from three dimensions, employing an MLLM-based question generation and answering framework for calculations. Our metric is well-aligned with human preferences.

Abstract

With the rapid development of foundation video generation technologies, long video generation models have exhibited promising research potential thanks to greater content creation space. Recent studies reveal that the goal of long video generation tasks is not only to extend video duration but also to accurately express richer narrative content within longer videos. However, due to the lack of evaluation benchmarks specifically designed for long video generation models, the current assessment of these models primarily relies on benchmarks with simple narrative prompts (\eg, VBench). To comprehensively assess the Narrative expression capabilities of Long Video generation models, we propose NarrLV- a novel benchmark inspired by film narrative theory. (i) First, we introduce the basic narrative unit maintaining continuous visual presentation in videos as Temporal Narrative Atom (TNA), and use its count to quantitatively measure narrative richness. Guided by three key film narrative elements influencing TNA changes, we construct an automatic prompt generation pipeline capable of producing evaluation prompts with a flexibly expandable number of TNAs. (ii) Then, based on the three progressive levels of narrative content expression, we design an effective evaluation metric using the MLLM-based question generation and answering framework. (iii) Finally, we conduct extensive evaluations on existing long video generation models and the foundation generation models that underpin them. Experimental results demonstrate that our metric aligns closely with human judgments. The derived evaluation outcomes reveal the detailed capability boundaries of current video generation models in narrative content expression.