NarraBench

Benchmarks for Testing Narrative Understanding

NarraBench is an evolving collection of benchmarks useful for testing LLM narrative understanding abilities. These benchmarks are collected according to a theory-informed taxonomy of narrative-understanding tasks:

Table of Contents

• Benchmarks
  • Story
  • Narration
  • Discourse
  • Situatedness
  • Run
  • Submission

Benchmarks

Story

Agent

Aspect	Question	Properties	Benchmarks
name	who are the characters in the text?	local discrete deterministic	-
name	who are the main characters in the text?	global holistic consensus	-
role	what is the character's role in the text?	global holistic perspectival	-
attributes	what attributes does this character have?	local discrete deterministic	-
attributes	what attributes does this character have?	global holistic consensus	AustenAlike
emotional state	what is the character feeling right now?	local discrete perspectival	-
emotional state	what are the central emotional states the character has experienced?	global holistic perspectival	CULEMO
motivation	why is the character doing what they are doing right now?	local discrete perspectival	-
motivation	what have been the character's core motivations behind their actions? What motivates this character?	global progressive perspectival	-

Social Networks

Aspect	Question	Properties	Benchmarks
interaction type	how are these two characters interacting?	local discrete deterministic	-
connections	who does the character know?	global holistic deterministic	PhantomWiki
relationship type	what is the relationship type between these two characters?	global holistic consensus	-

Event

Aspect	Question	Properties	Benchmarks
event	what is happening?	local discrete deterministic	-
event	what happened?	global discrete consensus	-
schema	what is the narrative schema of these events?	global holistic consensus	-
causality	what caused this event to happen?	global progressive perspectival	-

Plot

Aspect	Question	Properties	Benchmarks
topic	what is/are the topic(s) of this story?	global holistic consensus	-
plot	what is the plot summary?	global holistic perspectival	-
plotline	what happened in this plotline?	global holistic consensus	StorySumm
moral	what is the moral of the story?	global holistic perspectival	-
obstacle	what is the central negative force of the story?	global holistic perspectival	-
conflict	what is the central conflict of this story?	global holistic perspectival	-
archetype	what is the narrative archetype of this story?	global holistic consensus	-

Structure

Aspect	Question	Properties	Benchmarks
plot arc	what is the plot arc structure used?	global progressive consensus	-

Setting

Aspect	Question	Properties	Benchmarks
setting	what is the setting?	local discrete deterministic	-
setting	what is the setting?	global holistic consensus	-
location	where is this taking place?	local discrete deterministic	-
location	what locations has the story taken place in?	global discrete deterministic	-

Discourse

Time

Aspect	Question	Properties	Benchmarks
duration	how much time is passing?	local discrete deterministic	-
duration	how much time has passed since the previous scene?	global progressive deterministic	-
duration	how much time has passed?	global holistic deterministic	-
order	does this scene come before the prior scene, come after, or indicate a future that has not yet happened?	global progressive deterministic	TRaVelER, ToT
order	does this story tell events out of order?	global holistic deterministic	TRAM

---

Revelation

Aspect	Question	Properties	Benchmarks
suspense	is key information that is already known being withheld or deferred?	global progressive perspectival	-
curiosity	are there key causal antecedents to the current events that are being withheld?	global progressive perspectival	-
surprise	is key information suddenly revealed?	global progressive perspectival	-

Narration

Perspective

Aspect	Question	Properties	Benchmarks
point of view	who is telling?	global discrete deterministic	-
focalization	who knows?	local discrete deterministic	-
dialogue	who speaks?	local discrete deterministic	-

Style

Aspect	Question	Properties	Benchmarks
allusion	what other texts is this passage alluding to?	local discrete perspectival	-
figurative language	is this sentence using figurative language?	local discrete perspectival	-
imageability	how well can you imagine this scene?	local holistic perspectival	-
complexity	how complex is the sentence structure?	local holistic perspectival	-
evaluative language	is this sentence engaging in evaluative discourse?	local discrete perspectival	-

Situatedness

Paratext

Aspect	Question	Properties	Benchmarks
genre	what is the genre?	global holistic consensus	-
author	who is/are the author(s)?	global discrete deterministic	-
date	what is the date of creation / publication?	global discrete deterministic	-
medium	what medium does the story appear in? (book, podcast, image, etc.)	global discrete deterministic	-
platform	what platform does it appear through? (website, publishing house, etc.)	global discrete deterministic	-

Motivation

Aspect	Question	Properties	Benchmarks
intent	what is the author's intent in telling this story?	global holistic perspectival	-

Run

1. Install Prolog and enable git lfs
2. Install dependencies: pip install -e .
3. Run setup.py to pull all benchmarks
4. Start Ollama server on port 11434
5. Start judge model server on port 11435 (we use gpt-oss:20b to normalize results)
6. Run run.py to run all models.

Submission

To submit a new benchmark to NarraBench, please raise a PR with this template:

Name: <fill>
Link: <fill>
License: <fill>
Dimension: <fill>
Feature: <fill>
Aspect: <fill>
Scale: <fill>
Mode: <fill>
Variance: <fill>

Citation: 

```

Rationale: <1-2 sentences describing what this benchmark provides.>