forked from nerfies/nerfies.github.io
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
650 lines (586 loc) · 22.7 KB
/
index.html
File metadata and controls
650 lines (586 loc) · 22.7 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="description"
content="Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulation">
<meta name="keywords" content="Embodied AI, Mobile Manipulation">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>IALP
</title>
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-PYVRSFMDRL"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag() {
dataLayer.push(arguments);
}
gtag('js', new Date());
gtag('config', 'G-PYVRSFMDRL');
</script>
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet">
<link rel="stylesheet" href="./static/css/bulma.min.css">
<link rel="stylesheet" href="./static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="./static/css/bulma-slider.min.css">
<link rel="stylesheet" href="./static/css/fontawesome.all.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="./static/css/index.css">
<!-- <link rel="icon" href="./static/images/favicon.svg"> -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script defer src="./static/js/fontawesome.all.min.js"></script>
<script src="./static/js/bulma-carousel.min.js"></script>
<script src="./static/js/bulma-slider.min.js"></script>
<script src="./static/js/index.js"></script>
<style>
.tab-pane {
display: none;
}
.tab-pane.active {
display: block;
}
</style>
<script>
$(document).ready(function () {
$(".tabs ul li a").click(function (e) {
e.preventDefault();
// Get the target tab
var targetTab = $(this).attr("href");
// Remove 'is-active' class from all tab-items
$(".tabs ul li").removeClass("is-active");
// Add 'is-active' class to current tab-item
$(this).parent().addClass("is-active");
// Hide all tab content
$(".tab-content .tab-pane").removeClass("active");
// Show the target tab content
$(targetTab).addClass("active");
});
});
</script>
</head>
<body>
<section class="hero">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-1 publication-title">Instruction-Augmented Long-Horizon Planning: Embedding Grounding
Mechanisms in Embodied Mobile Manipulation</h1>
<div class="is-size-5 publication-authors">
<span class="author-block">
<a href="https://orcid.org/0000-0002-7492-632X">Fangyuan Wang</a><sup>1,2</sup>,</span>
<span class="author-block">
<a href="https://">Shipeng Lyu</a><sup>1,2</sup>,</span>
<span class="author-block">
<a href="https://orcid.org/0000-0002-7020-0943">Peng Zhou</a><sup>3</sup>,
</span>
<span class="author-block">
<a href="https://orcid.org/0000-0002-9666-018X">Anqing Duan</a><sup>4</sup>,
</span>
<span class="author-block">
<a href="https://orcid.org/0000-0001-9583-0055">Guodong Guo</a><sup>2</sup>,
</span>
<span class="author-block">
<a href="https://orcid.org/0000-0002-3426-6638">David Navarro-Alarcon</a><sup>1</sup>,
</span>
</div>
<div class="is-size-5 publication-authors">
<span class="author-block"><sup>1</sup>Department of Mechanical Engineering, The Hong Kong Polytechnic
University</span></br>
<span class="author-block"><sup>2</sup>Ningbo Institute of Digital Twin, Eastern Institute of
Technology</span>
<span class="author-block"><sup>3</sup>School of Advanced Engineering, Great Bay University</span>
<span class="author-block"><sup>4</sup>Robotics Department, Mohamed Bin Zayed University of Artificial
Intelligence</span>
</div>
<div class="column has-text-centered">
<div class="publication-links">
<!-- PDF Link. -->
<!-- <span class="link-block">
<a href="https://arxiv.org/pdf/2011.12948"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-file-pdf"></i>
</span>
<span>Paper</span>
</a>
</span> -->
<span class="link-block">
<a href="#" class="external-link button is-normal is-rounded is-dark" title="Arxiv will come soon">
<span class="icon">
<i class="ai ai-arxiv"></i>
</span>
<span>arXiv</span>
</a>
</span>
<!-- Video Link. -->
<span class="link-block">
<a href="#" class="external-link button is-normal is-rounded is-dark" title="Video will come soon">
<span class="icon">
<i class="fab fa-youtube"></i>
</span>
<span>Video</span>
</a>
</span>
<!-- Code Link. -->
<span class="link-block">
<a href="#" class="external-link button is-normal is-rounded is-dark" title="Code will come soon">
<span class="icon">
<i class="fab fa-github"></i>
</span>
<span>Code</span>
</a>
</span>
<!-- Dataset Link. -->
<!-- <span class="link-block">
<a href="https://github.com/google/nerfies/releases/tag/0.1"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="far fa-images"></i>
</span>
<span>Data</span>
</a> -->
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<!-- <section class="hero teaser">
<div class="container is-max-desktop">
<div class="hero-body">
<video id="teaser" autoplay muted loop playsinline height="100%">
<source src="./static/videos/teaser.mp4"
type="video/mp4">
</video>
<h2 class="subtitle has-text-centered">
<span class="dnerf">Nerfies</span> turns selfie videos from your phone into
free-viewpoint
portraits.
</h2>
</div>
</div>
</section> -->
<section class="hero is-light is-small">
<div class="hero-body">
<div class="container">
<div id="results-carousel" class="carousel results-carousel">
<div class="item item-steve">
<video poster="" id="steve" autoplay controls muted loop playsinline height="100%">
<source src="./static/videos/pick.mp4" type="video/mp4">
</video>
</div>
<div class="item item-chair-tp">
<video poster="" id="chair-tp" autoplay controls muted loop playsinline height="100%">
<source src="./static/videos/cloth.mp4" type="video/mp4">
</video>
</div>
<div class="item item-shiba">
<video poster="" id="shiba" autoplay controls muted loop playsinline height="100%">
<source src="./static/videos/pill.mp4" type="video/mp4">
</video>
</div>
<div class="item item-fullbody">
<video poster="" id="fullbody" autoplay controls muted loop playsinline height="100%">
<source src="./static/videos/book.mp4" type="video/mp4">
</video>
</div>
<div class="item item-blueshirt">
<video poster="" id="blueshirt" autoplay controls muted loop playsinline height="100%">
<source src="./static/videos/bucket.mp4" type="video/mp4">
</video>
</div>
</div>
</div>
</div>
</section>
<section class="hero is-light is-small">
<div class="hero-body">
<div class="container">
<div id="results-carousel2" class="carousel results-carousel2" data-slides-to-show="1">
<div class="item item-steve">
<img class="carousel-image" src="./static/images/plans-box.png" alt="pick">
</div>
<div class="item item-chair-tp">
<img class="carousel-image" src="./static/images/plans-cloth.png" alt="cloth">
</div>
<div class="item item-shiba">
<img class="carousel-image" src="./static/images/plans-drawer.png" alt="drawer">
</div>
<div class="item item-fullbody">
<img class="carousel-image" src="./static/images/plans-book.png" alt="book">
</div>
<div class="item item-blueshirt">
<img class="carousel-image" src="./static/images/plans-bucket.png" alt="bucket">
</div>
</div>
</div>
</div>
</section>
<section class="section">
<div class="container is-max-desktop">
<!-- Abstract. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
Enabling humanoid robots to perform long-horizon mobile manipulation planning in real-world environments
based on embodied perception and comprehension abilities has been a longstanding challenge.
</p>
<p>
With the recent rise of large language models (LLMs), there has been a notable increase in the development
of LLM-based planners. These approaches either utilize human-provided textual representations of the real
world or heavily depend on prompt engineering to extract such representations, lacking the capability to
quantitatively understand the environment, such as determining the feasibility of manipulating objects.
</p>
<p>
To address these limitations, we present the Instruction-Augmented Long-Horizon Planning (IALP) system, a
novel framework that employs LLMs to generate feasible and optimal actions based on real-time sensor
feedback, including grounded knowledge of the environment, in a closed-loop interaction. Distinct from
prior works, our approach augments user instructions into PDDL problems by leveraging both the abstract
reasoning capabilities of LLMs and grounding mechanisms.
By conducting various real-world long-horizon tasks, each consisting of seven distinct manipulatory
skills, our results demonstrate that the IALP system can efficiently solve these tasks with an average
success rate exceeding 80%.
Our proposed method can operate as a high-level planner, equipping robots with substantial autonomy in
unstructured environments through the utilization of multi-modal sensor inputs.
</p>
</div>
</div>
</div>
<!--/ Abstract. -->
<!-- Paper video. -->
<!-- <div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Video</h2>
<div class="publication-video">
<iframe src="https://www.youtube.com/embed/MrKrnHhk8IA?rel=0&showinfo=0"
frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
</div>
</div>
</div>
</div> -->
</section>
<section class="section">
<div class="container is-max-widescreen">
<div class="column is-full-width">
<h2 class="title is-3">Methodology</h2>
<img src="./static/images/overview.png" alt="overview" style="width: 100%;">
</div>
</div>
</section>
<!--Classical Task Planner-->
<section class="section">
<div class="container is-max-widescreen">
<div class="rows">
<div class="rows is-centered ">
<div class="row is-full-width">
<h2 class="title is-3"><span class="dvima">Prompts used for LLM planner.</span></h2>
<span style="font-size: 110%">
Below lists comprehensive experimental details, including the design of actions, predicates, and the
domain as described by PDDL. We also lists all the prompts utilized in our experiments, consisting of
those for checking promptable predicates, generating PDDL problems, and serving as the LLM planner.
<!-- Inserted Tabs Start -->
<div class="tabs">
<ul>
<li class="is-active"><a href="#pddl1">PDDL Action Definition</a></li>
<li><a href="#pddl2">PDDL Domain Description</a></li>
<li><a href="#pddl3">Prompts to generate goal</a></li>
<li><a href="#pddl4">Prompts to generate observation</a></li>
<li><a href="#pddl5">Prompt for planning</a></li>
</ul>
</div>
<div class="tab-content">
<div class="tab-pane active" id="pddl1">
<pre><code class="scrollable-content">
(:action move ; move the robot to the object
:parameters (?obj - locatable)
:precondition (and
(find ?obj)
)
:effect (and
(at ?obj)
)
)
(:action scan ; scan the object
:parameters (?obj - locatable)
:precondition (and
(at ?obj)
(not (detected ?obj))
)
:effect (and
(detected ?obj)
)
)
(:action grasp ; grasp the object
:parameters (?obj - locatable)
:precondition (and
(at ?obj)
(detected ?obj)
(graspable ?obj)
(reachable ?obj)
)
:effect (and
(holding ?obj)
)
)
(:action place ; place the object on the place
:parameters (?obj - locatable ?pla - locatable)
:precondition (and
(at ?place)
(holding ?obj)
(placeable ?pla)
)
:effect (and
(on ?obj ?pla)
)
)
(:action pull ; pull the object
:parameters (?obj - locatable)
:precondition (and
(at ?obj)
(graspable ?obj)
(not (opened ?obj))
)
:effect (and
(opened ?obj)
)
)
(:action push ; push the object
:parameters (?obj - locatable)
:precondition (and
(at ?obj)
(graspable ?obj)
(opened ?obj)
)
:effect (and
(not (opened ?obj))
)
)
(:action lift ; lift the object
:parameters (?obj - locatable)
:precondition (and
(at ?obj)
(holding ?obj)
)
:effect (and
)
)
(:action rotate ; rotate the object
:parameters (?obj - locatable)
:precondition (and
(at ?obj)
(holding ?obj)
)
:effect (and
)
)
(:action reach ; reach the pose
:parameters (?pose - locatable)
:precondition (and
(reachable ?pose)
)
:effect (and
)
)
(:action adjust ; adjust the robot if the object is not reachable, graspable, or placeable
:parameters (?obj - locatable)
:precondition (or
(not (reachable ?obj))
(not (graspable ?obj))
(not (placeable ?obj))
)
:effect (and
)
)
(:action alert ; alert when there's an error the robot cannot solve based on the current state
:parameters ()
:precondition (and
)
:effect (and
)
)
(:action stop ; stop the robot when the taks is done
:parameters ()
:precondition (and
)
:effect (and
)
)
</code></pre>
</div>
<div class="tab-pane" id="pddl2">
<pre><code>
(define (domain room)
(:requirements :strips :fluents :durative-actions :timed-initial-literals :typing :conditional-effects :negative-preconditions :duration-inequalities :equality :disjunctive-preconditions)
(:types
locatable - object
floor - object
bot table coat_stand cloth box - locatable
robot - locatable
pose - locatable
)
(:predicates
(:predicates ;todo: define predicates here
(on ?obj1 - locatable ?obj2 - locatable) ; if the object is on another object
(in ?obj - locatable ?obj2 - locatable); if the object1 is inside others
(holding ?obj - locatable) ; if the arm is holding the object
(opened ?obj) ; if the object is opened
(at ?obj - locatable) ; if the robot is at the object (near)
(find ?obj - locatable); if has a path from the current state to the object
(detected ?obj - locatable); if the object is detected
(graspable ?obj - locatable); if the object is graspable
(reachable ?obj - locatable); if the object is reachable
(placeable ?obj - locatable); if the object is placeable
)
)
(:functions
)
; Actions
)
</code></pre>
</div>
<div class="tab-pane" id="pddl3">
<pre><code>
Based on the instruction, you are using the following predicates to generate the goal of the PDDL problem. The robot's name is Tiago.
Instruction: [INSTRUCTION]
The predicate candidates are:
"""
**Check predicate before**
"""
You can use (and ) and (or ) to combine the goal predicates. Please only return answers without any explanation. Do not return markdown code wrappers.
Here's an example:
[user]
Instruction: Pick up the box on the table and place it on the black table.
[assistant]
(on box black_table)
Example finished.
Here's what I give to you:
Instruction: [INSTRUCTION]
</code></pre>
</div>
<div class="tab-pane" id="pddl4">
<strong>Extract task-related objects in current task</strong>
<pre><code>
Please extract the object name and related object name from the given instruction. The related object name will help humans to find the object. Also, extract other object names that may be related to finishing the instruction, such as the target position-related objects. You need to concatenate multi-word object names by using "_." For example, the "black table" in the instruction should be converted to "black_table." The answer should be in JSON format without markdown code block triple backticks:
{
"object_name": "str",
"related_object_name": "str",
"other_object_names": [
"str",
"str"
]
}
The instruction: [INSTRUCTION]
</code></pre>
<strong>Listing all possible predicates that are necessary to check the current state.</strong>
<code><pre>
You are required to detect the observation of the current environment by using the predicates and related objects provided below. You need to give all predicates and objects necessary to check the current state so that the robot can choose the best action. You don't need to verify each predicate. Every predicate can be used multiple times. Do NOT return markdown code block triple backticks.
Instruction: [INSTRUCTION]
The possible ?obj could be: [OBJECTS]
The predicate candidates are:
"""
predicates ...
"""
The possible actions are:
"""
actions ...
"""
The output is formatted as:
"""
(on box table)
(holding robot box)
...
"""
</code></pre>
</div>
<div class="tab-pane" id="pddl5">
<pre><code>
{System Role}
[user]
You are an excellent interpreter of human instructions for daily tasks. Given an instruction and information about the working environment, you break it down into a sequence of robotic actions. Please do not begin working until I say "Start working." Instead, simply output the message "Waiting for next input." Understood?
[assistant]
Understood. Waiting for the next input.
\end{lstlisting}
{Environments}
[user]
Information about environments, objects, and tasks is given as a PDDL function.
The Planning Domain Definition Language (PDDL) is a domain-specific language designed for the Benchmark for creating a standard for Artificial Intelligence (AI) planning.
Here's the domain description you used:
"""
**Check domain description**
"""
The =:action= blocks define all the action/subtasks used for completing the task.
Later, you will receive the task/problem defined by PDDL and the above domain.
Here's an example:
"""
...
"""
The =:init= block defines the current observation of the environment.
The =:goal= block defines the goal of the task.
You need to take action from the current state, not from the start. If the current task is over and no action is needed for the robot, you can use the "stop" action. If the robot doesn't know what to execute in its current state, for example, it cannot find the target object, you can use the "alert" action and stop the robot. If the object is not reachable, graspable, or placeable after the scan, you can use the "adjust" action to adjust the robot's pose to make it reachable, graspable, or placeable.
-------------------------------------------------------
The texts above are part of the overall instruction. Do not start working yet:
[assistant]
Understood. Waiting for the next input.
\end{lstlisting}
{Observation input}
[user]
Start working. Resume from the environment below.
The instruction is as follows:
"""
[INSTRUCTION]
"""
The action executed last time is as follows:
"""
[ACTION]
"""
The observation of the current environment is as follows:
"""
[OBSERVATION]
"""
</code></pre>
</div>
<!-- Inserted Tabs End -->
</div>
</div>
</div>
</div>
</div>
</section>
<section class="section" id="BibTeX">
<div class="container is-max-desktop content">
<h2 class="title">BibTeX</h2>
<pre><code>@article{fangyuan2024instruction,
author = {Fangyuan Wang, Shipeng Lyu, Peng Zhou, Anqing Duan, Guodong Guo, David Navarro-Alarcon},
title = {Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulation},
journal = {AAAI},
year = {2024},
}</code></pre>
</div>
</section>
<footer class="footer">
<div class="container">
<div class="content has-text-centered">
<a class="icon-link" href="#">
<i class="fas fa-file-pdf"></i>
</a>
<a class="icon-link" href="#" class="external-link" disabled>
<i class="fab fa-github"></i>
</a>
</div>
<div class="columns is-centered">
<div class="column is-8 has-text-centered">
<div class="content">
<p>
This website is borrowed from <a href="https://nerfies.github.io">Nerfies</a> and <a
href="https://cowplanning.github.io/">Cowplanning</a>.
</p>
</div>
</div>
</div>
</div>
</footer>
</body>
</html>