Skip to content

Commit 5988d3a

Browse files
authored
Update README.md
1 parent 199cdd3 commit 5988d3a

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ This repository contains the sparse attention primitives used in [Sparse Transfo
1010

1111
We hope this code can further accelerate research into sparse attention.
1212

13+
An example Transformer implementation which is close to the version we use internally can be found at https://github.com/openai/blocksparse/blob/master/examples/transformer/enwik8.py.
14+
1315
# Overview of kernels
1416
The repository contains fused implementations of the attention operation, which takes in `Q`, `K`, `V` matrices (all of dimensionality `batch, time, dim`) representing the queries, keys, and values for a sequence. For every query element, a weighted sum of the values is returned, where the weightings are determined by the scaled matrix product of `Q` and `K^T`.
1517

0 commit comments

Comments
 (0)