Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Address comments
  • Loading branch information
mattdangerw committed Sep 19, 2024
commit 96f900f098ca0f6d16a65161233ac55393ec9af5
30 changes: 15 additions & 15 deletions CONTRIBUTING_MODELS.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ open to contributions!
In this guide, we will walk you through the steps one needs to take in order to
contribute a new pre-trained model to KerasHub. For illustration purposes, let's
assume that you want to contribute the DistilBERT model. Before we dive in, we encourage you to go through
[our getting started guide](https://keras.io/guides/keras_hub/getting_started/)
[our getting started guide](https://keras.io/guides/keras_nlp/getting_started/)
for an introduction to the library, and our
[contribution guide](https://github.com/keras-team/keras-nlp/blob/master/CONTRIBUTING.md).

Expand All @@ -22,29 +22,29 @@ Keep this checklist handy!

### Step 2: PR #1 - Add XXBackbone

- [ ] An `xx/xx_backbone.py` file which has the model graph \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/distil_bert/distil_bert_backbone.py)\].
- [ ] An `xx/xx_backbone_test.py` file which has unit tests for the backbone \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/distil_bert/distil_bert_backbone_test.py)\].
- [ ] An `xx/xx_backbone.py` file which has the model graph \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_backbone.py)\].
- [ ] An `xx/xx_backbone_test.py` file which has unit tests for the backbone \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_backbone_test.py)\].
- [ ] A Colab notebook link in the PR description which matches the outputs of the implemented backbone model with the original source \[[Example](https://colab.research.google.com/drive/1SeZWJorKWmwWJax8ORSdxKrxE25BfhHa?usp=sharing)\].

### Step 3: PR #2 - Add XXTokenizer

- [ ] An `xx/xx_tokenizer.py` file which has the tokenizer for the model \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/distil_bert/distil_bert_tokenizer.py)\].
- [ ] An `xx/xx_tokenizer_test.py` file which has unit tests for the model tokenizer \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/distil_bert/distil_bert_tokenizer_test.py)\].
- [ ] An `xx/xx_tokenizer.py` file which has the tokenizer for the model \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_tokenizer.py)\].
- [ ] An `xx/xx_tokenizer_test.py` file which has unit tests for the model tokenizer \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_tokenizer_test.py)\].
- [ ] A Colab notebook link in the PR description, demonstrating that the output of the tokenizer matches the original tokenizer \[[Example](https://colab.research.google.com/drive/1MH_rpuFB1Nz_NkKIAvVtVae2HFLjXZDA?usp=sharing)].

### Step 4: PR #3 - Add XX Presets

- [ ] An `xx/xx_presets.py` file with links to weights uploaded to a personal GCP bucket/Google Drive \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/distil_bert/distil_bert_presets.py)\].
- [ ] An `xx/xx_presets.py` file with links to weights uploaded to a personal GCP bucket/Google Drive \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_presets.py)\].
- [ ] A `tools/checkpoint_conversion/convert_xx_checkpoints.py` which is reusable script for converting checkpoints \[[Example](https://github.com/keras-team/keras-nlp/blob/master/tools/checkpoint_conversion/convert_distilbert_checkpoints.py)\].
- [ ] A Colab notebook link in the PR description, showing an end-to-end task such as text classification, etc. The task model can be built using the backbone model, with the task head on top \[[Example](https://gist.github.com/mattdangerw/bf0ca07fb66b6738150c8b56ee5bab4e)\].

### Step 5: PR #4 and Beyond - Add XX Tasks and Preprocessors

This PR is optional.

- [ ] An `xx/xx_<task>.py` file for adding a task model like classifier, masked LM, etc. \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/distil_bert/distil_bert_classifier.py)\]
- [ ] An `xx/xx_<task>_preprocessor.py` file which has the preprocessor and can be used to get inputs suitable for the task model \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/distil_bert/distil_bert_preprocessor.py)\].
- [ ] `xx/xx_<task>_test.py` file and `xx/xx_<task>_preprocessor_test.py` files which have unit tests for the above two modules \[[Example 1](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/distil_bert/distil_bert_classifier_test.py) and [Example 2](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/distil_bert/distil_bert_preprocessor_test.py)\].
- [ ] An `xx/xx_<task>.py` file for adding a task model like classifier, masked LM, etc. \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_classifier.py)\]
- [ ] An `xx/xx_<task>_preprocessor.py` file which has the preprocessor and can be used to get inputs suitable for the task model \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_preprocessor.py)\].
- [ ] `xx/xx_<task>_test.py` file and `xx/xx_<task>_preprocessor_test.py` files which have unit tests for the above two modules \[[Example 1](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_classifier_test.py) and [Example 2](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_preprocessor_test.py)\].
- [ ] A Colab notebook link in the PR description, demonstrating that the output of the preprocessor matches the output of the original preprocessor \[[Example](https://colab.research.google.com/drive/1GFFC7Y1I_2PtYlWDToqKvzYhHWv1b3nC?usp=sharing)].

## Detailed Instructions
Expand Down Expand Up @@ -81,7 +81,7 @@ around by a class to implement our models.

A model is typically split into three/four sections. We would recommend you to
compare this side-by-side with the
[`keras_hub.layers.DistilBertBackbone` source code](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/distil_bert/distil_bert_backbone.py)!
[`keras_hub.layers.DistilBertBackbone` source code](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_backbone.py)!

**Inputs to the model**

Expand Down Expand Up @@ -110,7 +110,7 @@ Standard layers used: `keras_hub.layers.TransformerDecoder`.

The standard layers provided in Keras and KerasHub are generally enough for
most of the usecases and it is recommended to do a thorough search
[here](https://keras.io/api/layers/) and [here](https://keras.io/api/keras_hub/layers/).
[here](https://keras.io/api/layers/) and [here](https://keras.io/api/keras_nlp/layers/).
However, sometimes, models have small tweaks/paradigm changes in their architecture.
This is when things might slightly get complicated.

Expand Down Expand Up @@ -165,12 +165,12 @@ All the underlying actual tokenization will be taken care of by the superclass.
The important thing here is adding "special tokens". Most models have
special tokens such as beginning-of-sequence token, end-of-sequence token,
mask token, pad token, etc. These have to be
[added as member attributes](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/distil_bert/distil_bert_tokenizer.py#L91-L105)
[added as member attributes](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_tokenizer.py#L91-L105)
to the tokenizer class. These member attributes are then accessed by the
preprocessor layers.

For a full list of the tokenizers KerasHub offers, please visit
[this link](https://keras.io/api/keras_hub/tokenizers/) and make use of the
[this link](https://keras.io/api/keras_nlp/tokenizers/) and make use of the
tokenizer your model uses!

#### Unit Tests
Expand All @@ -193,7 +193,7 @@ files. These files will then be uploaded to GCP by us!
After wrapping up the preset configuration file, you need to
add the `from_preset` function to all three classes, i.e., `DistilBertBackbone`,
and `DistilBertTokenizer`. Here is an
[example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/distil_bert/distil_bert_backbone.py#L187-L189).
[example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_backbone.py#L187-L189).

The testing for presets is divided into two: "large" and "extra large".
For "large" tests, we pick the smallest preset (in terms of number of parameters)
Expand Down Expand Up @@ -230,7 +230,7 @@ The preprocessor class might have a few intricacies depending on the model. For
the DeBERTaV3 tokenizer does not have the `[MASK]` in the provided sentencepiece
proto file, and we had to make some modifications [here](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/deberta_v3/deberta_v3_preprocessor.py). Secondly, we have
a separate preprocessor class for every task. This is because different tasks
might require different input formats. For instance, we have a [separate preprocessor](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/distil_bert/distil_bert_masked_lm_preprocessor.py)
might require different input formats. For instance, we have a [separate preprocessor](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_masked_lm_preprocessor.py)
for masked language modeling (MLM) for DistilBERT.

## Conclusion
Expand Down
51 changes: 27 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,30 @@
# KerasHub: Multi-framework NLP Models
# KerasNLP: Multi-framework NLP Models
[![](https://github.com/keras-team/keras-nlp/workflows/Tests/badge.svg?branch=master)](https://github.com/keras-team/keras-nlp/actions?query=workflow%3ATests+branch%3Amaster)
![Python](https://img.shields.io/badge/python-v3.9.0+-success.svg)
[![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/keras-team/keras-nlp/issues)

> [!IMPORTANT]
> KerasHub is becoming KerasHub! Read the announcement [here](https://github.com/keras-team/keras-nlp/issues/1831).
> KerasNLP is becoming KerasHub! Read the announcement
> [in this issue](https://github.com/keras-team/keras-nlp/issues/1831). We have
> renamed our repo to KerasHub in preparation for the release, but have not yet
> released the new package. Follow the issue for news.

KerasHub is a natural language processing library that works natively
with TensorFlow, JAX, or PyTorch. KerasHub provides a repository of pre-trained
KerasNLP is a natural language processing library that works natively
with TensorFlow, JAX, or PyTorch. KerasNLP provides a repository of pre-trained
models and a collection of lower-level building blocks for language modeling.
Built on Keras 3, models can be trained and serialized in any framework
and re-used in another without costly migrations.

This library is an extension of the core Keras API; all high-level modules are
Layers and Models that receive that same level of polish as core Keras.
If you are familiar with Keras, congratulations! You already understand most of
KerasHub.
KerasNLP.

All models support JAX, TensorFlow, and PyTorch from a single model
definition and can be fine-tuned on GPUs and TPUs out of the box. Models can
be trained on individual accelerators with built-in PEFT techniques, or
fine-tuned at scale with model and data parallel training. See our
[Getting Started guide](https://keras.io/guides/keras_hub/getting_started)
[Getting Started guide](https://keras.io/guides/keras_nlp/getting_started)
to start learning our API. Browse our models on
[Kaggle](https://www.kaggle.com/organizations/keras/models).
We welcome contributions.
Expand All @@ -30,9 +33,9 @@ We welcome contributions.

### For everyone

- [Home Page](https://keras.io/keras_hub)
- [Developer Guides](https://keras.io/guides/keras_hub)
- [API Reference](https://keras.io/api/keras_hub)
- [Home Page](https://keras.io/keras_nlp)
- [Developer Guides](https://keras.io/guides/keras_nlp)
- [API Reference](https://keras.io/api/keras_nlp)
- [Pre-trained Models](https://www.kaggle.com/organizations/keras/models)

### For contributors
Expand All @@ -51,7 +54,7 @@ Fine-tune BERT on IMDb movie reviews:
import os
os.environ["KERAS_BACKEND"] = "jax" # Or "tensorflow" or "torch"!

import keras_hub
import keras_nlp
import tensorflow_datasets as tfds

imdb_train, imdb_test = tfds.load(
Expand All @@ -61,7 +64,7 @@ imdb_train, imdb_test = tfds.load(
batch_size=16,
)
# Load a BERT model.
classifier = keras_hub.models.Classifier.from_preset(
classifier = keras_nlp.models.Classifier.from_preset(
"bert_base_en",
num_classes=2,
activation="softmax",
Expand All @@ -74,38 +77,38 @@ classifier.predict(["What an amazing movie!", "A total waste of my time."])

Try it out [in a colab](https://colab.research.google.com/gist/mattdangerw/e457e42d5ea827110c8d5cb4eb9d9a07/kerasnlp-quickstart.ipynb).
For more in depth guides and examples, visit
[keras.io/keras_hub](https://keras.io/keras_hub/).
[keras.io/keras_nlp](https://keras.io/keras_nlp/).

## Installation

To install the latest KerasHub release with Keras 3, simply run:
To install the latest KerasNLP release with Keras 3, simply run:

```
pip install --upgrade keras-hub
pip install --upgrade keras-nlp
```

To install the latest nightly changes for both KerasHub and Keras, you can use
To install the latest nightly changes for both KerasNLP and Keras, you can use
our nightly package.

```
pip install --upgrade keras-hub-nightly
pip install --upgrade keras-nlp-nightly
```

Note that currently, installing KerasHub will always pull in TensorFlow for use
Note that currently, installing KerasNLP will always pull in TensorFlow for use
of the `tf.data` API for preprocessing. Even when pre-processing with `tf.data`,
training can still happen on any backend.

Read [Getting started with Keras](https://keras.io/getting_started/) for more
information on installing Keras 3 and compatibility with different frameworks.

> [!IMPORTANT]
> We recommend using KerasHub with TensorFlow 2.16 or later, as TF 2.16 packages
> We recommend using KerasNLP with TensorFlow 2.16 or later, as TF 2.16 packages
> Keras 3 by default.

## Configuring your backend

If you have Keras 3 installed in your environment (see installation above),
you can use KerasHub with any of JAX, TensorFlow and PyTorch. To do so, set the
you can use KerasNLP with any of JAX, TensorFlow and PyTorch. To do so, set the
`KERAS_BACKEND` environment variable. For example:

```shell
Expand All @@ -118,7 +121,7 @@ Or in Colab, with:
import os
os.environ["KERAS_BACKEND"] = "jax"

import keras_hub
import keras_nlp
```

> [!IMPORTANT]
Expand All @@ -134,21 +137,21 @@ may break compatibility at any time and APIs should not be consider stable.

## Disclaimer

KerasHub provides access to pre-trained models via the `keras_hub.models` API.
KerasNLP provides access to pre-trained models via the `keras_nlp.models` API.
These pre-trained models are provided on an "as is" basis, without warranties
or conditions of any kind. The following underlying models are provided by third
parties, and subject to separate licenses:
BART, BLOOM, DeBERTa, DistilBERT, GPT-2, Llama, Mistral, OPT, RoBERTa, Whisper,
and XLM-RoBERTa.

## Citing KerasHub
## Citing KerasNLP

If KerasHub helps your research, we appreciate your citations.
If KerasNLP helps your research, we appreciate your citations.
Here is the BibTeX entry:

```bibtex
@misc{kerasnlp2022,
title={KerasHub},
title={KerasNLP},
author={Watson, Matthew, and Qian, Chen, and Bischof, Jonathan and Chollet,
Fran\c{c}ois and others},
year={2022},
Expand Down
2 changes: 1 addition & 1 deletion RELEASE_PROCESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Our release process consists of two main components:

- Adding a new release to the [keras-hub](https://pypi.org/project/keras-hub/)
project on the Python Package Index (pypi).
- Updating our documentation on [keras.io](https://keras.io/keras_hub/) to match
- Updating our documentation on [keras.io](https://keras.io/keras_nlp/) to match
the release.

We follow [semantic versioning](https://semver.org/) for our releases, and
Expand Down
23 changes: 10 additions & 13 deletions keras_hub/src/api_export.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

import types

import keras
from keras.saving import register_keras_serializable

try:
import namex
Expand All @@ -23,22 +23,19 @@


def maybe_register_serializable(path, symbol):
# If we have multiple export names, actually make sure to register these
# first. This makes sure we have a backward compat mapping of old serialized
# name to new class.
if isinstance(path, (list, tuple)):
# If we have multiple export names, actually make sure to register these
# first. This makes sure we have a backward compat mapping of old
# serialized names to new class.
for name in path:
name = name.split(".")[-1]
keras.saving.register_keras_serializable(
package="keras_nlp", name=name
)(symbol)
register_keras_serializable(package="keras_nlp", name=name)(symbol)
register_keras_serializable(package="keras_hub", name=name)(symbol)
if isinstance(symbol, types.FunctionType) or hasattr(symbol, "get_config"):
# We register twice, first with the old name, second with the new name,
# so loading still works under the old name.
# TODO replace keras_nlp with keras-hub after rename.
compat_name = "keras_nlp"
keras.saving.register_keras_serializable(package=compat_name)(symbol)
keras.saving.register_keras_serializable(package="keras_hub")(symbol)
# We register twice, first with keras_nlp, second with keras_hub,
# so loading still works for classes saved as "keras_nlp".
register_keras_serializable(package="keras_nlp")(symbol)
register_keras_serializable(package="keras_hub")(symbol)


if namex:
Expand Down
4 changes: 2 additions & 2 deletions keras_hub/src/utils/preset_utils_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,10 @@ def test_preset_errors(self):
backbone = BertBackbone.from_preset("bert_tiny_en_uncased")
preset_dir = self.get_temp_dir()
config = keras.utils.serialize_keras_object(backbone)
config["registered_name"] = "keras_nlp>BortBackbone"
config["registered_name"] = "keras_hub>BortBackbone"
with open(os.path.join(preset_dir, CONFIG_FILE), "w") as config_file:
config_file.write(json.dumps(config, indent=4))
with self.assertRaisesRegex(ValueError, "class keras_nlp>BortBackbone"):
with self.assertRaisesRegex(ValueError, "class keras_hub>BortBackbone"):
BertBackbone.from_preset(preset_dir)

def test_upload_empty_preset(self):
Expand Down