fairseq v2

[cbalioglu]

fairseq v2

Hi everyone,

We wanted to let you know that FAIR will soon kick off the fairseq v2 project in order to modernize our code base and to address fairseq’s long standing structural and technical issues. Our high-level goals are for fairseq to:

• have a clean, maintainable, well-tested code base that leverages the latest features of PyTorch and its ecosystem.
• preserve the performance optimizations of the original fairseq that have accumulated throughout the years.
• enforce user code to be separate from library code via well-defined APIs so that it can be used both as a framework and as a library.
• offer a familiar interface for researchers used to the original fairseq.
• come with new fault-tolerant features and tools for large-scale training and inference.

Note that, although our aim is to keep them at a minimum, in order to realize some of our goals we will be making backwards-incompatible changes, mostly in our trainer and data loading APIs. Along with these changes we also plan to offer a large set of extensibility APIs that will mitigate the need to fork/branch fairseq in majority of use cases.

We will share our pre-release versions in the upcoming months and will be looking forward to your feedback. In the meantime you can follow our GitHub Issues page to track our progress.

Thank you!

FAQs

Should I keep using fairseq right now? What will happen to my work when fairseq v2 gets released?
Yes, please continue using fairseq. fairseq, as it is today, already provides a great set of features for researchers. None of that is going away. As we mentioned, there will be backwards-incompatible changes, but the procedure to migrate an existing work to fairseq v2 will be clearly documented and our goal is to confine it to mostly syntactical updates and minor refactorings.

Will fairseq v2 have feature-parity with fairseq?
The official v2 release at the end of the project will have feature-parity with fairseq. Our pre-release versions will start with a smaller set of features and then gradually increase the scope until we have full coverage of fairseq features.

What will happen to current fairseq?
Till we officially release v2 we will continue to support and maintain the current fairseq. However we won’t accept any PRs besides critical bug fixes and minor feature improvements (mostly to unblock ongoing research projects). The release of v2 will be an in-place update, meaning we will be using the fairseq repository and its main branch to host our next version. After that point for researchers that need access to the original fairseq, there will be a “v1” branch in the fairseq repository. Note though that once v2 is officially released, the original version won’t be maintained anymore.

How can I test fairseq v2?
In the upcoming months we will communicate on our GitHub Issues page when pre-release versions of fairseq v2 are ready to download and test.

[maoquan-ms]

Hi @cbalioglu ,
good to know fairseq v2 is coming.
in the v2, fairseq should think about how to integrate with huggingface models (train from huggingface models)
That could be useful.

[YooSungHyun]

PLZ, making easily...!

nowadays, i`m always using huggingface 😥

[Hannibal046]

So glad to know fairseq v2 is coming !
Currently, I am familiar with Huggingface/transformers and a rookie for fairseq. I find it easy to extend transformers but a little bit hard for fairseq. From my personal perspective, a detailed and thorough document and code decomposability are what I want.

[SirRob1997]

Great to see effort being made in this direction and not just abondening fairseq all together which felt like it with the release of metaseq. A few suggestions:

1. Please make arguments accessible globally to avoid piping arguments through multiple levels of inheritance and have one consistent way of adding arguments and setting their defaults, it is one of the most annoying things currently. In its current version, we mix argument defaults in argparse, registering architectures and setting defaults there, configs (with the most recent changes), or just using getattr with defaults throughout the codebase. With arguments being accessible everywhere, class/method signatures also remain way more concise than they currently are.
2. Also I'd generally argue that less inheritance and integrating functionality into existing procedures with flags is often beneficial for maintainability and to avoid jumping between multiple files.
3. From a code-organization standpoint, it probably would make sense to have shared folders containing modules like linears, dropout, multi-head attention, and even combinations of those like the transformer but we should also consider having split-off folders for specific use-cases i.e. machine translation, language modeling, speech, etc. This would allow sub-communities to contribute (mostly) only to these respective parts and make things a little bit more maintainable from a PR perspective.
4. Please add a linting/formatting workflow to PR's and general contributions to have a consistent codebase.

Feel free to reach out in case you want any more feedback or contributors to the current architecture drafts of fairseq v2, I'd be happy to help out as I'm a heavy user of fairseq v1!

[WilliamTambellini]

Please keep an eye on torch JIT export: as today, transformers in v1 are not exportable:
#4459

[erip]

Is there are plan to support external maintainers/contributors? I was discussing the possibility of being an external maintainer with Myle some time ago, but it sorted of petered out. Many Meta-external PRs go unmerged in the current fairseq which is discouraging and can hurt adoption. I'm happy to continue this conversation as appropriate.

I'm hopeful that going forward, regular versions can be cut for fairseq v2 as well. Backwards compatibility is very iffy with the current fairseq and since there are no regular versions being cut, often I have to tote around commit hashes as a proxy which can be a bit of a headache.

[st-vincent1]

Thrilled to hear that fairseq is getting a re-release. I think it's a great idea to keep the core library and implemented papers separate as in its current state the repo can be quite overwhelming. One of the things I'd suggest is perhaps more tutorials on how the library is built and how one can alter/extend individual components of a full model (that cannot be easily adapted with command line options, so input/output, search etc.). While figuring it out on your own is a lot of fun, I think it can be frustrating to new users of fairseq.

[awan-10]

Looking forward to fairseq-v2! Is there any ETA on the v2 release?

[cbalioglu]

Hey everyone, sorry for the delay in checking your messages! Thanks a lot for all your feedback so far! Hope the new version will address most of your concerns raised here.

@awan-10 we plan to have an early release before the end of the year. Most likely in December so that we can start gathering more concrete feedback and also start accepting external contributions.

@erip I see no reason for not supporting external maintainers/contributors. Quite contrary I believe it is critical for the project to be as open as possible if we want it succeed (similar to PyTorch). In the new version we will have a proper CODEOWNERS model which will be open to anyone willing to commit to the project. I will get in touch with you once we get close to the first public release.