Extract solana-config-program-api and use it in solana-account-decoder

[kevinheavey]

Problem

solana-account-decoder depends on solana-config-program, which depends on solana-program-runtime which is an awful thing to depend on if you don't need it (I can elaborate if you like, but one reason is that it brings in reqwest as a dependency). The good news is that solana-program-runtime is not part of solana-config-program's API, so if we extract a separate solana-config-program-api crate, we can use that instead and be free of solana-program-runtime in solana-account-decoder.

Summary of Changes

• Rips the API code (basically the stuff in lib.rs) out of solana-config-program and moves it into a new crate solana-config-program-api
• Re-exports the contents of solana-config-program-api in solana-config-program for backwards compatibility.
• Replaces solana-config-program with solana-config-program-api in solana-account-decoder

Per CONTRIBUTING.md I have set the version of the new crate to 0.0.1

There are more crates in the repo that could also swap out for the lighter dependency but I decided to keep the PR small

[buffalojoec]

Thanks for taking this on.

It appears that the desired outcome could also be achieved by refactoring everything from the config program besides it's processor into the Solana SDK. This is the pattern a lot of other builtin programs follow.

However, we're planning to migrate this program to BPF, which would eliminate the solana-program-runtime dependency altogether. It also will no longer reside in the monorepo once it becomes BPF.

I think it might be more prudent to wait on the BPF implementation and make its generated client available for use in solana-account-decoder. What do you think?

[kevinheavey]

refactoring everything from the config program besides it's processor into the Solana SDK. This is the pattern a lot of other builtin programs follow.

IIUC this is a side point, but there is a lot to be said for not doing this. solana-sdk is already extremely bloated from being used as a dumping ground. The typical usage of solana-sdk involves only a small fraction of the code it contains, but with how Rust compilation works you still have to wait for the whole thing to compile. As a result solana-sdk compile times are really bad and I do not encourage making them worse solana-labs#31373

As for the main point, what is the timeline for the BPF migration? And will the API change? Because if it doesn't change then moving out the API is something that may as well be done sooner rather than later

[buffalojoec]

IIUC this is a side point, but there is a lot to be said for not doing this. solana-sdk is already extremely bloated from being used as a dumping ground. The typical usage of solana-sdk involves only a small fraction of the code it contains, but with how Rust compilation works you still have to wait for the whole thing to compile. As a result solana-sdk compile times are really bad and I do not encourage making them worse solana-labs#31373

Yeah, I also believe we should bust the SDK up. It's also another reason I'd rather aim for the BPF approach, so the program code never lands in the SDK. This approach applied to other builtin programs evicts their code from the SDK as well.

As for the main point, what is the timeline for the BPF migration? And will the API change? Because if it doesn't change then moving out the API is something that may as well be done sooner rather than later

We can't say for certain when the program itself will be migrated on-chain, but since the API should be completely unchanged, all we'd need to do is write the new BPF version. Then its API could be used instead of the config builtin here. Given the fact that this program is pretty small, it shouldn't take too long to do.

[kevinheavey]

Yeah, I also believe we should bust the SDK up. It's also another reason I'd rather aim for the BPF approach, so the program code never lands in the SDK

Are you saying you would still have the config program API added to solana-sdk? I am suggesting not adding anything related to solana-config-program to solana-sdk.

We can't say for certain when the program itself will be migrated on-chain, but since the API should be completely unchanged, all we'd need to do is write the new BPF version. Then its API could be used instead of the config builtin here

This sounds like the BPF migration isn't a blocker for moving the API into its own crate? Since we already have the API and it won't change

[buffalojoec]

@kevinheavey I'm saying migrating it to BPF would be creating a brand new crate and evicting it from the monorepo. So, both splitting into two crates here or adding the API to the SDK are likely wasted work in the end, especially for this desired goal. It's better to just use the BPF version in solana-account-decoder when it's available.

[kevinheavey]

@buffalojoec but will there not be a separate API crate for config program after the BPF migration? If not, where will the API code live?

[buffalojoec]

@buffalojoec but will there not be a separate API crate for config program after the BPF migration? If not, where will the API code live?

The jury is still out on whether it becomes a new crate or still uses solana-config-program, but no it'd be one crate just a BPF program implementation.

Think Token, etc. but obviously much smaller.

[kevinheavey]

Think Token, etc. but obviously much smaller.

Token22 is my go-to example of why using a program as a library is a bad idea but I'm optimistic this won't run into the same problems since it's so small 🙏

[buffalojoec]

@kevinheavey Let me close this for now and I'll circle back when we start the BPF migration. I'll do my best not to waste your work here! Thanks for rolling this out.

[joncinque]

Token22 is my go-to example of why using a program as a library is a bad idea but I'm optimistic this won't run into the same problems since it's so small 🙏

I'm coming in late to this, but we could move all of the integration tests out of program crates, and do that as a convention in general, in order to limit the dependencies to just a few needed ones. We do that often anyway with things like the address lookup table, associated token account, etc. Would that help?

[kevinheavey]

@joncinque I only see integration tests for token22 in token/program-2022/tests which would make them already not part of spl-token-2022 right? Perhaps I am missing something.

I was thinking about how anything that wants to use the instruction.rs file for token22 needs to compile a bunch of dependencies not used in that file, such as solana-zk-token-sdk. For example the other day I found myself reimplementing get_associated_token_account because spl-associated-token-account depends on spl-token-2022 and the extraneous dependencies were causing conflicts

[buffalojoec]

I was thinking about how anything that wants to use the instruction.rs file for token22 needs to compile a bunch of dependencies not used in that file, such as solana-zk-token-sdk.

Splitting out the API is still a good idea, and we can probably do this with a program client crate when Config becomes BPF. I was moreso saying let's not add the API separation now, since we're already going to rewrite it and make a change to the SDK/monorepo to do so.

I think what makes most sense is write the program in BPF and generate a client for it in Rust, then use that client in solana-account-decoder. It's basically what you've done here, but we have a process in-place to migrate the programs, and this work in this PR would basically be overwritten/void.

we could move all of the integration tests out of program crates, and do that as a convention in general, in order to limit the dependencies to just a few needed ones.

I like this. It would certainly save on compile times especially when testing. Fwiw, an implementation-agnostic test harness fits nicely with this as well.

[kevinheavey]

I think what makes most sense is write the program in BPF and generate a client for it in Rust, then use that client in solana-account-decoder

btw @buffalojoec I was looking at Kinobi (I take it that's what will generate the client) and I don't think anything it generates will be used in solana-account-decoder. That crate uses the following things from solana-config-program/solana-config-program-api:

• ConfigKeys, which has field serialized using short_vec, which I believe Kinobi does not support
• get_config_data, which is a custom helper function using and relies on short_vec via ConfigKeys
• The ConfigState trait which Kinobi won't generate since I believe it doesn't generate traits
• create_config_account, a helper function that relies on ConfigKeys (and short_vec) to build an AccountSharedData

So from what I can tell none of the above would be generated by Kinobi, nor would they use anything generated by Kinobi

[buffalojoec]

@kevinheavey alright good looks, thanks. Let's see how this shakes out when we roll the BPF version. As you mentioned before it's a small program, so worst case we use the BPF program crate. If the client can give us what we need, we'll use that.

I may start on this pretty soon actually. I can tag whenever I do if you want!

[joncinque]

I was thinking about how anything that wants to use the instruction.rs file for token22 needs to compile a bunch of dependencies not used in that file, such as solana-zk-token-sdk. For example the other day I found myself reimplementing get_associated_token_account because spl-associated-token-account depends on spl-token-2022 and the extraneous dependencies were causing conflicts

Sorry for the delayed response here @kevinheavey -- I wasn't sure what kind of build you were referencing, so never mind! It seems like good client crates will be the best way forward here as @buffalojoec was mentioning.