Runconfig instead of many compiler flags

[clangenb]

Compiler flags make CI testing difficult, it is easy to miss a feature flag, which will then not be tested at all. It might be better that we introduce RunConfigs that might lock like the following:

/// Naming to be defined
struct RunConfig {
     /// Sidechain, OffchainWorker, Teeracle
     mode: WorkerMode,

     /// DCAP, IAS. Probably not necessary. IAS will be deprecated soonish.
     attestation_type: AttestationType

     /// Hmm, can we find a way to compile EVM stuff without including it in the binary?
     /// I think this is hard, because we can't just include the `enclave-runtime` at process start like they do it in substrate with the wasm blob. 
     /// It will always be linked at compile time. Needs some more thought.
     with_evm: bool
 
    // what else??
}

Needs some more thought and testing to see if we can do this without decreasing the performance...

[clangenb]

This is even more interesting considering that we can reduce the build time of a worker to 15 minutes of a flavour with our self-hosted github actions runner, which is however not parallelizable. Hence, being able to build everything with only one compiler config, would greatly reduce CI time.

[brenzi]

does this mean that a single binary can then execute all flavors? I'm not sure if this makes sense. I think the basic problem is that we have a single repo for very different use cases. In my POV we should rather go towards making worker repo a library repo and having lightweight repos for sidechains, OCW, teeracle templates which can then be forked by 3rd parties to make customizations.
However, if CI for libraries should check that all templates still work, we're back at square one.....
And, this approach obviously conflicts with the monorepo idea and introduces dependency maintenance, but it may support the idea of an SDK with release cycles

Bottom line: we shouldn't take this change too easily IMO

(related #1142 #1138)

[clangenb]

Yeah, I think so too, this needs some careful thought.

The underlying problem here that makes everything complex is, actually, that the compiler flags reach far too deep into the core code and make it very hard to trace the effects. My current perspective on this is now: We should keep compiler flags. However, the compiler flags should only have an impact in the app-libs domain where the customizations happen, which means that we need to refactor more core-code to be generic over some config values. Something that should not be too far-fetched could be like:

// somewhere in app-libs

#[cfg(feature = "evm")]
pub use evm_config::*;

#[cfg(feature = "offchain-worker")]
pub use offchain_worker_config::*;

In these config modules, we'd have things like:

pub mod offchain_worker_config {
    
   /// filter out direct calls.
   /// This is actually a good example. For the oli guys I had to do a patch deep inside the core code 
   /// to allow direct calls for their best-energy worker, but if we have these config modules, we could just add
   /// a `best_energy_config` module where they can define their own top filter.
   type TopFilter = IndirectCallsFilter;
}

From this point onwards, I think we could start more easily separating a library codebase and a user-application templates (sidechain, teeracle, etc) if this is still what we want.