XCM v2: Scripting, Query responses, Exception handling and Error reporting

[gavofyork]

Moves XCM away from a hybrid script/structured language and into a pure script model. This means no recusion and an equivalence between Order and instruction (Xcm).

Advantages are:

• Simpler to learn & understand - it's just another scripting language.
• No issues with stack depth.
• No need for shallow/deep weight distinction.
• No need for distinguishing between order & instruction, particularly problematic for juggling BuyExecution and ReportOutcome.

Disadvantages are:

• Less overall structure, which means a greater possibility of nonsensical or buggy fragments and of making mistakes in construction.
• More global state in the XCM interpreter: holding is now available to instructions as well as orders, and there are several other specialised registers.

To alleviate the latter issue and especially if global machine state increases further, a later version of XCM may introduce instructions which utilise hierarchy to reduce global state.

Query responses

pallet_xcm can (and probably should) now be used as an OnResponse handler. Three functions are provided to create queries and poll for their response:

• new_query: A helper function to return a unique and registered query ID.
• take_response: A function to query and (if available) return the response of a previous XCM query created with new_query.
• new_notify_query: A helper function to return a unique query ID registered to dispatch a given handler once the outcome is returned. Such a query cannot be polled with take_response as it will be forgotten as soon as it returns and has its handler called.

Additional functions are provided for dealing with specifically outcome reporting (see later).

Exception handling

We now have an exception handling system built into the XCM executor; there are two new code registers in the VM state as well as a third register for storing an error. If regular code execution ends with an error, the error is placed in the Error Register and any XCM instructions in the Error Handler Register are removed from it and then executed. If the execution completes with no error then the Error Handler contents are cleared without being executed. Once there is no error handler to be executed (either due to the Error Handler Register being empty or because no error occured), then the Appendix Register is emptied and its code executed. This continues until there are no errors/error handler code, and no further code in the Appendix Register.

New XCM instructions:

• SetAppendix: to set the Appendix (code) Register;
• SetErrorHandler: to set the Error Handler (code) Register.

Weight management

The apparent (pre-execution) weight of the SetAppendix and SetErrorHandler instructions include the weight of the inner code that they are setting the register to, even though it may never actually get executed. Once it becomes clear that some such code will not be executed, then the weight of that code is accrued into the surplus (and may be refunded with the use the RefundSurplus if previously purchased). This happens in two situations:

• When any such instructions are executed, a surplus of weight is accrued equal to the weight of the code already in the register that it replaces.
• In the case that an XCM message executes successfully, the weight of the Error Handler Register is added to the surplus as it is dropped.

Error (outcome) reporting and the Error Register

This introduces XCM instruction outcome reporting with two new API functions in pallet_xcm::Pallet to help use it:

• report_outcome_notify: A helper function which is able to alter a given XCM into one whose outcome (success/failure) will be passed into a provided handler (which must be a dispatchable function).
• report_outcome: A helper function which is able to alter a given XCM into one whose outcome (success/failure) will be recorded and is queryable by the take_response function. A query identifier is returned to be used as the argument to take_response.

These functions work through the use of SetAppendix (which allows code to be executed even if an error occurs), the Error Register and ReportError (which sends an XCM of QueryResponse containing the current value of the Error Register to the named destination).

In general (should you wish to author your own XCM), the Error Register may be used with two new instructions ReportError and ClearError:

• ReportError: sends a QueryRepsonse with the contents of the Error Register;
• ClearError: resets the Error Register;

Additional changes

• New XCM instruction ClearOrigin which removes the origin (thereby preventing further instructions from executing with its associated privileges, useful for InitiateTeleport and InitiateReserveWithdraw).
• Alters Response type, Order::QueryBalance and Xcm::QueryResponse.

All Xcm items and Order items are merged into Instruction enum. Xcm becomes a unit struct of inner type Vec<Instruction>.

All Xcm items which included a field Vec<Order> to be executed in the local context now have those fields removed (they're no longer needed since the Orders can be placed as Instructions immediately afterwards in the Xcm vector). For the remote context, xcm: Xcm is used as the field name/type.

Compatibility

v1 messages can be auto-converted to v2 provided:

• BuyExecution::instructions is empty.
• RelayedFrom is unused.

v2 messages can be auto-converted to v1 provided:

• Only a single item corresponding to a v1::Xcm item is used for local execution in a single v2::Xcm and it must be at the begining.
• BuyExecution::weight_limit is Limited.
• ClearOrigin if used, may only follow immediately after ReceiveTeleportedAsset or ReserveAssetDeposited. The latter two may not appear with it.
• DescendOrigin, ReportOutcome, RefundSurplus are not used.

Migrating

XCM fragments

• Xcm:: and Order:: become Instruction::.
• orders/effects fields describing effects on the remote side are renamed xcm.
• For WithdrawAsset, ReceiveTeleportedAsset, DepositReserveAsset have no effects field; contents should be moved to follow the instruction in the containing Vec.
• BuyExecution:
  • New field weight_limit, which replaces debt and weight; it should be set to Limited with the inner value being the sum of weight and debt, or, alternatively just Unlimited.
  • If instructions is not empty, then contents should be moved to follow BuyExecution in the containing Vec.
  • Only fields remaining are fees and weight.
• RelayedFrom becomes DescendOrigin; RelayedFrom::instruction removed and contents should be moved to follow RelayedFrom in the containing Vec.

XCM Pallet

pallet_xcm::Config now contains two more items Origin and Call; they should just be defined to the corresponding outer types:

type Origin = Origin;
type Call = Call;

pallet_xcm has a special Origin which must be specified in the construct_runtime! macro.

XCM builder tools

The FixedWeightBounds type now has an additional type argument M: Get<u32>, which should provide the maximum number of instructions including for the contents of all code register set instructions (which are naturally recursive in nature with regards weight calculation). This limits the complexity of the weight calculation to something manageable, even in the face of recursive bombs of the form:

Xcm(vec![SetAppendix(
   Xcm(vec![SetAppendix(
      Xcm(vec![SetAppendix(
         Xcm(vec![SetAppendix(
            Xcm(vec![SetAppendix(
               Xcm(vec![])
            ); 100])
         ); 100])
      ); 100])
   ); 100])
); 100])

Most of the time you should just change:

type Weigher = FixedWeightBounds<...>;

to:

type Weigher = FixedWeightBounds<..., MaxInstructions>;

and add a parameter type:

parameter_types! {
    pub const MaxInstructions: u32 = 100;
}

TODO

• Test on_report
• Test new_query
• Test new_query_notify
• Test conversions v1 -> v2
• Test conversions v2 -> v1
• Weigher for the new instructions
• Test, especially for checking weights of error handler/appendix

[xlc]

What will happen to https://github.com/paritytech/xcm-format ? Do we want to keep it up to date?

Definitely need some RPC or something to help estimate weights on dest chain.

The name QueryReponse is a bit confusing. I initially read it as the action to query some execution response, not the response of a previous query.

This assumes transaction_version remains the same. Maybe it should be included in the payload somewhere to assert that's the case?

[KiChjang]

Really dumb question: did we decide that we can't reuse the weights system in FRAME, and thus we have to define the Weight type here?

[gavofyork]

XCM is independent of Frame.

[gavofyork]

What will happen to https://github.com/paritytech/xcm-format ? Do we want to keep it up to date?

Yes.

Definitely need some RPC or something to help estimate weights on dest chain.

Exporting the WeightBounds impl in a crate would allow runtimes to estimate each other's weights.

This assumes transaction_version remains the same. Maybe it should be included in the payload somewhere to assert that's the case?

Not sure what you mean here. We already have polkadot-fellows/xcm-format#22 as a later change to manage transction versioning for Transact. Outcide of Transact, I don't think transaction versioning is an issue.

[KiChjang]

Don't see any big issues with functionality, only some observations and comments on them. They don't really block this PR from landing, so I'm approving.

[shawntabrizi]

nothing too controversial here from a first pass. will be able to look deeper as i complete and merge my weight pr

[shawntabrizi]

I made a note that probably we want to use BoundedVec inside of the XCM