feat: chainSpec based controller config; Types from apps-config

CHAIN_INTEGRATION.md

@@ -0,0 +1,77 @@
+# Substrate Api Sidecar chain integration guide
+
+This guide aims to help chain builders integrate their Substrate FRAME based chain with Substrate API Sidecar.
+
+## Table of contents
+
+- [Polkadot-js API type definition support](#polkadot-js-API-type-definition-support)
+- [Controller configuration](controller-configuration)
+
+## Polkadot-js API type definition support
+
+In order decode the SCALE encoded data from a substrate based node, polkadot-js needs to have a registry of type definitions. Sidecar pulls in chain type definitions from the [@polkadot/apps-config package hosted on NPM](https://www.npmjs.com/package/@polkadot/apps-config).
+
+If the chain's type definitions do not already exist in [@polkadot/apps-config](https://github.com/polkadot-js/apps/tree/master/packages/apps-config) they will need to be added via PR to polkadot-js/apps by following their [instructions for API config](https://github.com/polkadot-js/apps/tree/master/packages/apps-config#api).
+
+Before taking any other steps to integrate a chain with Sidecar, a chain's up-to-date type definitions must be included in a published version of @polkadot/apps-config.
+
+## Controller configuration
+
+Sidecar offers the ability to configure which controllers to mount. Sidecar uses a chain's spec name to determine which controller config to use, and if no config is linked to a spec name, then the [default config](/src/chains-config/defaultControllers.ts) is used.
+
+A chain builder can follow the steps below and submit a chain's controller config via PR, where it will be reviewed and merged once deemed ready by the maintainers.
+
+#### 1) Create a controller config
+
+ Create a controller config for your chain. The shape of the controller config is specified [here](/src/chains-config/ControllerConfig.ts). The `controller` property has keys from the [controller export](/src/controllers/index.ts), which is an exhaustive collection of the available controller classes. In order to see the path(s) associated with a controller one must look in the controller source code.
+
+ The easiest way to start creating a controller config would be to copy [defaultControllers.ts](/src/chains-config/ControllerConfig.ts) and name the file and export `{specName}Controllers`. Then change the boolean values to indicate wether or not to mount a controller and its paths. Ensure to export the controller config from `chains-config` by adding `export * from './{specName}Controllers.ts'` in [/src/chains-config/index.ts](/src/chains-config/index.ts).
+
+ To determine what controllers to include, one must consider the runtime logic, specifically what pallets the chain uses. It is important to keep in mind the assumptions the service's logic makes and what exact pallets the service queries.
+
+An example is in order. If we want to use [`PalletsStakingProgressController`](/src/controllers/pallets/PalletsStakingProgressController.ts), first check [`PalletsStakingProgressService.ts`](/src/services/pallets/PalletsStakingProgressService.ts); here we see it queries `staking`, `sessions`, `babe` pallets and makes certain assumptions about how the pallets are used together in the runtime. If we determine that both have all storage items queried, and the assumptions made about the staking system are correct, we can then declare in `{specName}Controllers` (the controller config object) that we want to mount the controller by giving `PalletsStakingProgressController` property `true` as the value (`{ PalletsStakingProgressController: true }`). Finally, we can test it out by starting up Sidecar against our chain and accessing the `pallets/staking/progress` endpoint, verifying that the information returned is correct.
+
+In some circumstances, a chain may need a new path, modify a path or alter business logic for a path. Path changes that help a chain support wallets will be given priority. Breaking changes to paths are usually rejected.
+
+##### Basic balance transfer support
+
+In order to support traditional balance transfers the chain's Sidecar endpoints should support account balance lookup, transaction submission, transaction material retrieval, and block queries.
+
+To support these features the following endpoints are necessary:
+
+|                   Path                   |           Controller          |                                 Description                                |
+|:----------------------------------------:|:-----------------------------:|:--------------------------------------------------------------------------:|
+|       GET  `/transaction/material`       | TransactionMaterialController | Get all the network information needed to construct a transaction offline. |
+|            POST `/transaction`           |  TransactionSubmitController  |            Submit a transaction to the node's transaction pool.            |
+| GET `/blocks/head`  & `/blocks/{number}` |        BlocksController       |                                Get a block.                                |
+|  GET `accounts/{accountId}/balance-info` | AccountsBalanceInfoController |                   Get balance information for an account.                  |
+
+#### 2) Update `specToControllerMap`
+
+In order for Sidecar to use your controller config, the `specToControllerMap` in [/src/chains-config/index.ts](/src/chains-config/index.ts) must be updated with the chain's `specName` and controller config by adding them as a property to `specToControllerMap`:
+
+```javascript
+const specToControllerMap = {
+  kulupu: kulupuControllers,
+  mandala: mandalaControllers,
+  {specName}: {specName}Controllers,
+};
+```
+
+#### 3) Test
+
+Run it against a node running your chain in archive mode:
+
+- Ensure all the expected paths work, including the root path, preferably with tests
+- Exercise each query param of every path
+- Make sure transaction submission works
+- Try out historic queries across runtimes where types might change
+
+#### 4) Submit your PR
+
+Make sure it passes lint with `yarn lint --fix` and tests with `yarn test`. Then submit a PR for review.
+
+#### 5) Maintenance
+
+- Keep types up-to-date in `@polkadot/apps-config`
+- If the business logic or storage of a chain's pallet queried by a Sidecar endpoint is changed, ensure that the corresponding service logic is updated in Sidecar as well

README.md

@@ -36,7 +36,7 @@ This service requires Node version 12 or higher.
 - [Configuration](#configuration)
 - [Debugging fee and payout calculations](#debugging-fee-and-payout-calculations)
 - [Available endpoints](https://paritytech.github.io/substrate-api-sidecar/dist/)
-- [Chain compatibility](#chain-compatibility)
+- [Chain integration guide](/CHAIN_INTEGRATION.md)
 - [Docker](#docker)
 - [Note for maintainers](#note-for-maintainers)
 - [Roadmap](#roadmap)
@@ -200,15 +200,9 @@ CALC_DEBUG=1 yarn
 
 [Click here for full endpoint docs.](https://paritytech.github.io/substrate-api-sidecar/dist/)
 
-## Chain compatibility
+## Chain integration guide
 
-Sidecar should be compatible with any [Substrate](https://substrate.dev/) based chain, given
-constraints:
-
--   The chain ought to use FRAME and the `balances` pallet.
--   The chain is being finalized (by running `grandpa`).
--   If the chain is running on custom Node binaries, the JSON-RPC API should be backwards compatible
-    with the default Substrate Node.
+[Click here for chain integration guide.](/CHAIN_INTEGRATION.md)
 
 ## Docker
 

docs/src/openapi-v1.yaml

@@ -45,6 +45,15 @@ paths:
           description: Block height (as a non-negative integer) or hash
             (as a hex string).
           format: unsignedInteger or $hex
+      - name: token
+        in: query
+        description: 'Token to query the balance of. If not specified it will query
+          the chains native token (e.g. DOT for Polkadot). Note: this is only relevant
+          for chains that support multiple tokens through the ORML tokens pallet.'
+        required: false
+        schema:
+          type: string
+          description: Token symbol
       responses:
         "200":
           description: successful operation
@@ -717,6 +726,10 @@ components:
           type: string
           description: Account nonce.
           format: unsignedInteger
+        tokenSymbol:
+          type: string
+          description: Token symbol of the balances displayed in this response.
+          format: unsignedInteger
         free:
           type: string
           description: Free balance of the account. Not equivalent to _spendable_

package.json

@@ -36,7 +36,8 @@
     "test": "jest --silent"
   },
   "dependencies": {
-    "@polkadot/api": "^2.9.1",
+    "@polkadot/api": "^2.10.1",
+    "@polkadot/apps-config": "^0.70.1",
     "@polkadot/util-crypto": "^4.2.1",
     "@substrate/calc": "^0.1.2",
     "confmgr": "^1.0.6",
@@ -52,11 +53,11 @@
     "@types/jest": "^26.0.16",
     "@types/morgan": "^1.9.2",
     "@types/triple-beam": "^1.3.2",
-    "@typescript-eslint/eslint-plugin": "4.9.0",
-    "@typescript-eslint/parser": "4.9.0",
-    "eslint": "^7.14.0",
-    "eslint-config-prettier": "^6.15.0",
-    "eslint-plugin-prettier": "^3.1.4",
+    "@typescript-eslint/eslint-plugin": "4.9.1",
+    "@typescript-eslint/parser": "4.9.1",
+    "eslint": "^7.15.0",
+    "eslint-config-prettier": "^7.0.0",
+    "eslint-plugin-prettier": "^3.2.0",
     "eslint-plugin-simple-import-sort": "^6.0.1",
     "jest": "^26.6.3",
     "prettier": "^2.2.1",

src/App.ts

@@ -83,6 +83,9 @@ export default class App {
 	listen(): void {
 		this.app.listen(this.port, this.host, () => {
 			console.log(`Listening on http://${this.host}:${this.port}/`);
+			console.log(
+				`Check the root endpoint (http://${this.host}:${this.port}/) to see the available endpoints for the current node`
+			);
 		});
 	}
 

src/Config.ts → src/SidecarConfig.ts

@@ -2,7 +2,7 @@ import { ConfigManager } from 'confmgr';
 
 import * as configTypes from '../config/types.json';
 import { Specs } from './Specs';
-import { CONFIG, ISidecarConfig, MODULES } from './types/config';
+import { CONFIG, ISidecarConfig, MODULES } from './types/sidecar-config';
 
 function hr(): string {
 	return Array(80).fill('━').join('');
@@ -11,7 +11,7 @@ function hr(): string {
 /**
  * Access a singleton config object that will be intialized on first use.
  */
-export class Config {
+export class SidecarConfig {
 	private static _config: ISidecarConfig | undefined;
 	/**
 	 * Gather env vars for config and make sure they are valid.

src/Specs.ts

@@ -1,6 +1,6 @@
 import { ConfigSpecs, SpecsFactory } from 'confmgr';
 
-import { CONFIG, MODULES } from './types/config';
+import { CONFIG, MODULES } from './types/sidecar-config';
 
 /**
  * Access a singleton specification for config enviroment variables that will

src/chains-config/defaultControllers.ts

@@ -0,0 +1,30 @@
+import { ControllerConfig } from '../types/chains-config';
+
+/**
+ * Controllers that Sidecar will always default to. This will always be
+ * the optimal controller selection for Polkadot and Kusama.
+ */
+export const defaultControllers: ControllerConfig = {
+	controllers: {
+		Blocks: true,
+		AccountsStakingPayouts: true,
+		AccountsBalanceInfo: true,
+		AccountsStakingInfo: true,
+		AccountsVestingInfo: true,
+		NodeNetwork: true,
+		NodeVersion: true,
+		NodeTransactionPool: true,
+		RuntimeCode: true,
+		RuntimeSpec: true,
+		RuntimeMetadata: true,
+		TransactionDryRun: true,
+		TransactionMaterial: true,
+		TransactionFeeEstimate: true,
+		TransactionSubmit: true,
+		PalletsStakingProgress: true,
+		PalletsStorage: true,
+	},
+	options: {
+		finalizes: true,
+	},
+};

src/chains-config/index.ts

@@ -0,0 +1,56 @@
+import { ApiPromise } from '@polkadot/api';
+import AbstractController from 'src/controllers/AbstractController';
+import { AbstractService } from 'src/services/AbstractService';
+
+import { controllers } from '../controllers';
+import { ControllerConfig } from '../types/chains-config';
+import { defaultControllers } from './defaultControllers';
+import { kulupuControllers } from './kulupuControllers';
+import { mandalaControllers } from './mandalaControllers';
+
+const specToControllerMap = {
+	kulupu: kulupuControllers,
+	mandala: mandalaControllers,
+};
+
+/**
+ * Return an array of instantiated controller instances based off of a `specName`.
+ *
+ * @param api ApiPromise to inject into controllers
+ * @param implName
+ */
+export function getControllersForSpec(
+	api: ApiPromise,
+	specName: string
+): AbstractController<AbstractService>[] {
+	if (specToControllerMap[specName]) {
+		return getControllersFromConfig(api, specToControllerMap[specName]);
+	}
+
+	// If we don't have the specName in the specToControllerMap we use the default
+	// contoller config
+	return getControllersFromConfig(api, defaultControllers);
+}
+
+/**
+ * Return an array of instantiated controller instances based off of a
+ * `ControllerConfig`.
+ *
+ * @param api ApiPromise to inject into controllers
+ * @param config controller mount configuration object
+ */
+function getControllersFromConfig(api: ApiPromise, config: ControllerConfig) {
+	// If we don't typecast here, tsc thinks its just [string, any][]
+	const controllersToInclude = Object.entries(config.controllers) as [
+		keyof typeof controllers,
+		boolean
+	][];
+
+	return controllersToInclude.reduce((acc, [controllerName, shouldMount]) => {
+		if (shouldMount) {
+			acc.push(new controllers[controllerName](api, config.options));
+		}
+
+		return acc;
+	}, [] as AbstractController<AbstractService>[]);
+}

src/chains-config/kulupuControllers.ts

@@ -0,0 +1,26 @@
+import { ControllerConfig } from '../types/chains-config';
+
+export const kulupuControllers: ControllerConfig = {
+	controllers: {
+		Blocks: true,
+		AccountsStakingPayouts: false,
+		AccountsBalanceInfo: true,
+		AccountsStakingInfo: false,
+		AccountsVestingInfo: false,
+		NodeNetwork: true,
+		NodeVersion: true,
+		NodeTransactionPool: true,
+		RuntimeCode: true,
+		RuntimeSpec: true,
+		RuntimeMetadata: true,
+		TransactionDryRun: true,
+		TransactionMaterial: true,
+		TransactionFeeEstimate: true,
+		TransactionSubmit: true,
+		PalletsStakingProgress: false,
+		PalletsStorage: true,
+	},
+	options: {
+		finalizes: false,
+	},
+};

src/chains-config/mandalaControllers.ts

@@ -0,0 +1,29 @@
+import { ControllerConfig } from '../types/chains-config';
+
+/**
+ * Controllers for mandala, acala's test network.
+ */
+export const mandalaControllers: ControllerConfig = {
+	controllers: {
+		Blocks: true,
+		AccountsStakingPayouts: true,
+		AccountsBalanceInfo: true,
+		AccountsStakingInfo: true,
+		AccountsVestingInfo: true,
+		NodeNetwork: true,
+		NodeVersion: true,
+		NodeTransactionPool: true,
+		RuntimeCode: true,
+		RuntimeSpec: true,
+		RuntimeMetadata: true,
+		TransactionDryRun: true,
+		TransactionMaterial: true,
+		TransactionFeeEstimate: true,
+		TransactionSubmit: true,
+		PalletsStakingProgress: true,
+		PalletsStorage: true,
+	},
+	options: {
+		finalizes: true,
+	},
+};

src/controllers/accounts/AccountsBalanceInfoController.ts

@@ -62,14 +62,19 @@ export default class AccountsBalanceController extends AbstractController<Accoun
 	 * @param res Express Response
 	 */
 	private getAccountBalanceInfo: RequestHandler<IAddressParam> = async (
-		{ params: { address }, query: { at } },
+		{ params: { address }, query: { at, token } },
 		res
 	): Promise<void> => {
+		const tokenArg =
+			typeof token === 'string'
+				? token.toUpperCase()
+				: this.api.registry.chainToken;
+
 		const hash = await this.getHashFromAt(at);
 
 		AccountsBalanceController.sanitizedSend(
 			res,
-			await this.service.fetchAccountBalanceInfo(hash, address)
+			await this.service.fetchAccountBalanceInfo(hash, address, tokenArg)
 		);
 	};
 }

src/controllers/blocks/BlocksController.ts

@@ -5,6 +5,10 @@ import { BlocksService } from '../../services';
 import { INumberParam } from '../../types/requests';
 import AbstractController from '../AbstractController';
 
+interface ControllerOptions {
+	finalizes: boolean;
+}
+
 /**
  * GET a block.
  *
@@ -65,7 +69,7 @@ import AbstractController from '../AbstractController';
  * - `OnFinalize`: https://crates.parity.io/frame_support/traits/trait.OnFinalize.html
  */
 export default class BlocksController extends AbstractController<BlocksService> {
-	constructor(api: ApiPromise) {
+	constructor(api: ApiPromise, private readonly options: ControllerOptions) {
 		super(api, '/blocks', new BlocksService(api));
 		this.initRoutes();
 	}
@@ -91,7 +95,7 @@ export default class BlocksController extends AbstractController<BlocksService>
 		const extrsinsicDocsArg = extrinsicDocs === 'true';
 
 		const hash =
-			finalized === 'false'
+			finalized === 'false' || !this.options.finalizes
 				? (await this.api.rpc.chain.getHeader()).hash
 				: await this.api.rpc.chain.getFinalizedHead();