AWS APIGateway × OpenAPI (3. Output)

2022-08-02

thumbnail

I have been working on a library that integrates an OpenAPI definition with a REST API definition on the CDK. This is the third blog post of the series that will walk you through the development of the library.

Background

In the first blog post of this series, we left the following challenge,

When should the library actually output the OpenAPI definition file?

  • Should a user explicitly call a function to save?
  • Or, can we magically save the OpenAPI definition file like the CDK does to the CloudFormation template?

In this blog post, we tackle the latter case; i.e., make the library output an OpenAPI definition without an explicit call by a user.

When does the CDK output a CloudFormation template?

If we can trap the timing when the AWS Cloud Development Kit (CDK) outputs a CloudFormation template, we may output an OpenAPI definition without an explicit call by a user. So we are going to walk through the CDK source code* to locate hooks we can utilize to write an OpenAPI definition. Let's look into the cdk command first. If you want to skip the extensive code review, you can jump to the "Hooks" section.

* I have analyzed the then latest version 2.34.2 of the CDK. Things may be different in other CDK versions.

Where is the cdk command defined?

The cdk command is included in the packages/aws-cdk folder in the CDK repository. Since the repository of the CDK is huge, it was not that obvious to me where the cdk command was defined. In this blog post, I abbreviate the packages/aws-cdk folder to aws-cdk.

What do the synth and deploy subcommands do?

In terms of the CloudFormation template generation, the following two cdk subcommands are our concern,

The command line interface of the cdk command is defined in aws-cdk/lib/cli.ts. In the long lines of yargs settings, you can find definitions of synth and deploy subcommands.

The cdk subcommands eventually call corresponding methods of CdkToolkit defined in aws-cdk/lib/cdk-toolkit.ts.

CdkToolkit#synth

Definition: aws-cdk/lib/cdk-toolkit.ts#L517-L545

It calls CdkToolkit#selectStacksForDiff at aws-cdk/lib/cdk-toolkit.ts#L518:

    const stacks = await this.selectStacksForDiff(stackNames, exclusively, autoValidate);

Although it is not clear from the method name, CdkToolkit#selectionStacksForDiff does essential work for the CloudFormation template generation. It eventually calls CdkToolkit#assembly at aws-cdk/lib/cdk-toolkit.ts#L621:

    const assembly = await this.assembly();
CdkToolkit#deploy

Definition: aws-cdk/lib/cdk-toolkit.ts#L126-L282

It calls CdkToolkit#selectStacksForDeploy at aws-cdk/lib/cdk-toolkit.ts#L140:

    const stacks = await this.selectStacksForDeploy(options.selector, options.exclusively, options.cacheCloudAssembly);

Like CdkToolkit#synth, CdkToolkit#selectStacksForDeploy does essential work for the CloudFormation template generation. It also ends up with a call to CdkToolkit#assembly at aws-cdk/lib/cdk-toolkit.ts#L608:

    const assembly = await this.assembly(cacheCloudAssembly);
CdkToolkit#assembly

Definition: aws-cdk/lib/cdk-toolkit.ts#L690-L692

  private assembly(cacheCloudAssembly?: boolean): Promise<CloudAssembly> {
    return this.props.cloudExecutable.synthesize(cacheCloudAssembly);
  }

It is equivalent to a call to CloudExecutable#synthesize. CloudExecutable is defined in aws-cdk/lib/api/cxapp/cloud-executable.

CloudExecutable#synthesize

Definition: aws-cdk/lib/api/cxapp/cloud-executable.ts#L63-L68

It calls CloudExecutable#doSynthesize at aws-cdk/lib/api/cxapp/cloud-executable.ts#L65:

      this._cloudAssembly = await this.doSynthesize();
CloudExecutable#doSynthesize

Definition: aws-cdk/lib/api/cxapp/cloud-executable.ts#L70-L124

The actual synthesis is done at aws-cdk/lib/api/cxapp/cloud-executable.ts#L79:

      const assembly = await this.props.synthesizer(this.props.sdkProvider, this.props.configuration);

this.props.synthesizer is a Synthesizer defined at aws-cdk/lib/api/cxapp/cloud-executable.ts#L14:

type Synthesizer = (aws: SdkProvider, config: Configuration) => Promise<cxapi.CloudAssembly>;

In the context of aws-cdk/lib/cli.ts, this.props.synthesizer is always execProgram as it is initialized in aws-cdk/lib/cli.ts#L305-L309.

  const cloudExecutable = new CloudExecutable({
    configuration,
    sdkProvider,
    synthesizer: execProgram,
  });

execProgram is defined in aws-cdk/lib/api/cxapp/exec.ts#L12-L136. According to the following lines, it runs the command specified to the app option of the cdk command (aws-cdk/lib/api/cxapp/exec.ts#L54, aws-cdk/lib/api/cxapp/exec.ts#L65, and aws-cdk/lib/api/cxapp/exec.ts#L86 respectively):

  const app = config.settings.get(['app']);

  const commandLine = await guessExecutable(appToArray(app));

  await exec(commandLine.join(' '));

You may think you have never specified the app option to the cdk command. If you initialize your CDK project with the cdk init command, the command creates the cdk.json file and saves the default app option value in it. If you look into your cdk.json file, you will find a line similar to the following:

  "app": "npx ts-node --prefer-ts-exts bin/cdk.ts",

This command (npx ts-node --prefer-ts-exts bin/cdk.ts) is what execProgram executes.

After running the command given to the app option, execProgram loads the artifacts from the output folder specified to the output option of the cdk command (aws-cdk/lib/api/cxapp/exec.ts#L67, aws-cdk/lib/api/cxapp/exec.ts#L78, and aws-cdk/lib/api/cxapp/exec.ts#L88 respectively):

  const outdir = config.settings.get(['output']);

  env[cxapi.OUTDIR_ENV] = outdir;

  return createAssembly(outdir);

The output option is "cdk.out" by default (aws-cdk/lib/settings.ts#L75-L79):

  public readonly defaultConfig = new Settings({
    versionReporting: true,
    pathMetadata: true,
    output: 'cdk.out',
  });

Thus, our next focus is what our bin/cdk.ts does.

When does App output a CloudFormation template?

In our bin/cdk.ts, we usually create an instance of App. So when does App output a CloudFormation template? Let's look into the source code of App.

App is defined in core/lib/app.ts#L94-L155. In this blog post, I abbreviate the packages/@aws-cdk/core folder to core.

In the constructor of App, there is an interesting line at core/lib/app.ts#L131:

      process.once('beforeExit', () => this.synth());

It makes the process running App call App#synth at exit. So we can anticipate the answer would be in App#synth. Since App extends Stage, App#synth is equivalent to Stage#synth.

Stage#synth

Definition: core/lib/stage.ts#L174-L183

  public synth(options: StageSynthesisOptions = { }): cxapi.CloudAssembly {
    if (!this.assembly || options.force) {
      this.assembly = synthesize(this, {
        skipValidation: options.skipValidation,
        validateOnSynthesis: options.validateOnSynthesis,
      });
    }

    return this.assembly;
  }

It calls synthesize defined at core/lib/private/synthesis.ts#L23-L50.

I have found that the following two lines in synthesize lead to two possible hooks we can count on,

validateTree

Definition: core/lib/private/synthesis.ts#L201-L214

function validateTree(root: IConstruct) {
  const errors = new Array<ValidationError>();

  visit(root, 'pre', construct => {
    for (const message of construct.node.validate()) {
      errors.push({ message, source: construct });
    }
  });

  if (errors.length > 0) {
    const errorList = errors.map(e => `[${e.source.node.path}] ${e.message}`).join('\n  ');
    throw new Error(`Validation failed with the following errors:\n  ${errorList}`);
  }
}

validateTree traverses (visits) all the nodes in the constructs tree starting from root and applies Node#validate to each of them. Node#validate calls IValidation#validate of IValidations attached to the node with Node#addValidation.

So IValidation can be a hook to output an OpenAPI definition. Please refer to "Validator hook" for how to use it.

synthesizeTree

Definition: core/lib/private/synthesis.ts#L174-L191

function synthesizeTree(root: IConstruct, builder: cxapi.CloudAssemblyBuilder, validateOnSynth: boolean = false) {
  visit(root, 'post', construct => {
    const session = {
      outdir: builder.outdir,
      assembly: builder,
      validateOnSynth,
    };


    if (Stack.isStack(construct)) {
      construct.synthesizer.synthesize(session);
    } else if (construct instanceof TreeMetadata) {
      construct._synthesizeTree(session);
    } else {
      const custom = getCustomSynthesis(construct);
      custom?.onSynthesize(session);
    }
  });
}

synthesizeTree traverses (visits) all the nodes in the constructs tree starting from root and processes each construct. The important line is at core/lib/private/synthesis.ts#L183:

      construct.synthesizer.synthesize(session);

Here construct is an instance of Stack, and Stack#synthesizer is DefaultStackSynthesizer by default. So the above line usually becomes a call to DefaultStackSynthesizer#synthesize.

DefaultStackSynthesizer#synthesize

Definition: core/lib/stack-synthesizers/default-synthesizer.ts#L387-L424

DefaultStackSynthesizer#synthesize calls DefaultStackSynthesizer#synthesizeStackTemplate at core/lib/stack-synthesizers/default-synthesizer.ts#L400:

    this.synthesizeStackTemplate(this.stack, session);

DefaultStackSynthesizer#synthesizeStackTemplate is equivalent to a call to Stack#_synthesizeTemplate (core/lib/stack-synthesizers/default-synthesizer.ts#L380-L382):

  protected synthesizeStackTemplate(stack: Stack, session: ISynthesisSession) {
    stack._synthesizeTemplate(session, this.lookupRoleArn);
  }

Stack#_synthesizeTemplate

Definition: core/lib/stack.ts#L770-L804

This method calls Stack#_toCloudFormation at core/lib/stack.ts#L779:

    const template = this._toCloudFormation();

Stack#_toCloudFormation

Definition: core/lib/stack.ts#L1007-L1045

Please pay attention to the following two lines (core/lib/stack.ts#L1031-L1032):

    const elements = cfnElements(this);
    const fragments = elements.map(e => this.resolve(e._toCloudFormation()));

This method collects all the child CfnElements with cfnElements, and applies CfnElement#_toCloudFormation to each of them. CfnElement is the base class of every L1 construct, and CfnElement#_toCloudFormation is an internal method defined at core/lib/cfn-element.ts#L161.

So the CfnElement#_toCloudFormation method can be another hook to output an OpenAPI definition. Please refer to "_toCloudFormation hook" for how to use it.

Hooks

According to the above analysis, there may be two hooks we can utilize to output an OpenAPI definition.

Validator hook

A "validator hook" utilizes IValidation that we can attach to Nodes with Node#addValidation. Please refer to the section "validateTree" for more detailed analysis.

I have chosen this hook to implement my library so far because of its simplicity. The following code is an excerpt from the constructor of RestApiWithSpec. You can find the full definition on my GitHub repository.

  constructor(scope: Construct, id: string, readonly props: RestApiWithSpecProps) {
    // ... other initialization steps
    Node.of(this).addValidation({
      validate: () => this.synthesizeOpenApi(), // synthesizeOpenApi writes the OpenAPI definition
    });
  }

A drawback is that it does not work if validation is disabled.

_toCloudFormation hook

A "_toCloudFormation hook" utilizes an internal method CfnElement#_toCloudFormation that we can override. Please refer to the section "synthesizeTree" for more detailed analysis.

If I used this hook in my library, the constructor of RestApiWithSpec could become something similar to the following:

  constructor(scope: Construct, id: string, readonly props: RestApiWithSpecProps) {
    // ... other initialization steps
    class ToCloudFormationHook extends CfnElement {
      constructor(private scope: RestApiWithSpec, id: string) {
        super(scope, id);
      }

      _toCloudFormation() {
        this.scope.synthesizeOpenApi(); // synthesizeOpenApi writes the OpenAPI definition
        return {}; // no CloudFormation resource is actually added
      }
    }
    new ToCloudFormationHook(this, 'ToCloudFormationHook');
  }

There are some disadvantages to this hook.

  • CfnElement#_toCloudFormation is an internal method that may be subject to change.
  • In my experiment, ToCloudFormationHook#_toCloudFormation was called twice (I have not figured out why).

Conclusion

In this blog post, we have walked through the source code of the CDK. We have located the following two hooks we can use to output an OpenAPI definition,

I have chosen "Validator hook" for my library because of its simplicity.

Appendix

This section introduces some utility functions used in the CDK.

visit

Definition: core/lib/private/synthesis.ts#L219-L232

function visit(root: IConstruct, order: 'pre' | 'post', cb: (x: IConstruct) => void) {
  if (order === 'pre') {
    cb(root);
  }

  for (const child of root.node.children) {
    if (Stage.isStage(child)) { continue; }
    visit(child, order, cb);
  }

  if (order === 'post') {
    cb(root);
  }
}

This function traverses all the nodes in the constructs tree starting from root, and applies a function specified to cb to each of them.

cfnElements

Definition: core/lib/stack.ts#L1279-L1292

function cfnElements(node: IConstruct, into: CfnElement[] = []): CfnElement[] {
  if (CfnElement.isCfnElement(node)) {
    into.push(node);
  }

  for (const child of Node.of(node).children) {
    // Don't recurse into a substack
    if (Stack.isStack(child)) { continue; }


    cfnElements(child, into);
  }

  return into;
}

This function recursively collects every CfnElement in the constructs tree starting from node.

Tweet