CI

Continuous Integration details for the Multicast project.

CI Service Providers

Continuous integration testing for the Multicast project is handled by GitHub Actions and the generous CircleCI service.

CI/CD Metrics

GitHub

GitHub Actions Metrics

CircleCI

CircleCI

Other Metrics

Many additional services are used for enhancing CI/CD with additional metrics, insights, and automated analysis.

DeepSource CodeCov OpenSSF Best Practices CodeRabbit Pull Request Reviews Average time to resolve an issue Percentage of issues still open

Triggering CI/CD

GitHub Actions

Triggers

Workflows

push

.github/workflows/bandit.yml, .github/workflows/CI-BUILD.yml, .github/workflows/codeql-analysis.yml, .github/workflows/makefile-lint.yml, .github/workflows/markdown-lint.yml, .github/workflows/shellcheck.yml, .github/workflows/yaml-lint.yml

workflow_run

.github/workflows/CI-CHGLOG.yml, .github/workflows/CI-MATs.yml, .github/workflows/CI-DOCS.yml, .github/workflows/Tests.yml

pull_request

.github/workflows/bandit.yml, .github/workflows/codeql-analysis.yml, .github/workflows/makefile-lint.yml, .github/workflows/markdown-lint.yml, .github/workflows/shellcheck.yml, .github/workflows/yaml-lint.yml

pull_request_target

.github/workflows/Labeler.yml

schedule

.github/dependabot.yml, .github/workflows/codeql-analysis.yml, .github/workflows/scorecard.yml

release

.github/workflows/CD-PyPi.yml

CircleCI Pipelines

Triggers

Pipelines

push (via a webhook)

.circleci/config.yml

Appveyor Jobs

[!CAUTION] The Appveyor integration is still experimental, as the multicast module is not currently able to support running on Windows without Python socket support.

Triggers

Jobs

pull_request (via a webhook)

.appveyor.yml

Testing in CI/CD

Acceptance Testing with GitHub Actions

How on-workflow_run triggers propagate

        %%{ init: { 'theme': 'base', 'themeVariables': { 'darkMode': true, 'primaryColor': '#3e3e3e', 'background': 'transparent', 'primaryBorderColor': 'Orange', 'lineColor': 'darkOrange', 'secondaryColor': 'transparent', 'tertiaryColor': '#030303' }} }%%
graph TD;
    PUSH-EVENT-->CI-Build.yml;
    CI-Build.yml-->multicast-build-*;
    CI-Build.yml-->BUILD-info.txt;
    CI-Build.yml-->Build-Summary-Artifact.txt;
    multicast-build-*-->CI-MATs.yml;
    multicast-build-*-->CI-CHGLOG.yml;
    BUILD-info.txt-->CI-MATs.yml;
    BUILD-info.txt-->CI-CHGLOG.yml;
    CI-CHGLOG.yml-->CHANGELOG.md;
    CI-CHGLOG.yml-->chglog-info.txt;
    CI-CHGLOG.yml-->chglog-Summary-Artifact.txt;
    CI-MATs.yml-->multicast-info-*;
    CI-MATs.yml-->MATs-Summary-Artifact.txt;
    multicast-build-*-->CI-Tests.yml;
    multicast-info-*-->CI-Tests.yml;
    CI-Tests.yml-->COVERAGE;
    CI-Tests.yml-->\*-Test-Report-\*-*;
    CI-Tests.yml-->Integration-Summary-Artifact.txt;
    multicast-build-*-->CI-DOCS.yml;
    multicast-info-*-->CI-DOCS.yml;
    CI-DOCS.yml-->Multicast-Documentation-*-ALL;
    CI-DOCS.yml-->DOCUMENTATION-Summary-Artifact.txt;

  1. PUSH-EVENT: This event triggers the entire Acceptance Testing CI/CD workflow.

  2. CI-Build.yml: The main configuration file for the build process, which generates several artifacts: A. multicast-build-{{ sha }}: The built package artifact. B. BUILD-info.txt: Contains essential details about the build execution, including: i. Build Run ID: A unique identifier for the build run. ii. Build Artifact’s ID/URL/Name/Digest: Information about the generated artifact, such as its identifier, location, name, and digest for verification. iii. Git Commit Info: Details about the commit associated with the build, including the SHA, reference, and branch. C. Build-Summary-Artifact.txt (BUILD-COMMENT-BODY-{{ sha }}): A summary of the build process, highlighting key outcomes and metrics.

  3. CI-CHGLOG.yml: A configuration file that processes the build info to create: A. CHANGELOG.md (multicast-chglog-{{ build_sha }}): The generated CHANGELOG document. B. chglog-info.txt (multicast-chglog-info-{{ build_sha }}): Contains all the information from the “Build-Info.txt” along with additional details about the “CI-CHGLOG.yml” workflow run, including: i. CHANGELOG Workflow Run ID: A unique identifier for the MATs workflow run. ii. CHANGELOG Artifact’s ID/URL/Name/Digest: Information about the generated CHANGELOG.md artifact, such as its identifier, location, name, and digest for verification. iii. Git Commit Info: Details about the commit associated with the build, including the nearest TAG (e.g., release or pre-release), and the previous release used for comparing changes. C. chglog-Summary-Artifact.txt: A summary of the CHANGELOG generation process, highlighting key outcomes and metrics.

  4. CI-MATs.yml: A configuration file that processes the build artifacts to create: A. multicast-info.txt (multicast-info-{{ build_sha }}): Contains all the information from the “Build-Info.txt” along with additional details about the “CI-MATs.yml” workflow run, including: i. MATs Workflow Run ID: A unique identifier for the MATs workflow run. ii. Conclusion Statuses: The outcomes of the Minimal Acceptance Tests. B. MATs-Summary-Artifact.txt (MATS-COMMENT-BODY-{{ build_sha }}): A summary of the Minimal Acceptance Tests conducted.

  5. CI-Tests.yml: A configuration file for executing tests, which processes the build artifacts to create: A. Coverage reports: Uploading various coverage reports for the tests executed to multiple services for analysis (e.g., codecov.io, coveralls.io, app.deepsource.io, etc.). B. Test-Results-Artifacts ({{ test-group }}-Test-Report-{{ matrix.os }}-{{ matrix.python-version }}): Contains the results of the tests grouped by coverage|doctests|integration, os and python-version. C. Integration-Summary-Artifact.txt (INTEGRATION-COMMENT-BODY-{{ build_sha }}): A summary of the test results.

  6. CI-DOCs.yml: A configuration file for generating documentation, which produces: A. Documentation-Artifact.zip (Multicast-Documentation-{{ build_sha }}-ALL): A zip file containing the generated documentation. B. DOCUMENTATION-Summary-Artifact.txt (DOCUMENTATION-COMMENT-BODY-{{ build_sha }}): A summary of the documentation results.

In summary, as the diagram illustrates, a GitHub Actions CI/CD workflow begins with a push event, leading to the building of artifacts, the execution of Minimal Acceptance Tests, and the generation of test reports. The workflow includes passing detailed information about the build process, such as the build run ID, artifact details, and associated git commit information, as well as comprehensive details about the Minimal Acceptance Tests, including the MATs workflow run ID and conclusion statuses, ensuring thorough traceability and accountability throughout the CI/CD pipeline.

Acceptance Testing with CircleCI

While the comprehensive results from the extensive GHA pipeline offer a detailed look at the state of the codebase, the process can take upwards of 30 minutes to complete. The Multicast Project also utilizes the much faster CircleCI offering to provide pass or fail status much earlier in CI/CD. While the underlying tests are the same for both GHA and CircleCI, they are only tested in a single environment on CircleCI, and typically perform faster.

How push triggers propagate on CircleCI

        %%{ init: { 'theme': 'base', 'themeVariables': { 'darkMode': true, 'primaryColor': '#3e3e3e', 'background': 'transparent', 'primaryBorderColor': 'Orange', 'lineColor': 'darkOrange', 'secondaryColor': 'transparent', 'tertiaryColor': '#030303' }} }%%
graph TD;
  PUSH-EVENT-->config.yml;
  config.yml-->build;
  config.yml-->test;
  config.yml-->lint;
  config.yml-->pytest;
  build-->lint;
  build-->test;
  test-->pytest;

  build-->GH-Checks;
  test-->GH-Checks;
  lint-->GH-Checks;
  pytest-->GH-Checks;
  pytest-->Test-Results;
  Test-Results-->CircleCI-Metrics;

  1. PUSH-EVENT: This event triggers the entire Acceptance Testing CI/CD workflow.

  2. config.yml: The sole configuration file for the CircleCI jobs: A. build: Tests that the build process works without critical error, (albeit these quick builds are ephemeral and not attested) B. test: Tests that the Minimal Acceptance tests pass without failure, (albeit the test details are discarded, only the logs remain for a while on CircleCI) C. lint: Selectively lints (See Linting for details) the multicast python source (e.g., multicast/*.py), failing on any linter flagged issues or passing on none. D. pytest: Runs the now deprecated make test-pytest target to discover, and then run, unittests via the pytest testing framework. i. Test-Results: the produced test results. See Collect Tests with CircleCI for more.

  3. GH-Checks: Each CI/CD job will report back a GitHub Check run result.

In summary, as the diagram illustrates, a CircleCI pipeline CI/CD workflow begins with a push event, leading to the build, test, lint, and pytest jobs reporting back to GitHub Checks, indicating the status of each job. Additionally, the output from pytest generates some Test Results, which are then used to produce CircleCI Metrics.

How Integrations are triggered from Testing in CI/CD

Key integrations

There are many integrations with various service providers used in the Multicast project’s CI/CD pipeline.

        %%{ init: { 'theme': 'base', 'themeVariables': { 'darkMode': true, 'primaryColor': '#3e3e3e', 'background': 'transparent', 'primaryBorderColor': 'Orange', 'lineColor': 'darkOrange', 'secondaryColor': 'transparent', 'tertiaryColor': '#030303' }} }%%
graph TD;
  PUSH-EVENT["PUSH EVENT"];
  PUSH-EVENT-->CI-Build.yml;
  PUSH-EVENT-->config.yml;

  subgraph GHA
  multicast-build-*@{ shape: card };
  BUILD-info.txt@{ shape: card };
  Build-Summary-Artifact.txt@{ shape: card };
  CI-Build.yml-->multicast-build-* & BUILD-info.txt & Build-Summary-Artifact.txt;
  multicast-build-*-->CI-MATs.yml;
  multicast-build-*-->CI-CHGLOG.yml;
  BUILD-info.txt-->CI-MATs.yml;
  BUILD-info.txt-->CI-CHGLOG.yml;
  CHANGELOG.md@{ shape: card };
  chglog-info.txt@{ shape: card };
  chglog-Summary-Artifact.txt@{ shape: card };
  CI-CHGLOG.yml-->CHANGELOG.md;
  CI-CHGLOG.yml-->chglog-info.txt;
  CI-CHGLOG.yml-->chglog-Summary-Artifact.txt;
  multicast-info-*@{ shape: card };
  MATs-Summary-Artifact.txt@{ shape: card };
  CI-MATs.yml-->multicast-info-* & MATs-Summary-Artifact.txt;
  multicast-build-*-->CI-Tests.yml & CI-DOCS.yml;
  multicast-info-*-->CI-Tests.yml & CI-DOCS.yml;
  DOCUMENTATION-Summary-Artifact.txt@{ shape: card };
  Integration-Summary-Artifact.txt@{ shape: card };
  Multicast-Documentation-*-ALL@{ shape: card };
  \*-Test-Report-\*-*@{ shape: card };
  CI-Tests.yml-->Integration-Summary-Artifact.txt;
  CI-DOCS.yml-->DOCUMENTATION-Summary-Artifact.txt & Multicast-Documentation-*-ALL;
  CI-Tests.yml-->\*-Test-Report-\*-* & COVERAGE;
  PUSH-EVENT-->markdown-lint.yml & yaml-lint.yml;
  PUSH-EVENT-->bandit.yml & codeql-analysis.yml & flake8.yml & makefile-lint.yml;
  PUSH-EVENT-->shellcheck.yml;
  end

  subgraph "Github Attestations"
  multicast-build-*-->Attestations;
  end

  subgraph "Github Commit Comments"
  Build-Summary-Artifact.txt-->Comments;
  chglog-Summary-Artifact.txt-->Comments;
  MATs-Summary-Artifact.txt-->Comments;
  Integration-Summary-Artifact.txt-->Comments;
  DOCUMENTATION-Summary-Artifact.txt-->Comments;
  end

  subgraph "3rd-Party Integrations"

  subgraph "CodeCov service"
  CodeCov-->CodeCov-Metrics;
  end

  subgraph "DeepSource service"
  DeepSource-->DeepSource-Metrics;
  end

  subgraph "Coveralls service"
  Coveralls-->Coveralls-Metrics;
  end

  COVERAGE-->CodeCov & DeepSource & Coveralls;

  subgraph "GHA Marketplace"
  bandit.yml--oPython-Bandit-Scan;
  codeql-analysis.yml--ocodeql-action;
  flake8.yml--oflake8-cq;

  shellcheck.yml--oshellcheck-scan;
  end

  end

  subgraph "Github Checks"
  CI-Build.yml-->GH-Checks;
  CI-MATs.yml-->GH-Checks;
  CI-Tests.yml-->GH-Checks;
  CI-DOCS.yml-->GH-Checks;
  markdown-lint.yml-->GH-Checks;
  makefile-lint.yml-->GH-Checks;
  yaml-lint.yml-->GH-Checks;
  bandit.yml-->GH-Checks;
  codeql-analysis.yml-->GH-Checks;
  flake8.yml-->GH-Checks;
  shellcheck.yml-->GH-Checks;

  CodeCov-->GH-Checks;
  DeepSource-->GH-Checks;
  end

  subgraph "GitHub Code Scanning"
  Python-Bandit-Scan-->GH-Code-Scanning;
  codeql-action-->GH-Code-Scanning;
  flake8-cq-->GH-Code-Scanning;
  shellcheck-scan-->GH-Code-Scanning;
  end

  subgraph CircleCI
  config.yml-->build;
  config.yml-->test;
  config.yml-->lint;
  config.yml-->pytest;
  build-->lint;
  build-->test;
  test-->pytest;

  build-->GH-Checks;
  test-->GH-Checks;
  lint-->GH-Checks;
  pytest-->GH-Checks;
  pytest-->Test-Results;

  Test-Results-->CircleCI-Metrics;
  end

  subgraph PR
  PRC["GitHub Pull-Request Comments"]
  GH-Checks-->PRC;
  GH-Code-Scanning-->PRC;
  CodeCov-->PRC;
  DeepSource-->PRC;
  end

Linting in CI/CD

TL;DR Context of Linting

In the constantly evolving ecosystem of software development, where code quality and maintainability matter, the role of linters has become increasingly common practice. So it should be no surprise that various linters are used in the Multicast project’s CI/CD workflows. By incorporating the linting directly into the CI/CD workflows, this automation alleviates the load on developers to manually check much of the code style and formatting across various languages, including Python, YAML, Makefile, Bash, and Markdown.

Some of the Multicast project styles and conventions are quite specific (e.g., custom locking conventions of CEP-5), and not yet automated. However, by leveraging linters, we not only ensure a level of maintainability but also foster a collaborative environment where developers can focus on writing effective code rather than getting bogged down by stylistic concerns.

Linting Design Overview

Most of the linting in CI/CD is performed by GitHub workflows with the exception of a minimal Flake-8 scan for python source-code performed via CircleCI by the aptly named Lint job. All of the linter reporting (e.g., anything more than pass/fail status) is from the GitHub linter workflows. There are two noteworthy forms of feedback from the various linting automation, code-scanning reports, and GitHub Annotated Messages. All linting workflows failures result in alerts that can be reviewed with the relevant PR via GitHub’s UI, albeit with different levels of details per linter workflow and target branch.

Linting phases

The generalized design of linter workflows in the Multicast project CI/CD pipeline follows these same phases:

  • Initialization - Bootstraps environment and any initial setup automatically

  • Clone - git clone the Multicast Git Repository and any submodules needed

  • Linting - performs the actual linting on the resulting clone

  • Post-processing (optional) - any post-processing of the resulting linting results

  • Reporting - report any results and/or linting status

[!IMPORTANT] This overview does not address the complexities of CI/CD timing, concurrency, or the various combinations of linter workflows that run simultaneously. Each linter workflow is separately triggered (e.g., push versus PR, etc.) and thus logically disjoint (See CI/CD Triggering for details). Each linter workflow varies in its exact implementation of the afore mentioned phases.

Logically (e.g., ignoring complexities of concurrency and trigger conditionals, etc.) the order of phases is sequential per single CI/CD linter workflow.

        %%{ init: { 'theme': 'base', 'themeVariables': { 'darkMode': true, 'primaryColor': '#3e3e3e', 'background': 'transparent', 'primaryBorderColor': 'Orange', 'lineColor': 'darkOrange', 'secondaryColor': 'transparent', 'tertiaryColor': '#030303' }} }%%
sequenceDiagram
  participant Repo as Repository

  create participant GA as GitHub Actions
  Repo->>GA: Trigger
  GA-->>GA: Initialization
  GA->>GA: Clone
  create participant SC as Linter
  GA->>SC: Linting
  opt: Optional
    SC->>SC: Post-processing
  end
  destroy SC
  SC-->>GA: Scan Results
  destroy GA
  GA->>Repo: Reporting

Linting Workflows

Bandit

Property

Value

Linter

Python-Bandit-Scan

Workflow

bandit.yml

Language

Python

Category

Security Linter

Badge

Bandit
CodeQL

Property

Value

Linter

GitHub’s CodeQL Analysis

Workflow

codeql-analysis.yml

Languages

Python, Javascript (github actions)

Category

Security Linter

Badge

CodeQL
Flake-8

Property

Value

Linter

flake8-cq

Workflow

flake8.yml

Language

Python

Category

Style Linter

Badge

Flake8 Analysis
Checkmake

Property

Value

Linter

checkmake

Workflow

makefile-lint.yml

Language

Makefile

Category

Style Linter

Badge

Makefile Lint
MarkdownLintCLI

Property

Value

Linter

MarkdownLint

Workflow

markdown-lint.yml

Language

Markdown

Category

Format Linter

Badge

Markdown Lint
Shellcheck-Scan

[!CAUTION] Shellcheck-Scan is intended ONLY for use in CI on GitHub Actions, because it is under different restrictions, please see its License for details.

Property

Value

Linter

Shellcheck-Scan

Workflow

shellcheck.yml

Languages

Bash, sh

Category

Format Linter

Badge

shellcheck
YAMLLint

Property

Value

Linter

YAMLLint

Workflow

yaml-lint.yml

Languages

YAML

Category

Format Linter

Badge

YAML Lint

How linting is triggered in CI/CD

Logically (e.g., ignoring complexities of concurrency and trigger conditionals, etc.) the various linters are all run independently per CI/CD linter workflow.

        %%{ init: { 'theme': 'base', 'themeVariables': { 'darkMode': true, 'primaryColor': '#3e3e3e', 'background': 'transparent', 'primaryBorderColor': 'Orange', 'lineColor': 'darkOrange', 'secondaryColor': 'transparent', 'tertiaryColor': '#030303' }} }%%
graph TD;
    PUSH-EVENT-->bandit.yml;
    PUSH-EVENT-->codeql-analysis.yml;
    PUSH-EVENT-->flake8.yml;
    PUSH-EVENT-->makefile-lint.yml;
    PUSH-EVENT-->shellcheck.yml;
    PUSH-EVENT-->markdown-lint.yml;
    PUSH-EVENT-->yaml-lint.yml;
    flake8.yml--oflake8-cq;
    shellcheck.yml--oshellcheck-scan;
    bandit.yml-->Python-Bandit-Scan;
    codeql-analysis.yml-->codeql-action;
    flake8-cq-->GH-Code-Scanning;
    shellcheck-scan-->GH-Code-Scanning;
    Python-Bandit-Scan-->GH-Code-Scanning;
    codeql-action-->GH-Code-Scanning;
    makefile-lint.yml;

Configurable CI Variables

This section documents environment variables used across CI workflows to ensure consistency and simplify maintenance.

Python Version Variables

To standardize Python version management across all CI workflows, we use a set of environment variables defined at the top of each workflow file. This approach centralizes version definitions while maintaining separation between workflows, making future updates easier and ensuring consistency.

Standard Python Version Variables

Variable

Purpose

Example Value

PYTHON_DEFAULT

The default Python version used for single-version jobs

"3.12"

PYTHON_OLD_MIN

Oldest/minimum Python version for cross-python portability testing

"3.10"

PYTHON_OLD_EXTRA

Additional Python version for coverage testing

"3.11"

PYTHON_EXPERIMENTAL

Future/experimental Python version for optional testing

"3.13"

Usage Examples

Setting up Python with the default version
- uses: actions/setup-python@v6
  with:
    python-version: "${{ vars.PYTHON_DEFAULT }}"
Using matrix strategy for multi-version testing
jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        python-version: ["${{ vars.PYTHON_DEFAULT }}", "${{ vars.PYTHON_EXPERIMENTAL }}"]
    steps:
      - uses: actions/setup-python@v6
        with:
          python-version: ${{ matrix.python-version }}