Skip to main content

Generic JSON

Generic JSON Example Workflow

Complete GitHub Actions workflows using DxCore with the generic JSON adapter.

Minimal Example

The simplest way to try the generic adapter. Create a graph.json:

{
"tasks": [
{
"taskId": "setup",
"package": "infra",
"task": "setup",
"command": "echo Setting up environment",
"dependencies": []
},
{
"taskId": "check-a",
"package": "service-a",
"task": "check",
"command": "echo Checking service A",
"dependencies": ["setup"]
},
{
"taskId": "check-b",
"package": "service-b",
"task": "check",
"command": "echo Checking service B",
"dependencies": ["setup"]
}
]
}

Dispatch it:

cat graph.json | dxcore dispatch \
-c $COORDINATOR_URL -s $SESSION_ID -t $TOKEN -b generic

DxCore runs setup first, then check-a and check-b in parallel since they share no dependencies between them.

Realistic Example: Make Project

This workflow demonstrates using the generic adapter to parallelize a C project built with GNU Make. Make has no built-in JSON export, so we use a parser script that extracts the dependency graph from make -pnr output.

Step 1: Example Makefile

A multi-target C project with a library, application, and tests:

CC = gcc
CFLAGS = -Wall -Wextra
# Library
lib/libutils.a: lib/utils.o
ar rcs $@ $^
lib/utils.o: lib/utils.c lib/utils.h
$(CC) $(CFLAGS) -c -o $@ $<
# Application
app/main: app/main.o lib/libutils.a
$(CC) $(CFLAGS) -o $@ app/main.o -Llib -lutils
app/main.o: app/main.c lib/utils.h
$(CC) $(CFLAGS) -Ilib -c -o $@ $<
# Tests
test/run_tests: test/test_utils.o lib/libutils.a
$(CC) $(CFLAGS) -o $@ test/test_utils.o -Llib -lutils
test/test_utils.o: test/test_utils.c lib/utils.h
$(CC) $(CFLAGS) -Ilib -c -o $@ $<
.PHONY: all test clean
all: app/main test/run_tests
test: test/run_tests
./test/run_tests
clean:
rm -f lib/*.o lib/*.a app/*.o app/main test/*.o test/run_tests

Step 2: Parser script

make -pnr dumps Make’s internal database without executing anything. The section between # Files and # Finished Make data base contains all targets with their prerequisites and recipe commands. This script parses that section and outputs DxCore-compatible JSON:

#!/usr/bin/env bash
# make2dxcore.sh — Convert Make dependency graph to DxCore generic JSON
set -euo pipefail
# Dump Make's database without built-in rules or execution
DB=$(make -pnr 2>/dev/null)
# Extract the section between "# Files" and "# Finished Make data base"
FILES_SECTION=$(echo "$DB" | sed -n '/^# Files$/,/^# Finished Make data base$/p')
# Two-pass: collect all targets first, then emit JSON with deps filtered
# to only reference other tasks (source files like *.c and *.h are not tasks).
# Uses the directory as the package name (e.g., lib/utils.o → package "lib").
echo "$FILES_SECTION" | awk '
BEGIN { skip_next = 0; ntargets = 0 }
/^# Not a target:/ { skip_next = 1; next }
/^[^\t#][^:]*:/ {
if (skip_next) { skip_next = 0; next }
if (in_target && command != "") {
targets[ntargets] = target
commands[ntargets] = command
raw_deps[ntargets] = deps
ntargets++
}
in_target = 1; command = ""; deps = ""
split($0, parts, ":"); target = parts[1]
gsub(/^[ \t]+|[ \t]+$/, "", target)
dep_str = parts[2]; gsub(/^[ \t]+|[ \t]+$/, "", dep_str)
n = split(dep_str, dep_arr, " ")
for (i = 1; i <= n; i++) {
if (dep_arr[i] != "" && dep_arr[i] !~ /^\|/) {
if (deps != "") deps = deps "|"
deps = deps dep_arr[i]
}
}
}
/^\t[^#]/ && in_target { command = "make " target }
END {
if (in_target && command != "") {
targets[ntargets] = target
commands[ntargets] = command
raw_deps[ntargets] = deps
ntargets++
}
# Build set of valid task IDs
for (i = 0; i < ntargets; i++) valid[targets[i]] = 1
print "{\"tasks\":["
first = 1
for (i = 0; i < ntargets; i++) {
if (!first) printf ","
first = 0
# Derive package from directory (e.g., "lib/utils.o" → "lib")
pkg = targets[i]
if (index(pkg, "/") > 0) {
pkg = substr(pkg, 1, index(pkg, "/") - 1)
}
# Derive task from filename (e.g., "lib/utils.o" → "utils.o")
tsk = targets[i]
if (index(tsk, "/") > 0) {
tsk = substr(tsk, index(tsk, "/") + 1)
}
# Filter deps to only valid task IDs
filtered = ""
n = split(raw_deps[i], darr, "|")
for (j = 1; j <= n; j++) {
if (darr[j] in valid) {
if (filtered != "") filtered = filtered ","
filtered = filtered "\"" darr[j] "\""
}
}
printf "{\"taskId\":\"%s\",\"package\":\"%s\",\"task\":\"%s\",\"command\":\"%s\",\"dependencies\":[%s]}\n",
targets[i], pkg, tsk, commands[i], filtered
}
print "]}"
}
'

Note

make -pnr uses three flags: -p prints the database, -n prevents execution, and -r suppresses built-in implicit rules. The output format is stable and documented in the GNU Make manual.

Step 3: GitHub Actions workflow

name: CI with DxCore (Make)
on: [push, pull_request]
env:
DXCORE_URL: ${{ secrets.DXCORE_URL }}
DXCORE_TOKEN: ${{ secrets.DXCORE_TOKEN }}
jobs:
session:
runs-on: ubuntu-latest
outputs:
session_id: ${{ steps.create.outputs.session_id }}
steps:
- name: Create DxCore session
id: create
run: |
SESSION=$(dxcore ci create-session -c $DXCORE_URL -t $DXCORE_TOKEN)
echo "session_id=$SESSION" >> "$GITHUB_OUTPUT"
- name: Wait for coordinator
run: dxcore ci wait -c $DXCORE_URL --timeout 300
dispatch:
needs: session
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Dispatch graph
run: |
chmod +x ./make2dxcore.sh
./make2dxcore.sh | dxcore dispatch \
-c $DXCORE_URL -s ${{ needs.session.outputs.session_id }} \
-t $DXCORE_TOKEN -b generic
agent:
needs: session
runs-on: ubuntu-latest
strategy:
matrix:
shard: [0, 1, 2]
steps:
- uses: actions/checkout@v4
- name: Run DxCore agent
run: |
dxcore agent \
-c $DXCORE_URL \
-a agent-${{ matrix.shard }} \
-s ${{ needs.session.outputs.session_id }} \
-t $DXCORE_TOKEN \
-b generic
finish:
needs: [session, dispatch, agent]
if: always()
runs-on: ubuntu-latest
steps:
- name: Finish session
run: |
dxcore ci finish \
-c $DXCORE_URL \
-s ${{ needs.session.outputs.session_id }} \
-t $DXCORE_TOKEN

What Happens

  1. session — Creates a coordinator session and waits for readiness
  2. dispatch — Runs the parser script to extract Make’s dependency graph, pipes it to the coordinator
  3. agent (x3) — Each agent connects, receives tasks, and runs make <target> for each assigned task
  4. finish — Marks the session complete regardless of outcome

Note

The dispatch and agent jobs run concurrently. Agents will wait for tasks once connected — they do not need the graph to be submitted first.

Adapting to Other Build Tools

The Make example demonstrates a pattern that works for any build system:

  1. Extract your build tool’s task graph (using its API, CLI, or a custom script)
  2. Map each task to the generic JSON format (taskId, package, task, command, dependencies)
  3. Pipe the JSON to dxcore dispatch -b generic

This works for Make, Bazel, Pants, Buck, or any custom pipeline — as long as you can express the work as a DAG with shell commands.

Tip

Some build systems now have dedicated DxCore adapters. Check the Getting Started page for the full list — for example, Gradle has its own first-class adapter (-b gradle) with support for cacheable task annotations.

Next Steps