Generic JSON
Generic JSON Example Workflow
Complete GitHub Actions workflows using DxCore with the generic JSON adapter.
Minimal Example
The simplest way to try the generic adapter. Create a graph.json:
{
"tasks": [
{
"taskId": "setup",
"package": "infra",
"task": "setup",
"command": "echo Setting up environment",
"dependencies": []
},
{
"taskId": "check-a",
"package": "service-a",
"task": "check",
"command": "echo Checking service A",
"dependencies": ["setup"]
},
{
"taskId": "check-b",
"package": "service-b",
"task": "check",
"command": "echo Checking service B",
"dependencies": ["setup"]
}
]
}
Dispatch it:
cat graph.json | dxcore dispatch \
-c $COORDINATOR_URL -s $SESSION_ID -t $TOKEN -b generic
DxCore runs setup first, then check-a and check-b in parallel since they share no dependencies between them.
Realistic Example: Make Project
This workflow demonstrates using the generic adapter to parallelize a C project built with GNU Make. Make has no built-in JSON export, so we use a parser script that extracts the dependency graph from make -pnr output.
Step 1: Example Makefile
A multi-target C project with a library, application, and tests:
CC = gcc
CFLAGS = -Wall -Wextra
# Library
lib/libutils.a: lib/utils.o
ar rcs $@ $^
lib/utils.o: lib/utils.c lib/utils.h
$(CC) $(CFLAGS) -c -o $@ $<
# Application
app/main: app/main.o lib/libutils.a
$(CC) $(CFLAGS) -o $@ app/main.o -Llib -lutils
app/main.o: app/main.c lib/utils.h
$(CC) $(CFLAGS) -Ilib -c -o $@ $<
# Tests
test/run_tests: test/test_utils.o lib/libutils.a
$(CC) $(CFLAGS) -o $@ test/test_utils.o -Llib -lutils
test/test_utils.o: test/test_utils.c lib/utils.h
$(CC) $(CFLAGS) -Ilib -c -o $@ $<
.PHONY: all test clean
all: app/main test/run_tests
test: test/run_tests
./test/run_tests
clean:
rm -f lib/*.o lib/*.a app/*.o app/main test/*.o test/run_tests
Step 2: Parser script
make -pnr dumps Make’s internal database without executing anything. The section between # Files and # Finished Make data base contains all targets with their prerequisites and recipe commands. This script parses that section and outputs DxCore-compatible JSON:
#!/usr/bin/env bash
# make2dxcore.sh — Convert Make dependency graph to DxCore generic JSON
set -euo pipefail
# Dump Make's database without built-in rules or execution
DB=$(make -pnr 2>/dev/null)
# Extract the section between "# Files" and "# Finished Make data base"
FILES_SECTION=$(echo "$DB" | sed -n '/^# Files$/,/^# Finished Make data base$/p')
# Two-pass: collect all targets first, then emit JSON with deps filtered
# to only reference other tasks (source files like *.c and *.h are not tasks).
# Uses the directory as the package name (e.g., lib/utils.o → package "lib").
echo "$FILES_SECTION" | awk '
BEGIN { skip_next = 0; ntargets = 0 }
/^# Not a target:/ { skip_next = 1; next }
/^[^\t#][^:]*:/ {
if (skip_next) { skip_next = 0; next }
if (in_target && command != "") {
targets[ntargets] = target
commands[ntargets] = command
raw_deps[ntargets] = deps
ntargets++
}
in_target = 1; command = ""; deps = ""
split($0, parts, ":"); target = parts[1]
gsub(/^[ \t]+|[ \t]+$/, "", target)
dep_str = parts[2]; gsub(/^[ \t]+|[ \t]+$/, "", dep_str)
n = split(dep_str, dep_arr, " ")
for (i = 1; i <= n; i++) {
if (dep_arr[i] != "" && dep_arr[i] !~ /^\|/) {
if (deps != "") deps = deps "|"
deps = deps dep_arr[i]
}
}
}
/^\t[^#]/ && in_target { command = "make " target }
END {
if (in_target && command != "") {
targets[ntargets] = target
commands[ntargets] = command
raw_deps[ntargets] = deps
ntargets++
}
# Build set of valid task IDs
for (i = 0; i < ntargets; i++) valid[targets[i]] = 1
print "{\"tasks\":["
first = 1
for (i = 0; i < ntargets; i++) {
if (!first) printf ","
first = 0
# Derive package from directory (e.g., "lib/utils.o" → "lib")
pkg = targets[i]
if (index(pkg, "/") > 0) {
pkg = substr(pkg, 1, index(pkg, "/") - 1)
}
# Derive task from filename (e.g., "lib/utils.o" → "utils.o")
tsk = targets[i]
if (index(tsk, "/") > 0) {
tsk = substr(tsk, index(tsk, "/") + 1)
}
# Filter deps to only valid task IDs
filtered = ""
n = split(raw_deps[i], darr, "|")
for (j = 1; j <= n; j++) {
if (darr[j] in valid) {
if (filtered != "") filtered = filtered ","
filtered = filtered "\"" darr[j] "\""
}
}
printf "{\"taskId\":\"%s\",\"package\":\"%s\",\"task\":\"%s\",\"command\":\"%s\",\"dependencies\":[%s]}\n",
targets[i], pkg, tsk, commands[i], filtered
}
print "]}"
}
'
Note
make -pnr uses three flags: -p prints the database, -n prevents execution, and -r suppresses built-in implicit rules. The output format is stable and documented in the GNU Make manual.
Step 3: GitHub Actions workflow
name: CI with DxCore (Make)
on: [push, pull_request]
env:
DXCORE_URL: ${{ secrets.DXCORE_URL }}
DXCORE_TOKEN: ${{ secrets.DXCORE_TOKEN }}
jobs:
session:
runs-on: ubuntu-latest
outputs:
session_id: ${{ steps.create.outputs.session_id }}
steps:
- name: Create DxCore session
id: create
run: |
SESSION=$(dxcore ci create-session -c $DXCORE_URL -t $DXCORE_TOKEN)
echo "session_id=$SESSION" >> "$GITHUB_OUTPUT"
- name: Wait for coordinator
run: dxcore ci wait -c $DXCORE_URL --timeout 300
dispatch:
needs: session
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Dispatch graph
run: |
chmod +x ./make2dxcore.sh
./make2dxcore.sh | dxcore dispatch \
-c $DXCORE_URL -s ${{ needs.session.outputs.session_id }} \
-t $DXCORE_TOKEN -b generic
agent:
needs: session
runs-on: ubuntu-latest
strategy:
matrix:
shard: [0, 1, 2]
steps:
- uses: actions/checkout@v4
- name: Run DxCore agent
run: |
dxcore agent \
-c $DXCORE_URL \
-a agent-${{ matrix.shard }} \
-s ${{ needs.session.outputs.session_id }} \
-t $DXCORE_TOKEN \
-b generic
finish:
needs: [session, dispatch, agent]
if: always()
runs-on: ubuntu-latest
steps:
- name: Finish session
run: |
dxcore ci finish \
-c $DXCORE_URL \
-s ${{ needs.session.outputs.session_id }} \
-t $DXCORE_TOKEN
What Happens
- session — Creates a coordinator session and waits for readiness
- dispatch — Runs the parser script to extract Make’s dependency graph, pipes it to the coordinator
- agent (x3) — Each agent connects, receives tasks, and runs
make <target>for each assigned task - finish — Marks the session complete regardless of outcome
Note
The dispatch and agent jobs run concurrently. Agents will wait for tasks once connected — they do not need the graph to be submitted first.
Adapting to Other Build Tools
The Make example demonstrates a pattern that works for any build system:
- Extract your build tool’s task graph (using its API, CLI, or a custom script)
- Map each task to the generic JSON format (
taskId,package,task,command,dependencies) - Pipe the JSON to
dxcore dispatch -b generic
This works for Make, Bazel, Pants, Buck, or any custom pipeline — as long as you can express the work as a DAG with shell commands.
Tip
Some build systems now have dedicated DxCore adapters. Check the Getting Started page for the full list — for example, Gradle has its own first-class adapter (-b gradle) with support for cacheable task annotations.