Long Discussion

Governance Problem

Governance in DSLCore sits at the boundary between structure and execution. This note explores how we think about it, how we intend to build it, and what we are deliberately avoiding.

Two governance opportunities

There are two distinct places where governance can operate.

Design-time governance

This sits over the DSL/YAML definitions and runs before generation or deployment.

It checks things like:

  • required metadata present
  • field classifications declared
  • allowed state transitions defined
  • whether denormalised fields are declared as derived or manually maintained
  • whether JSON fields are unconstrained, schema-constrained, or policy-constrained
  • audit requirements for certain entities or actions

Design-time governance catches structural problems early, before they become runtime problems.

Runtime governance

This is the stronger opportunity for DSLCore.

DSLCore already holds executable structure — entities, fields, workflows, forms, actions. That gives us a natural foundation for a policy and interceptor pipeline at these levels:

  • field-level checks — read-only rules, conditional visibility, required-on-state, classification, masking, referential and semantic checks
  • form-level checks — cross-field validation, role and state-dependent obligations, warnings, explanation messages
  • pre-commit checks — before create, update, delete, or state transition; can block, warn, require override, or escalate
  • post-commit checks — audit logging, denormalised field refresh, event emission, notification, anomaly checks
  • workflow and state checks — transition guards, separation of duties, approval path existence, mandatory attachments or evidence

Governance starts light, then tightens as the system is used.

Why governance should be an add-on

At first install, teams often think they need strict approvals everywhere, fully constrained fields, and every transition governed.

Real use reveals something different:

  • which checks are noise
  • which controls genuinely reduce risk
  • where overrides are common
  • where data quality actually drifts

The most important controls in mature systems and legacy migrations usually only become visible after watching real exceptions, bad data paths, workarounds, and operator behaviour.

Making governance an add-on means:

  • initial deployments stay fast and easier to deliver
  • governance can be introduced where risk actually justifies it
  • clients are not forced into enterprise-style overhead too early
  • rules can mature after real usage exposes where the true control points are

Keeping the base DSL clean

The base DSL should continue to answer only:

  • what entities exist
  • what fields and relationships exist
  • what forms and actions exist
  • what workflows and states exist

That is the operational structure layer.

Governance lives in a parallel metadata layer — not mixed into the first pass unless specifically needed. Structure first, controls second.

Governance metadata — a working example

entity: shipment

fields:
  customer_id:
    type: fk
  cargo_type:
    type: select
  dg_declaration:
    type: file
  status:
    type: workflow_state
  customer_name_cached:
    type: text
    denormalised:
      source: customer.name
      refresh_on: [create, update]

governance:
  rules:
    - id: shipment-customer-on-hold
      scope: action
      trigger: before_commit
      actions: [create, update]
      condition: "customer.credit_status == 'hold'"
      outcome: block
      message: 'Customer account is on hold.'

    - id: shipment-dg-doc-required
      scope: form
      trigger: validate
      actions: [create, update]
      condition: "cargo_type == 'DG' and dg_declaration is null"
      outcome: block
      message: 'DG declaration is required.'

    - id: shipment-status-release-check
      scope: transition
      trigger: before_transition
      from: draft
      to: released
      condition: 'all_required_docs_present == true'
      outcome: block
      message: 'Required documents must be present before release.'

    - id: shipment-customer-name-sync
      scope: field
      trigger: after_commit
      field: customer_name_cached
      condition: 'true'
      outcome: refresh
      source: 'customer.name'

  overrides:
    enabled: true
    allowed_roles: [admin, supervisor]
    require_reason: true

  audit:
    log_hits: true
    log_blocks: true
    log_overrides: true

This is enough to prove the concept without overbuilding.

Checkpoint model

Field-level

Use for:

  • required-if conditions
  • visibility and editability rules
  • masking and classification
  • derived and denormalised field rules
  • JSON key constraints

Examples:

  • field editable only in draft state
  • field hidden unless role is finance
  • JSON field must contain certain keys in approved state

Form-level

Use for:

  • cross-field checks
  • state-aware validation
  • warnings before submission
  • human-readable explanations

Examples:

  • if cargo_type = hazardous, attach docs and approval code
  • if payment_terms = prepaid, invoice reference required

Pre-commit

Use for:

  • create, update, delete approvals
  • workflow guardrails
  • separation of duties
  • policy blocks and warnings

Examples:

  • cannot approve an own submission
  • cannot transition to released until all mandatory checks pass

Post-commit

Use for:

  • audit records
  • denormalised field refreshes
  • event generation
  • anomaly checks
  • downstream sync

Examples:

  • refresh cached customer and port names
  • emit event to analytics ledger
  • run exception scan

Separating rule intent from rule execution

A rule should declare what it is trying to do:

  • block
  • warn
  • require approval
  • refresh derived data
  • notify
  • audit only

A policy engine then decides how to apply that at runtime.

That separation makes it possible to tighten controls after go-live without rewriting the application.

Starting rule classes

Rather than building a full expression language from the start, we begin with four classes:

Validation rules

Field and form checks.

  • mandatory when status changes
  • cross-field consistency
  • JSON key required in approved state

Transition rules

Workflow movement.

  • cannot approve an own submission
  • cannot close until all exceptions are resolved

Derivation rules

Denormalised fields and sync.

  • refresh cached customer name
  • maintain summary totals
  • stamp last-reviewed date

Audit and escalation rules

Traceability and governance hardening.

  • log every override
  • notify supervisor when a blocked rule repeats three times
  • record missing evidence attachment

Denormalised fields as a governance opportunity

Many systems use denormalised fields informally. DSLCore makes them explicit by declaring:

  • source field and path
  • refresh trigger
  • editable or not
  • overrideable or not
  • stale tolerance
  • reconciliation policy

Instead of “we cache this for performance,” we get:

  • why it exists
  • when it refreshes
  • what happens if it drifts
  • who may override it

That is valuable in mature systems and legacy refactors.

JSON fields governed by state, not just schema

A binary “allowed or not” for JSON fields misses the point. A more useful approach:

  • draft state: flexible
  • submitted state: minimum key set required
  • approved state: immutable except for privileged override
  • critical keys: typed and validated
  • non-critical keys: freeform

That preserves agility early while still allowing governance to tighten later.

The explanation engine

A governance layer that only says “save failed” is not useful.

A useful governance layer says:

  • what failed
  • why it failed
  • which field, state, or action triggered it
  • what needs to happen next
  • whether an override is available

That makes governance feel operational rather than bureaucratic.

Where DuckDB fits

DuckDB is more useful on the observability and tuning side than as an inline enforcement mechanism.

Good uses:

  • rule hit analysis
  • exception frequency
  • override hotspots
  • process bottlenecks
  • comparing before and after governance changes
  • which forms generate the most friction
  • which rules are mostly noise
  • where operators are repeatedly overriding controls
  • which denormalised fields drift most often

Where graph fits

Graph becomes useful when governance depends on traversing a path rather than checking a single row.

Examples:

  • actor → role → approval authority → entity type
  • customer → contract → rate agreement → shipment
  • site → regulation → asset → maintenance obligation
  • user → team → delegated authority → override permission

If those checks stay shallow, relational is enough. If dynamic traversal across variable relationships is needed, graph becomes more compelling. This is likely a phase 3 concern.

Where vector fits

Lowest priority, but potentially useful for:

  • policy lookup and explanation retrieval
  • linking failed actions to relevant guidance text
  • searching exception narratives or supporting evidence

A blocked rule could surface not just the failure, but the most relevant internal guidance note. Useful, but not the first thing to build.

Architecture components

A. Governance schema

A YAML/DSL extension defining scope, trigger, condition, severity, message, override rules, and audit behaviour.

B. Policy engine

A standard interceptor pipeline:

  • before_render
  • before_validate
  • before_commit
  • after_commit
  • before_transition

C. Explanation engine

Not just the block message — the full context of what failed, why, what resolves it, and whether override is available.

D. Audit and overrides

Capture rule hits, who overrode, why, when, and what changed.

E. Governance dashboard

Show top triggered rules, blocked operations, repeated override hotspots, entities with poor data quality, and forms with the highest friction.

Implementation phases

Phase 1 — lightweight controls

  • field, form, and action checkpoints
  • warn and block outcomes
  • audit trail
  • override with reason

Phase 2 — workflow governance

  • state transition rules
  • approval requirements
  • separation of duties
  • evidence and attachment requirements

Phase 3 — governance analytics

  • DuckDB reporting layer
  • rule hit dashboards
  • friction analysis
  • tuning recommendations

Phase 4 — advanced traversal and semantic support

  • graph-backed relationship checks where justified
  • vector-backed policy and guidance retrieval where useful

What to avoid early

  • a full custom expression language
  • mandatory governance for every entity
  • graph database dependency from the start
  • mixing governance deeply into the base generator
  • enforcing enterprise-level control before clients have asked for it

Keeping the first version small and operational is more important than making it complete.

Summary

DSLCore’s governance module is not a compliance layer bolted on top. It is an optional execution layer tied directly to the DSL model — activated where risk justifies it, tightened after the system has been used long enough to reveal where the real control points are.

The distinction that matters:

  • base DSL — operational structure
  • governance add-on — executable control layer
  • analytics add-on — governance tuning and friction discovery

That keeps the initial implementation lean while creating room to grow into stronger governance as real usage demands it.