HyperPBI data model

HyperPBI receives a host-neutral normalized data workspace, augments its default source schema with validated root calculated-field metadata, applies those fields to rows at runtime, and evaluates named in-memory logical datasets. Power BI contributes one normalized visual data view. The Playground contributes normalized CSV and XLSX-sheet sources. A specification may also declare read-only MiniUp table, MiniUp Function, or web-only trusted REST GET sources; their JSON responses are normalized into the same workspace before components consume them. Dataset schemas therefore expose calculated fields to chained select/derive/group/metrics operations even when the current result has zero rows. Root scalar metrics remain separate from row fields. HyperPBI still does not execute SQL, arbitrary network URLs, mutations, custom request headers, or user code.

Base powerbi dataset

The built-in dataset name is powerbi. In Power BI it is the normalized visual data view and can carry selection identities. In the Playground it is an alias for the currently selected default source. In both hosts it contains normalized rows, stable row keys, field metadata, calculated aggregates, and normalized map bindings/warnings.

Normalized field metadata includes canonical key, display/query/qualified names, source table/column, type, kind, data type, format, Power BI roles, query aggregation, implicit-aggregation flag, origin, and selection/filter capability.

New 2.0 JSON uses Field Manifest aliases. Preparation resolves aliases in dataset definitions and component bindings to canonical keys before evaluation. A component without dataset uses powerbi.

MiniUp remote sources

data.sources declares named read-only sources. A source name can be used directly by any component through dataset, or as the source of a logical dataset.

miniup.table

{
  "data": {
    "sources": {
      "orders": {
        "type": "miniup.table",
        "site": "demo",
        "table": "orders",
        "params": { "status": "Open" },
        "maxRows": 1000
      }
    }
  }
}

Table reads use the fixed https://<site>.miniup.app/api/data/<site>/<table> surface and page in bounded requests. HyperPBI never accepts an arbitrary URL or request method for this source type.

rest.get (web only)

{
  "data": {
    "sources": {
      "publicApi": {
        "type": "rest.get",
        "baseUrl": "https://api.example.com",
        "path": "/v1/rows",
        "dataPath": "rows"
      }
    }
  }
}

rest.get accepts an HTTPS origin plus a safe relative path. It is available only when the origin matches the web build's trusted host list. Set HYPERPBI_WEB_REST_HOSTS to a comma-separated list of exact HTTPS origins or subdomain wildcard origins at web build time. When the variable is omitted, the trusted default is https://*.miniup.app. Dashboard JSON cannot expand this trust list.

miniup.function

{
  "data": {
    "sources": {
      "summary": {
        "type": "miniup.function",
        "function": "order-summary",
        "path": "/query",
        "params": {
          "region": { "state": "region", "default": "all" }
        },
        "dataPath": "rows"
      }
    }
  }
}

Function calls use the fixed https://functions.miniup.app origin. A parameter binding { "state": "region" } reads state.values.region; changing that state value changes the request signature and refreshes the source. Static scalar query parameters are also supported.

Remote sources enforce GET-only requests, omitted credentials, no referrer, JSON responses, a streamed 5 MB per-request response limit, a configurable timeout from 1 to 15 seconds, a maximum of 10,000 normalized rows, per-consumer cancellation, and an optional TTL cache. Cached completed responses may be shared, but in-flight requests are not shared across visual instances. Protected table keys or other secrets must stay server-side behind a MiniUp Function. They must not be embedded in HyperPBI JSON.

Remote source lifecycle is exposed in render context as loading, ready, empty, or error. Before the first response arrives, static preparation treats the declared source as a dynamic schema rather than weakening validation for Power BI or local-file datasets.

In Power BI, only miniup.function is portable. It uses the network-enabled package and https://functions.miniup.app; direct miniup.table and rest.get are web-only because Power BI's opaque sandbox cannot rely on the MiniUp table CORS contract and generic REST hosts are intentionally excluded from the PBIVIZ trust boundary. The Core package remains network-free. Remote rows cannot select, filter, or clear Power BI external state.

Named dataset definition

data.datasets maps a name to:

Dataset metric operations are sum, avg, min, max, count, distinctCount, and first. Every operation except count requires a field. sum and avg require a numeric or statically unknown field.

Filter operators are =, !=, >, >=, <, <=, contains, in, and between.

Exact operation order

The runtime evaluator and static schema propagation use the same field-stage order after the source is resolved:

1. filter
2. derive
3. rename
4. select
5. groupBy and metrics
6. distinct
7. sort
8. limit

This matters. A filter cannot reference a field first created by the same dataset's derive stage. A derive can use source/filter-stage fields. Rename changes keys before select/grouping. Grouping replaces the row schema with group fields plus metrics. Distinct and sort validate against that resulting schema.

Source chains apply a complete definition at each link. Thus a second dataset can filter or derive from fields produced by its source dataset.

Static schemas and validation

resolveDatasetSchemas always registers powerbi first, then resolves named datasets in sorted name order while recursively resolving dependencies.

At every operation stage it validates field references against fields currently available:

• filter fields exist in the source schema
• each derive expression validates field and valueFromRow references before adding its output
• rename sources exist; targets are nonblank and may not collide with another output
• select fields exist, then all unselected fields disappear
• group-by fields and metric fields exist
• a metric name may not collide with a group field
• distinct and sort fields exist in the grouped/selected output

Generated field type inference is static:

• arithmetic, rounding, date-part, and date-difference operators → number
• comparisons, Boolean operators, string predicates, and null tests → Boolean
• string operators → text
• today, now, and dateAdd → datetime
• coalesce, if, and case retain a type only when their known branches agree
• sum, avg, count, and distinctCount metrics → number
• min, max, and first inherit their input type

Derived fields have origin dataset-derived; metrics dataset-metric; group fields dataset-group. Rename copies all other source metadata, so a renamed direct model column still has its model origin and target. Select also retains metadata.

Sources, cycles, and unknown names

A source may be powerbi or another named dataset. Unknown source names produce UNKNOWN_DATASET diagnostics. Recursive resolution tracks the current chain; a repeated name produces DATASET_CYCLE with the cycle path. A failed/cyclic dataset is not made available to components, and a component selecting it receives an unknown-dataset diagnostic.

Runtime evaluation details

Rows begin as shallow copies of source rows with lineage arrays:

• filter removes rows/lineage together
• derive evaluates fields in object-entry order against the progressively updated row
• rename moves values and deletes old keys
• select copies only selected entries
• grouping canonicalizes group values, calculates metrics, and unions sorted unique lineage from every member
• distinct canonicalizes selected/all values, keeps the first row, and unions lineage of duplicates
• sort compares string forms with locale numeric ordering and preserves earlier order when all sort keys compare equal
• limit floors the nonnegative value and slices the final array; zero returns zero rows

count counts group rows without a field. Other metric implementations use the shared aggregate functions.

Zero-row behavior

Runtime type inference alone cannot describe an empty result, so each evaluated dataset receives the statically propagated schema. A filter or limit: 0 can therefore return no rows while components still validate against known derived, renamed, grouped, and metric fields.

Fields with statically unknown types can be refined from nonempty runtime rows. An unknown field becomes a measure-like field when observed values are finite numbers.

Row lineage and interactions

Every output row carries a sorted unique array of contributing base source-row indices. Runtime row keys use the corresponding source row keys joined with |. A row with no source lineage receives a deterministic dataset-local fallback key.

Lineage enables a chart point, table row, map feature, or grouped dataset row to select the contributing Power BI identities when those identities exist. It does not make a derived field or metric a semantic-model column.

External selection and external filtering are separate:

• selection uses source identities/lineage and can represent a grouped output by selecting its contributing base rows
• filtering constructs a Power BI JSON filter and requires a real model column with source table/column metadata

Dataset metrics and derived fields cannot directly filter Power BI. Renamed direct columns and group-by direct columns may retain a valid external-filter target. Identity selection remains unavailable when the Power BI host supplied no identities.

Map layer dataset scope

Every map layer resolves a separate logical view using layer.dataset, then the map component dataset, then powerbi. Source bindings, renderer/label/popup/tooltip fields, visibility/filter fields, interaction fields, and the Power BI side of an ArcGIS join all use that view's rows and schema. A grouped map feature can therefore carry multiple contributing source row indices and row identities.

This does not create independent Power BI queries. The custom visual still receives one flattened data view; fields may originate from related model tables, but the visual query and semantic-model relationships determine row grain and combinations. Logical datasets only transform the received rows.

Cache signatures

The evaluator builds canonical JSON-like signatures with object keys sorted. The base signature includes row keys, rows, field keys, and base lineage. A named dataset signature includes its definition, its input signature, and static fields.

The caller-provided cache is keyed by that signature. A hit reuses the result, reports cacheStatus: "hit", and sets evaluation time to zero. A miss records input/output row count, elapsed time, unique lineage count, and schema/evaluation warnings.

Complete 2.0 example

The example uses aliases from a hypothetical Field Manifest: region, status, revenue, and cost.

{
  "version": "2.0",
  "title": "Regional revenue",
  "data": {
    "datasets": {
      "activeRows": {
        "source": "powerbi",
        "filter": {
          "field": "status",
          "operator": "!=",
          "value": "Cancelled"
        },
        "derive": {
          "margin": {
            "op": "-",
            "left": { "field": "revenue" },
            "right": { "field": "cost" }
          }
        },
        "rename": { "region": "area" },
        "select": ["area", "revenue", "margin"]
      },
      "areaSummary": {
        "source": "activeRows",
        "groupBy": ["area"],
        "metrics": {
          "totalRevenue": { "op": "sum", "field": "revenue" },
          "totalMargin": { "op": "sum", "field": "margin" },
          "recordCount": { "op": "count" }
        },
        "sort": [
          { "field": "totalRevenue", "direction": "descending" }
        ],
        "limit": 20
      }
    }
  },
  "components": [
    {
      "type": "barChart",
      "id": "revenue_by_area",
      "title": "Revenue by area",
      "dataset": "areaSummary",
      "category": "area",
      "measure": "totalRevenue",
      "aggregation": "sum",
      "interaction": {
        "enabled": true,
        "internalMode": "highlight",
        "externalMode": "selection",
        "selectionMode": "replace"
      }
    }
  ]
}

The component can identity-select contributing Power BI rows through lineage. It must not use totalRevenue as interaction.field with external filter mode because totalRevenue is a dataset metric.

Preloaded multi-resolution chart drill

Hierarchical chart drill is a view switch across declared logical datasets, not an on-demand query. Define one bounded dataset per resolution, retain source lineage through every group, and reference them in the chart's ordered drill.levels. The first level has no parentField; each child level names the field in its own dataset that equals the selected parent category.

{
  "type": "barChart",
  "id": "operational_drill",
  "drill": {
    "trigger": "doubleClick",
    "showBreadcrumbs": true,
    "levels": [
      {"id":"region","dataset":"byRegion","category":"region","measure":"amount"},
      {"id":"facility","dataset":"byFacility","category":"facility","measure":"amount","parentField":"region"}
    ]
  }
}

The runtime filters the already-prepared child view by the breadcrumb path and maps chart points back through dataset lineage. Missing datasets/fields, duplicate level IDs, an unknown initial level, or fewer than two levels fail strict validation. Because every resolution is prepared up front, navigation is deterministic, offline, and subject to the same dataset row and calculation limits as the rest of HyperPBI.

Deliberately unsupported data features

• SQL or DAX text execution
• arbitrary joins/unions
• arbitrary network/file data sources outside declared read-only sources and the web build trust list
• user JavaScript/functions
• mutation of the Power BI semantic model
• treating a dataset metric as a model measure
• direct Power BI filters on derived/metric outputs

These boundaries keep evaluation deterministic, inspectable, and safe inside the custom visual.