Copy & Snapshots

Create dataset copies and manage point-in-time snapshots.

POST /api/datasets/{id}/copy

Copy a dataset with optional data, visuals, filters, and sharing configuration.

Authentication: Required

Permission: dataset:copy

Path Parameters:

Parameter	Type	Description
id	string	Dataset ID or slug

Query Parameters:

Parameter	Type	Default	Description
data	boolean	true	Copy dataset records
visuals	boolean	false	Copy visual definitions
filters	boolean	false	Copy saved filters
sharing	boolean	false	Copy share configuration

Request Body:

Field	Type	Required	Description
name	string	Yes	Name for the new dataset
progress	number	No	Progress tracking identifier

Example Request:

POST /api/datasets/sales-2024/copy?data=true&visuals=true&filters=true

{
  "name": "Sales 2024 - Q1 Analysis",
  "progress": 22222
}

Response:

{
  "id": "sales-2024-q1-analysis",
  "name": "Sales 2024 - Q1 Analysis",
  "description": "Copy of Sales Data 2024",

  "datasourceId": "mysql-prod",
  "ownerId": "user:admin",
  "parentId": null,
  "records": 125000,
  "visuals": [
    {
      "id": "visual-abc",
      "name": "Sales by Region",
      "component": "bar-chart"
    }
  ],
  "filters": [
    {
      "id": "filter-xyz",
      "name": "High Value Orders",
      "criteria": { "amount": { "$gte": 1000 } }
    }
  ],
  "sharing": [
    {
      "principalId": "team:marketing",
      "accessLevel": 2
    }
  ],
  "createdAt": "2024-02-08T16:00:00Z"
}

Copy Behavior:

Option	Description
data=true	Copies all records to new Elasticsearch index
visuals=true	Copies visual definitions and configurations
filters=true	Copies saved filters with original ownership
sharing=true	Copies share permissions and access levels

Defaults:

• New dataset owned by current user
• Independent copy (not linked to original)
• New Elasticsearch index created
• Query and flow copied automatically

Selective Copying

Use query parameters to control what gets copied. For quick structure-only copies, use ?data=false to skip record duplication.

---

POST /api/datasets/{id}/snapshots

Create a snapshot of the dataset at a point in time.

Authentication: Required

Permission: dataset:copy

Path Parameters:

Parameter	Type	Description
id	string	Dataset ID or slug

Request Body:

Field	Type	Required	Description
name	string	Yes	Snapshot name
description	string	No	Snapshot description
progress	number	No	Progress tracking identifier

Example Request:

{
  "name": "Q1 2024 Snapshot",
  "description": "End of quarter snapshot for compliance",
  "progress": 33333
}

Response:

{
  "id": "sales-2024-snapshot-q1",
  "name": "Q1 2024 Snapshot",
  "description": "End of quarter snapshot for compliance",

  "parentId": "sales-2024",
  "isSnapshot": true,
  "ownerId": "user:admin",
  "records": 125000,
  "snapshotDate": "2024-02-08T16:30:00Z",
  "createdAt": "2024-02-08T16:30:00Z",
  "_links": {
    "self": { "href": "/api/datasets/sales-2024-snapshot-q1" },
    "inf:parent": { "href": "/api/datasets/sales-2024" }
  }
}

Snapshot Properties:

• parentId - References the source dataset
• isSnapshot - Flagged as snapshot child
• snapshotDate - Timestamp of snapshot creation
• Independent data - Has its own Elasticsearch index
• Read-only - Typically not refreshed after creation

Use Cases:

• Compliance - Preserve historical data for audits
• Reporting - Lock data for specific time periods
• Comparison - Compare current vs. historical data
• Backup - Create recovery points before major changes

---

GET /api/datasets/{id}/snapshot-list

List all snapshots (child datasets) with Elasticsearch stats and record counts.

Authentication: Required

Path Parameters:

Parameter	Type	Description
id	string	Parent dataset ID or slug

Response:

{
  "_links": {
    "self": { "href": "/api/datasets/{id}/snapshot-list" }
  },
  "_embedded": {
    "inf:snapshot": [
      {
        "id": "sales-2024-snapshot-q1",
        "name": "Q1 2024 Snapshot",
        "description": "End of quarter snapshot for compliance",
        "parentId": "sales-2024",
        "isSnapshot": true,
        "records": 125000,
        "size": 45678901,
        "sizeFormatted": "43.5 MB",
        "snapshotDate": "2024-02-08T16:30:00Z",
        "createdAt": "2024-02-08T16:30:00Z"
      },
      {
        "id": "sales-2024-snapshot-q4-2023",
        "name": "Q4 2023 Snapshot",
        "description": "End of year snapshot",
        "parentId": "sales-2024",
        "isSnapshot": true,
        "records": 118500,
        "size": 43123456,
        "sizeFormatted": "41.1 MB",
        "snapshotDate": "2023-12-31T23:59:59Z",
        "createdAt": "2023-12-31T23:59:59Z"
      }
    ]
  },
  "start": 0,
  "count": 2,
  "total": 2
}

---

GET /api/datasets/{id}/snapshots/{snapshotId}

Get a single snapshot's details.

Authentication: Required

Path Parameters:

Parameter	Type	Description
id	string	Parent dataset ID or slug
snapshotId	string	Snapshot ID or slug

Response:

{
  "id": "sales-2024-snapshot-q1",
  "name": "Q1 2024 Snapshot",
  "description": "End of quarter snapshot for compliance",

  "parentId": "sales-2024",
  "isSnapshot": true,
  "ownerId": "user:admin",
  "esIndex": "i5_abc123_sales_2024_snapshot_q1",
  "records": 125000,
  "size": 45678901,
  "snapshotDate": "2024-02-08T16:30:00Z",
  "createdAt": "2024-02-08T16:30:00Z",
  "fields": [...],
  "_links": {
    "self": { "href": "/api/datasets/sales-2024-snapshot-q1" },
    "inf:parent": { "href": "/api/datasets/sales-2024" },
    "inf:data": { "href": "/api/datasets/sales-2024-snapshot-q1/data" }
  }
}

---

PUT /api/datasets/{id}/snapshots/{snapshotId}

Update a snapshot's name and description.

Authentication: Required

Permission: dataset:write (on snapshot)

Path Parameters:

Parameter	Type	Description
id	string	Parent dataset ID or slug
snapshotId	string	Snapshot ID or slug

Request Body:

Field	Type	Description
name	string	Updated snapshot name
description	string	Updated description

Example Request:

{
  "name": "Q1 2024 Final Snapshot",
  "description": "Final end of quarter snapshot with adjustments"
}

Response:

Returns the updated snapshot object.

---

DELETE /api/datasets/{id}/snapshots/{snapshotId}

Delete a snapshot permanently.

Authentication: Required

Permission: dataset:write (on snapshot)

Path Parameters:

Parameter	Type	Description
id	string	Parent dataset ID or slug
snapshotId	string	Snapshot ID or slug

Response:

204 No Content

Permanent Deletion

Deleting a snapshot removes all snapshot data and metadata. This operation cannot be undone.

---

POST /api/datasets/{id}/snapshots/{snapshotId}/promote

Convert a snapshot back to an independent dataset (break parent relationship).

Authentication: Required

Permission: dataset:copy

Path Parameters:

Parameter	Type	Description
id	string	Parent dataset ID or slug
snapshotId	string	Snapshot ID or slug

Request Body:

Field	Type	Required	Description
name	string	Yes	Name for the promoted dataset
description	string	No	Description for promoted dataset
preserveSnapshot	boolean	No	Keep original snapshot (default: false)

Example Request:

{
  "name": "Sales Q1 2024 Analysis",
  "description": "Promoted from Q1 snapshot for independent analysis",
  "preserveSnapshot": true
}

Response:

{
  "id": "sales-q1-2024-analysis",
  "name": "Sales Q1 2024 Analysis",
  "description": "Promoted from Q1 snapshot for independent analysis",

  "parentId": null,
  "isSnapshot": false,
  "ownerId": "user:admin",
  "records": 125000,
  "createdAt": "2024-02-08T17:00:00Z"
}

Behavior:

If preserveSnapshot=false (default):

1. Snapshot is converted to independent dataset
2. parentId removed
3. isSnapshot flag removed
4. Original snapshot ID retained

If preserveSnapshot=true:

1. New independent dataset created
2. Snapshot data copied
3. Original snapshot preserved unchanged

Use Case:

Promote a snapshot when you need to:

• Make it editable and refreshable
• Remove the parent relationship
• Convert historical data to active dataset
• Create a new branch from a snapshot

---

Snapshot Workflow Examples

Create Quarterly Snapshots

// End of Q1
await POST('/api/datasets/sales-2024/snapshots', {
  name: 'Q1 2024',
  description: 'End of Q1 2024'
});

// End of Q2
await POST('/api/datasets/sales-2024/snapshots', {
  name: 'Q2 2024',
  description: 'End of Q2 2024'
});

// List all snapshots
const snapshots = await GET('/api/datasets/sales-2024/snapshot-list');

Compare Snapshots

// Get current data
const current = await GET('/api/datasets/sales-2024/data?limit=1000');

// Get Q1 snapshot data
const q1 = await GET('/api/datasets/sales-2024-snapshot-q1/data?limit=1000');

// Compare record counts
console.log(`Current: ${current.total}, Q1: ${q1.total}`);
console.log(`Growth: ${current.total - q1.total} records`);

Promote and Modify

// Promote snapshot to independent dataset
const promoted = await POST('/api/datasets/sales-2024/snapshots/sales-2024-snapshot-q1/promote', {
  name: 'Q1 Analysis Working Copy',
  preserveSnapshot: true
});

// Now can refresh and modify the promoted dataset
await POST(`/api/datasets/${promoted.id}/_refresh`);
await PUT(`/api/datasets/${promoted.id}`, {
  description: 'Updated working copy with new analysis'
});

---

Best Practices

When to Use Copy vs. Snapshot

Use Copy when:

• Creating an independent dataset
• No need to maintain parent relationship
• Want full control over structure and data
• Creating templates or prototypes

Use Snapshot when:

• Preserving point-in-time data
• Compliance or audit requirements
• Comparing historical vs. current data
• Maintaining relationship to parent

Snapshot Naming Conventions

• Date-based: Q1 2024, Jan 2024, 2024-01-31
• Event-based: Pre-Migration, After Cleanup, v2.0 Release
• Purpose-based: Audit Snapshot, Backup Before Update

Storage Management

• Regular cleanup: Delete old snapshots no longer needed
• Archive strategy: Define retention policy for snapshots
• Monitor size: Track total storage across all snapshots
• Compression: Consider exporting old snapshots to compressed formats

Performance

• Snapshot creation: Can be slow for large datasets
• Storage overhead: Each snapshot requires separate ES index
• Query performance: Same as independent datasets
• Promotion: Fast (metadata change) if not preserving original

Compliance Snapshots

For compliance and audit purposes, create snapshots at regular intervals and document the snapshot date/time clearly. Consider exporting snapshots to archival storage for long-term retention.