External API Migration Guide: From v1 to v2
This guide provides detailed instructions for migrating from DATAMIMIC External API v1 to v2. The v2 External API offers improved structure, better error handling, and enhanced functionality while maintaining backward compatibility during the transition period.
Overview
DATAMIMIC External API v2 introduces several improvements over v1:
- Better URL Structure: More RESTful and intuitive endpoint organization
- Enhanced Authentication: Improved project access token support
- Better Error Handling: More consistent error responses and status codes
- Improved Performance: Optimized endpoints with better caching
- Enhanced Security: Better validation and security measures
- File Locking System: New file lock/unlock endpoints for collaborative editing
File Locking for Collaborative Work
When multiple users work on the same project, DATAMIMIC v2 automatically locks files to prevent conflicts and ensure data integrity. Files are automatically locked when another user is accessing them.
Unlock File (Required before editing)
| curl -X 'POST' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/files/{filename}/unlock' \
-H 'Authorization: Bearer YOUR_TOKEN'
|
Lock File (After completing your work)
| curl -X 'POST' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/files/{filename}/lock' \
-H 'Authorization: Bearer YOUR_TOKEN'
|
Workflow:
1. Check if file is locked - Files are automatically locked
2. Unlock the file - Use the unlock endpoint before attempting to edit
3. Perform your edits - Make your changes to the file
4. Lock the file - Use the lock endpoint after completing your work to prevent conflicts
Best Practices:
- Always unlock a file before editing if it's currently locked
- Lock the file immediately after completing your changes
- Check file lock status before attempting to edit
- Use appropriate timeout values for long-running operations
Migration Timeline
- Current Status: Both v1 and v2 External APIs are available
- Deprecation Notice: v1 endpoints are marked as deprecated but still functional
- Future Removal: v1 endpoints will be removed in a future release
External API Endpoint Mapping
File Operations
Create File
v1 (Deprecated)
| curl -X 'POST' \
'https://your-datamimic-instance.com/api/v1/{project_id}/files' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-F '[email protected]'
|
v2 (Recommended)
| curl -X 'POST' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/files/create-from-upload' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-H 'accept: application/json' \
-H 'Content-Type: multipart/form-data' \
-F 'file_type=create' \
-F '[email protected]' \
-F 'custom_fields={}'
|
Input Changes:
- URL structure: /{project_id}/files β /projects/{project_id}/files/create-from-upload
- Added required file_type parameter (create, update, dropzone, database_generate_model, json_generate_model, json_generate_template, xml_generate_model, xml_generate_template, gwa_migration)
- Added custom_fields parameter (JSON string, default: "{}")
Output Changes:
- v1: {"message": "Success", "location": "file_uri"}
- v2: {"list_files_uri": ["file_uri1", "file_uri2"]}
Update File
v1 (Deprecated)
| curl -X 'PUT' \
'https://your-datamimic-instance.com/api/v1/{project_id}/files' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-F 'file=@updated_data.csv'
|
v2 (Recommended)
| curl -X 'PUT' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/files/{filename}' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-H 'accept: application/json' \
-H 'Content-Type: multipart/form-data' \
-F 'file=@updated_data.template.json;type=application/json'
|
Input Changes:
- URL structure: /{project_id}/files β /projects/{project_id}/files/{filename}
- Filename now required in URL path
Output Changes:
- v1: {"message": "File updated successfully", "location": "file_uri"}
- v2: {"message": "File 'filename' updated successfully"}
Read File
v1 (Deprecated)
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v1/{project_id}/files/{filename}' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-H 'accept: application/json'
|
v2 (Recommended)
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/files/{filename}' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-H 'accept: application/json'
|
Input Changes:
- URL structure: /{project_id}/files/{filename} β /projects/{project_id}/files/{filename}
Output Changes:
- Both versions return file content directly
- v2 has improved streaming and error handling
Delete File
v1 (Deprecated)
| curl -X 'DELETE' \
'https://your-datamimic-instance.com/api/v1/{project_id}/files/{filename}' \
-H 'Authorization: Bearer YOUR_TOKEN'
-H 'accept: application/json'
|
v2 (Recommended)
| curl -X 'DELETE' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/files/{filename}' \
-H 'Authorization: Bearer YOUR_TOKEN'
-H 'accept: application/json'
|
Input Changes:
- URL structure: /{project_id}/files/{filename} β /projects/{project_id}/files/{filename}
Output Changes:
- v1: {"message": "File deleted successfully"}
- v2: {"message": "File 'filename' deleted successfully"}
Rename File
v1 (Deprecated)
| curl -X 'PUT' \
'https://your-datamimic-instance.com/api/v1/{project_id}/files/rename' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-H 'Content-Type: application/json' \
-H 'accept: application/json' \
-d '{"old_name": "old_file.xml", "new_name": "new_file.xml"}'
|
v2 (Recommended)
| curl -X 'PUT' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/files/{filename}/rename' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-H 'Content-Type: application/json' \
-H 'accept: application/json' \
-d '{"new_filename": "new_file.xml"}'
|
Input Changes:
- URL structure: /{project_id}/files/rename β /projects/{project_id}/files/{filename}/rename
- Request body: {"old_name": "old", "new_name": "new"} β {"new_filename": "new"}
- Current filename now in URL path
Output Changes:
- v1: {"message": "Successfully renamed to new_file.txt", "location": "new_uri"}
- v2: {"message": "File 'old_file.txt' renamed successfully to 'new_file.txt'", "location": "new_uri"}
List Files
v1 (Deprecated)
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v1/{project_id}/files' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN'
|
v2 (Recommended)
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/files_uri' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN'
|
Input Changes:
- URL structure: /{project_id}/files β /projects/{project_id}/files_uri
Output Changes:
- v1: {"files": [{"name": "file1.csv", "extension": "csv", "uri": "uri1"}]}
- v2: The response has the following general structure:
| {
"current_project_files_uri": [
"<project_id>/file1.xml",
"<project_id>/file2.xml",
"<project_id>/path/to/file3.csv",
"<project_id>/script/script1.json",
"<project_id>/script/script2.sql"
],
"global_project_files_uri": {}
}
|
- current_project_files_uri: List of file URIs belonging to the current project.
- global_project_files_uri: Object for global/shared files URIs.
Task Operations
v1 (Deprecated)
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v1/{project_id}/task/{task_id}' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN'
|
v2 (Recommended)
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/tasks/{task_id}/metadata' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN'
|
Input Changes:
- URL structure: /{project_id}/task/{task_id} β /projects/{project_id}/tasks/{task_id}/metadata
- Authentication: Both v1 and v2 support user token and project access token
Output Changes:
- v1: Task metadata wrapped in search response format with pagination
- v2: Direct task metadata JSON response with enhanced structure
- v2: Enhanced error handling with specific error codes
Response Format Comparison:
v1 Response:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16 | {
"task_count": 1,
"result": [
{
"id": 123,
"task_id": "550e8400-e29b-41d4-a716-446655440000",
"project_id": "proj-123",
"status": "SUCCESS",
"datamimic_status": "SUCCESS",
"name": "data_generation_task",
"date_queued": "2024-01-01T10:00:00Z",
"date_started": "2024-01-01T10:00:30Z",
"date_done": "2024-01-01T10:05:00Z"
}
]
}
|
v2 Response:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21 | {
"id": 123,
"task_id": "550e8400-e29b-41d4-a716-446655440000",
"project_id": "proj-123",
"folder": "userdata",
"user_email": "[email protected]",
"status": "SUCCESS",
"datamimic_status": "SUCCESS",
"result": "Task completed successfully",
"traceback": null,
"date_queued": "2024-01-01T10:00:00Z",
"date_done": "2024-01-01T10:05:00Z",
"date_started": "2024-01-01T10:00:30Z",
"eta": null,
"name": "data_generation_task",
"routing_key": "default",
"exception_type": null,
"kwargs": null,
"args": null,
"meta": null
}
|
Log Operations
Get Task Logs
v1 (Deprecated)
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v1/{project_id}/task/{task_id}/log?encode=false' \
-H 'accept: text/plain' \
-H 'Authorization: Bearer YOUR_TOKEN'
|
v2 (Recommended)
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/tasks/{task_id}/logs?encode=false' \
-H 'accept: text/plain' \
-H 'Authorization: Bearer YOUR_TOKEN'
|
Input Changes:
- URL structure: /{project_id}/task/{task_id}/log β /projects/{project_id}/tasks/{task_id}/logs
- Authentication: Both v1 and v2 support user token and project access token
- Default encode parameter: v1 defaults to false, v2 defaults to true
Output Changes:
- v1: Plain text response with optional base64 encoding
- v2: Plain text response with optional base64 encoding (default: encoded)
- v2: Enhanced error handling with specific error codes
- v2: Better project validation and access control
Artifact Operations
List Task Artifacts
v1 (Deprecated)
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v1/{project_id}/task/{task_id}/artifacts' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN'
|
v2 (Recommended)
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/tasks/{task_id}/artifacts/metadata' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN'
|
Input Changes:
- URL structure: /{project_id}/task/{task_id}/artifacts β /projects/{project_id}/tasks/{task_id}/artifacts/metadata
- Authentication: Both v1 and v2 support user token and project access token
Output Changes:
- v1: [{"name": "artifact1.json", "type": "json", "size": 1024}]
- v2: {"artifacts": [{"name": "artifact1.json", "type": "json", "size": 1024, "metadata": {...}}]}
- v2: Enhanced response structure with artifacts array wrapper
Download Task Artifact
v1 (Deprecated)
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v1/{project_id}/task/{task_id}/artifacts/download' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-o artifacts.zip
|
v2 (Recommended)
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/tasks/{task_id}/artifacts/download' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-o artifacts.zip
|
Input Changes:
- URL structure: /{project_id}/task/{task_id}/artifacts/download β /projects/{project_id}/tasks/{task_id}/artifacts/download
- Authentication: Both v1 and v2 support user token and project access token
Output Changes:
- Both versions return ZIP file download
- v2: Enhanced streaming response with better error handling
- v2: Improved Content-Disposition headers
Download Specific Artifact File
v1 (Deprecated)
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v1/{project_id}/task/{task_id}/artifact/download/{file_name}' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-o artifact_file.json
|
v2 (Recommended)
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/tasks/{task_id}/artifacts/{entity_name}/download' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-o artifact_file.json
|
Input Changes:
- URL structure: /{project_id}/task/{task_id}/artifact/download/{file_name} β /projects/{project_id}/tasks/{task_id}/artifacts/{entity_name}/download
- Parameter name: file_name β entity_name
- entity_name must include the file extension (e.g., example.json).
- Authentication: Both v1 and v2 support user token and project access token
Output Changes:
- Both versions return binary file download
- v2: Enhanced streaming response with better error handling
- v2: Improved Content-Disposition headers
Project Operations
v1 API (Deprecated):
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v1/{project_id}/info' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN'
|
v2 API (Current):
| curl -X 'GET' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN'
|
Key Changes:
- URL Structure: Changed from /api/v1/{project_id}/info to /api/v2/projects/{project_id}
- Response Format:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20 | {
"identifier": "project_id",
"name": "demo-json",
"remote_url": null,
"tc_creation": "2024-10-10T04:20:35.148841",
"tc_update": "2024-10-10T08:23:32.102951",
"type": "standard",
"config": {
"csv": {
"delimiter": ";"
}
},
"users": [
{
"email": "[email protected]",
"first_name": "John",
"last_name": "Doe"
}
]
}
|
Update Project from Git
v1 API (Deprecated):
| curl -X 'PUT' \
'https://your-datamimic-instance.com/api/v1/{project_id}/git?branch_name=main' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN'
|
v2 API (Current):
| curl -X 'PUT' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/git?branch_name=main' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN'
|
Key Changes:
- URL Structure: Changed from /api/v1/{project_id}/git to /api/v2/projects/{project_id}/git
Generate Operations
Generate Synthetic Data
v1 API (Deprecated):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16 | curl -X 'POST' \
'https://your-datamimic-instance.com/api/v1/{project_id}/generate' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-H 'Content-Type: application/json' \
-d '{
"task_type": "standard",
"custom_property_file": {
"name": "runtime.properties",
"data": {
"count": 100,
"feature_premium": true
}
},
"snapshot_id": "optional-snapshot-id"
}'
|
v2 API (Current):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17 | curl -X 'POST' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/generate' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-H 'Content-Type: application/json' \
-d '{
"task_type": "standard",
"timeout": 30,
"custom_property_file": {
"name": "runtime.properties",
"data": {
"count": 100,
"feature_premium": true
}
},
"snapshot_id": "optional-snapshot-id"
}'
|
Key Changes:
- URL Structure: Changed from /api/v1/{project_id}/generate to /api/v2/projects/{project_id}/generate
- Response Format: v2 responses follow this structure:
| {
"returncode": 0,
"message": "string",
"data": {}
}
|
Schedule Advanced Generation
v1 API (Deprecated):
| curl -X 'POST' \
'https://your-datamimic-instance.com/api/v1/{project_id}/generate/advanced' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-H 'Content-Type: application/json' \
-d '{
"envs": "stage",
"crontab": "0 0 * * *",
"queue": "default",
"infinite": false
}'
|
v2 API (Current):
| curl -X 'POST' \
'https://your-datamimic-instance.com/api/v2/projects/{project_id}/generate/schedule' \
-H 'accept: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-H 'Content-Type: application/json' \
-d '{
"crontab": "0 0 * * *"
}'
|
Key Changes:
- URL Structure: Changed from /api/v1/{project_id}/generate/advanced to /api/v2/projects/{project_id}/generate/schedule
- Request Format:
- Simplified request body - only crontab parameter is required
- Removed envs, queue, and infinite parameters (handled internally)
- Response Format:
| {
"job_id": "string",
"message": "Job \"string\" successfully created.\nSchedule: <crontab: 0 0 * * * (m/h/dM/MY/d)>\nNext run: 2024-10-10 23:59:59.998123+00:00"
}
|
Complete Migration Examples
File Management Script Migration
v1 Script
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21 | #!/bin/bash
PROJECT_ID="your-project-id"
TOKEN="your-token"
BASE_URL="https://your-datamimic-instance.com/api/v1"
# Upload file
curl -X 'POST' \
'${BASE_URL}/${PROJECT_ID}/files' \
-H "Authorization: Bearer ${TOKEN}' \
-F '[email protected]"
# List files
curl -X 'GET' \
'${BASE_URL}/${PROJECT_ID}/files' \
-H "Authorization: Bearer ${TOKEN}"
# Download file
curl -X 'GET' \
'${BASE_URL}/${PROJECT_ID}/files/data.csv' \
-H "Authorization: Bearer ${TOKEN}' \
-o downloaded_data.csv
|
v2 Script
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23 | #!/bin/bash
PROJECT_ID="your-project-id"
TOKEN="your-token"
BASE_URL="https://your-datamimic-instance.com/api/v2"
# Upload file
curl -X 'POST' \
'${BASE_URL}/projects/${PROJECT_ID}/files/create-from-upload' \
-H "Authorization: Bearer ${TOKEN}' \
-F 'file_type=create' \
-F '[email protected]' \
-F 'custom_fields={}"
# List files
curl -X 'GET' \
'${BASE_URL}/projects/${PROJECT_ID}/files_uri' \
-H "Authorization: Bearer ${TOKEN}"
# Download file
curl -X 'GET' \
'${BASE_URL}/projects/${PROJECT_ID}/files/data.csv' \
-H "Authorization: Bearer ${TOKEN}' \
-o downloaded_data.csv
|
Task Management Script Migration
v1 Script
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15 | #!/bin/bash
PROJECT_ID="your-project-id"
TASK_ID="your-task-id"
TOKEN="your-token"
BASE_URL="https://your-datamimic-instance.com/api/v1"
# Get task info
curl -X 'GET' \
'${BASE_URL}/${PROJECT_ID}/task/${TASK_ID}' \
-H "Authorization: Bearer ${TOKEN}"
# Get task logs
curl -X 'GET' \
'${BASE_URL}/${PROJECT_ID}/task/${TASK_ID}/log?encode=false' \
-H "Authorization: Bearer ${TOKEN}"
|
v2 Script
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21 | #!/bin/bash
PROJECT_ID="your-project-id"
TASK_ID="your-task-id"
TOKEN="your-token"
BASE_URL="https://your-datamimic-instance.com/api/v2"
# Get task metadata
curl -X 'GET' \
'${BASE_URL}/projects/${PROJECT_ID}/tasks/${TASK_ID}/metadata' \
-H "Authorization: Bearer ${TOKEN}"
# Get task snapshot
curl -X 'GET' \
'${BASE_URL}/tasks/${TASK_ID}/snapshot/download' \
-H "Authorization: Bearer ${TOKEN}' \
-o snapshot_${TASK_ID}.zip
# Get task logs
curl -X 'GET' \
'${BASE_URL}/projects/${PROJECT_ID}/tasks/${TASK_ID}/logs?encode=true' \
-H "Authorization: Bearer ${TOKEN}"
|
Common Issues and Solutions
Issue 1: 404 Not Found
Problem: Endpoint not found after migration
Solution: Verify the new URL structure and ensure all path parameters are correct
| # Wrong (v1 style)
curl -X 'GET' \
'https://api.datamimic.com/api/v2/{project_id}/files"
# Correct (v2 style)
curl -X 'GET' \
'https://api.datamimic.com/api/v2/projects/{project_id}/files_uri"
|
Issue 2: Authentication Errors
Problem: 401 Unauthorized errors
Solution: Verify token format and ensure project access tokens are valid
| # Check token format
curl -X 'GET' \
'https://api.datamimic.com/api/v2/projects/{project_id}/files_uri' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-v
|
Issue 3: File Upload Issues
Problem: File upload failures
Solution: Ensure file_type parameter is included and custom_fields is properly formatted
1
2
3
4
5
6
7
8
9
10
11
12
13 | # Missing file_type (will fail)
curl -X 'POST' \
'https://api.datamimic.com/api/v2/projects/{project_id}/files/create-from-upload' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-F '[email protected]"
# Correct with file_type
curl -X 'POST' \
'https://api.datamimic.com/api/v2/projects/{project_id}/files/create-from-upload' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-F 'file_type=create' \
-F 'file=@data.csv' \
-F 'custom_fields={}"
|
Problem: Scripts break due to response format changes
Solution: Update response parsing logic to handle new response structure
| # v1 response parsing
response=$(curl -s -X GET "https://api.datamimic.com/api/v1/{project_id}/files' \
-H 'Authorization: Bearer YOUR_TOKEN')
echo $response | jq '.files[0].name'
# v2 response parsing
response=$(curl -s -X GET "https://api.datamimic.com/api/v2/projects/{project_id}/files_uri' \
-H 'Authorization: Bearer YOUR_TOKEN')
echo $response | jq '.files[0].name'
|
Best Practices
- Gradual Migration: Migrate endpoints gradually rather than all at once
- Error Handling: Implement robust error handling for both v1 and v2 responses during transition
- Testing: Thoroughly test each migrated endpoint before moving to the next
- Documentation: Update your internal documentation to reflect the new API structure
- Monitoring: Monitor API usage and errors during migration
Support
If you encounter issues during migration:
- Check Documentation: Review the External API Documentation for detailed endpoint information
- Swagger UI: Use the interactive Swagger documentation to test endpoints
- Contact Support: Reach out to the DATAMIMIC support team for assistance
Conclusion
Migrating from External API v1 to v2 will provide you with improved functionality, better error handling, and enhanced security. While the migration requires some script changes, the benefits of the new API structure make it worthwhile. Follow this guide step by step to ensure a smooth transition.