Skip to content

[DPE-1577] Supporting storage locations and migration OOP models#1315

Draft
BryanFauble wants to merge 32 commits intodevelopfrom
dpe-1577-storage-locations
Draft

[DPE-1577] Supporting storage locations and migration OOP models#1315
BryanFauble wants to merge 32 commits intodevelopfrom
dpe-1577-storage-locations

Conversation

@BryanFauble
Copy link
Copy Markdown
Member

@BryanFauble BryanFauble commented Feb 4, 2026

Problem:

Storage locations and migration functions were not ported over into the OOP models — users had to use the legacy synapseclient and synapseutils threading-based API to configure storage locations or migrate files between them.

Solution:

  1. Added StorageLocation model with store_async / get_async support, covering all location types: Synapse S3, External S3, GCS, SFTP, HTTPS, External Object Store, and Proxy servers
  2. Added StorageLocationConfigurableMixin on Project and Folder so storage locations can be set directly from those entities
  3. Added async migration.py service with SQLite-backed indexing and multipart_copy_async for migrating files between storage locations, replacing the threading-based synapseutils.migrate_functions
  4. Fixed StorageLocationType disambiguation: SFTP and HTTPS share the same concreteType, now resolved via (concreteType, uploadType) tuple lookup

Testing:

  1. Unit tests added for StorageLocation, migration, migration_types, and storage_location_services
  2. Integration tests added covering SYNAPSE_S3, External S3 (requires AWS credentials), GCS (requires google-cloud-storage), and project/folder storage location assignment

@danlu1 danlu1 changed the title [DPE-1577] Supporting storage locations in SYNPY (Starting point) [DPE-1577] Supporting storage locations and migration OOP models Mar 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants