Add initial CLAUDE.md for AI-assisted development

CLAUDE.md

@@ -0,0 +1,113 @@
+<!-- Last reviewed: 2026-03 -->
+
+## Project
+
+Synapse Python Client — official Python SDK and CLI for Synapse (synapse.org), a collaborative science platform by Sage Bionetworks. Provides programmatic access to entities (projects, files, folders, tables, views), metadata, permissions, evaluations, and data curation workflows. Published to PyPI as `synapseclient`.
+
+## Stack
+
+- Python 3.10–3.14 (`setup.cfg`: `python_requires = >=3.10, <3.15`)
+- HTTP: httpx (async), requests (sync/legacy)
+- Models: stdlib dataclasses (NOT Pydantic)
+- Tests: pytest 8.2, pytest-asyncio, pytest-socket, pytest-xdist
+- Docs: MkDocs with Material theme, mkdocstrings
+- Linting: ruff, black (line-length 88), isort (profile=black), bandit
+- CI: GitHub Actions → SonarCloud, PyPI deploy on release
+- Docker: `Dockerfile` at repo root, published to `ghcr.io/sage-bionetworks/synapsepythonclient`
+
+## Commands
+
+```bash
+# Install for development
+pip install -e ".[boto3,pandas,pysftp,tests,curator,dev]"
+
+# Unit tests
+pytest -sv tests/unit
+
+# Integration tests (requires Synapse credentials, runs in parallel)
+pytest -sv --reruns 3 tests/integration -n 8 --dist loadscope
+
+# Pre-commit checks (ruff, black, isort, bandit)
+pre-commit run --all-files
+
+# Build docs locally
+pip install -e ".[docs]" && mkdocs serve
+```
+
+## Conventions
+
+### Async-first with generated sync wrappers
+All new methods must be async with `_async` suffix. The `@async_to_sync` class decorator (`core/async_utils.py`) auto-generates sync counterparts at class definition time. Never write sync methods manually on model classes — the decorator handles it.
+
+### `wrap_async_to_sync()` for standalone functions
+Use `wrap_async_to_sync()` (not `@async_to_sync`) for free-standing async functions outside of classes — see `operations/` layer for the pattern. The class decorator only works on classes.
+
+### Protocol classes for sync type hints
+Each model in `models/` has a corresponding protocol in `models/protocols/` defining the sync method signatures. When adding a new async method to a model, add its sync signature to the protocol class so IDE type hints work.
+
+### Dataclass models with `fill_from_dict()`
+Models are `@dataclass` classes, NOT Pydantic. REST responses are deserialized via `fill_from_dict()` methods on each model. New models must follow this pattern.
+
+### Concrete types are Java class names
+`core/constants/concrete_types.py` maps Java class names (e.g., `org.sagebionetworks.repo.model.FileEntity`) for polymorphic entity deserialization. When adding new entity types, register the concrete type string here AND in `api/entity_factory.py` AND in `models/mixins/asynchronous_job.py` if it's an async job type.
+
+### Options dataclass pattern
+The `operations/` layer uses dataclass option objects (`StoreFileOptions`, `FileOptions`, `TableOptions`, etc.) to bundle type-specific configuration for CRUD operations. Follow this pattern for new entity-type-specific options.
+
+### Mixin composition for shared behavior
+Shared functionality lives in `models/mixins/` (AccessControllable, StorableContainer, AsynchronousJob, etc.). Prefer adding to existing mixins over duplicating logic across models.
+
+### `synapse_client` parameter pattern
+Most functions accept an optional `synapse_client` parameter. If omitted, `Synapse.get_client()` returns the cached singleton. Never pass `None` explicitly — omit the argument instead.
+
+### Branch naming
+Use `SYNPY-{issue_number}` or `synpy-{issue_number}` prefix for feature branches. PR titles follow `[SYNPY-XXXX] Description` format.
+
+## Architecture
+
+```
+synapseclient/
+├── client.py          # Synapse class — public entry point, REST methods, auth (9600+ lines)
+├── api/               # REST API layer — one file per resource type (21 files)
+│   └── entity_factory.py  # Polymorphic entity deserialization via concrete type dispatch
+├── models/            # Dataclass entities (Project, File, Table, etc.) (28 files)
+│   ├── protocols/     # Sync method type signatures for IDE hints (18 files)
+│   ├── mixins/        # Shared behavior (ACL, containers, async jobs, tables) (7 files)
+│   └── services/      # Model-level business logic (storable_entity, search)
+├── operations/        # High-level CRUD: get(), store(), delete() — factory dispatch
+├── core/              # Infrastructure: upload/download, retry, cache, creds, OTel
+│   ├── upload/        # Multipart upload (sync + async)
+│   ├── download/      # File download (sync + async)
+│   ├── credentials/   # Auth chain (PAT, env var, config file, AWS SSM)
+│   ├── constants/     # Concrete types, config keys, limits, method flags
+│   ├── models/        # ACL, Permission, DictObject, custom JSON serialization
+│   └── multithread_download/  # Threaded download manager
+├── extensions/
+│   └── curator/       # Schema curation (pandas, networkx, rdflib) — optional
+├── services/          # JSON schema validation services
+└── entity.py, table.py, ...  # Legacy classes (pre-OOP rewrite, read-only)
+
+synapseutils/          # Legacy bulk utilities (copy, sync, migrate, walk) — sync-only
+```
+
+Data flow: User → `operations/` factory → model async methods → `api/` service functions → `client.py` REST calls → Synapse API. Responses deserialized via `fill_from_dict()` on model instances.
+
+## Constraints
+
+- Do not use Pydantic for models — the codebase uses stdlib dataclasses with custom serialization. Mixing would break the `@async_to_sync` decorator and `fill_from_dict()` pattern.
+- Do not write synchronous test files — write async tests only. The `@async_to_sync` decorator is validated by a dedicated smoke test. Duplicate sync tests were removed to cut CI cost.
+- Unit tests must not make network calls — `pytest-socket` blocks all sockets. Use `pytest-mock` for HTTP mocking.
+- `develop` is the default/main branch, not `main` or `master`. PRs target `develop`.
+- Legacy classes in root `synapseclient/` (entity.py, table.py, etc.) are kept for backwards compatibility. New features go in `models/` using the dataclass pattern.
+- Avoid adding new methods to `client.py` (9600+ lines) — prefer the `api/` + `models/` layered pattern.
+- `synapseutils/` is legacy sync-only (uses `requests`, NOT `httpx`). Do not add async methods there — new async equivalents go in `models/` or `operations/`.
+
+## Testing
+
+- `asyncio_mode = auto` in pytest.ini — no need for `@pytest.mark.asyncio`
+- `asyncio_default_fixture_loop_scope = session` — all async tests share one event loop
+- Unit test client fixture: session-scoped, `skip_checks=True`, `cache_client=False`
+- Integration tests use `--reruns 3` for flaky retries and `-n 8 --dist loadscope` for parallelism
+- Integration fixtures create per-worker Synapse projects; use `schedule_for_cleanup()` for teardown
+- Auth env vars: `SYNAPSE_AUTH_TOKEN` (bearer token), `SYNAPSE_PROFILE` (config file profile, default: `"default"`), `SYNAPSE_TOKEN_AWS_SSM_PARAMETER_NAME` (AWS SSM path)
+- CI runs integration tests only on Python 3.10 and 3.14 (oldest + newest) to limit Synapse server load

docs/CLAUDE.md

@@ -0,0 +1,55 @@
+<!-- Last reviewed: 2026-03 -->
+
+## Project
+
+User-facing documentation for the Synapse Python Client. Built with MkDocs + Material theme, deployed via GitHub Pages. Follows the Diataxis documentation framework with four content types: tutorials, guides, reference, and explanations.
+
+## Stack
+
+MkDocs with Material theme, mkdocstrings (Google-style docstrings), termynal (CLI animations), markdown-include (file embedding).
+
+## Conventions
+
+### Content types (Diataxis framework)
+- **tutorials/** — Step-by-step learning (competence-building). Themed around a biomedical researcher working with Alzheimer's Disease data. Progressive build-up: Project → Folder → File → Annotations → etc.
+- **guides/** — How-to guides for specific use cases (problem-solution oriented). Includes extension-specific guides (curator).
+- **reference/** — API reference auto-generated from docstrings via mkdocstrings. Split into `experimental/sync/` and `experimental/async/` for new OOP API.
+- **explanations/** — Deep conceptual content ("why" not just "how"). Design decisions, internal machinery.
+
+### File inclusion pattern (markdown-include)
+Tutorial code lives in `tutorials/python/tutorial_scripts/*.py` and is embedded in markdown via line-range includes:
+```markdown
+{!docs/tutorials/python/tutorial_scripts/annotation.py!lines=5-23}
+```
+Single source of truth — edit the `.py` file, not the markdown. Changing line numbers in scripts requires updating the line ranges in the corresponding `.md` files.
+
+### mkdocstrings reference generation
+Reference markdown files use `::: synapseclient.ClassName` syntax to trigger auto-generation from docstrings. Key configuration:
+- `docstring_style: google` — parse Google-style docstrings
+- `members_order: source` — preserve source code order
+- `filters: ["!^_", "!to_synapse_request", "!fill_from_dict"]` — private members, `to_synapse_request()`, and `fill_from_dict()` are excluded from docs
+- `inherited_members: true` — shows mixin methods on inheriting classes
+- Member lists are explicit — each reference page specifies which methods to document
+
+### Anchor links for cross-referencing
+Pattern: `[](){ #reference-anchor }` in reference pages. Tutorials link to reference via `[API Reference][project-reference-sync]`. Explicit type hints use: `[syn.login][synapseclient.Synapse.login]`.
+
+### termynal CLI animations
+Terminal animation blocks marked with `<!-- termynal -->` HTML comment. Prompts configured as `$` or `>`. Used in authentication.md and installation docs.
+
+### Custom CSS (`css/custom.css`)
+- API reference indentation: `doc-contents` has 25px left padding with border
+- Smaller table font (0.7rem) for API docs
+- Wide layout: `max-width: 1700px` for complex content
+
+### Navigation structure
+Defined in `mkdocs.yml` nav section. 5 main sections: Home, Tutorials, How-To Guides, API Reference, Further Reading, News. API Reference has ~85 markdown files (~40 legacy, ~45 experimental).
+
+## Constraints
+
+- Do not edit tutorial code inline in markdown — edit the `.py` script file in `tutorial_scripts/` and update line ranges if needed.
+- Reference docs auto-generate from source docstrings — to change method documentation, edit the docstring in the Python source, not the markdown.
+- `mkdocs.yml` is at the repo root, not in `docs/` — it configures the entire doc build.
+- Docs deploy to Read the Docs (configured via `.readthedocs.yaml` at repo root).
+- Local build output goes to `docs_site/` (via `site_dir` in `mkdocs.yml`) — gitignored.
+- Cross-referencing uses the `autorefs` plugin: `[display text][synapseclient.ClassName.method]` auto-resolves to mkdocstrings anchors.

synapseclient/api/CLAUDE.md

@@ -0,0 +1,48 @@
+<!-- Last reviewed: 2026-03 -->
+
+## Project
+
+REST API service layer — thin async functions that map to Synapse REST endpoints. One file per resource type. Called by model layer, never by end users directly.
+
+## Conventions
+
+### Function signature pattern
+```python
+async def verb_resource(
+    required_param: str,
+    optional_param: str = None,
+    *,
+    synapse_client: Optional["Synapse"] = None,
+) -> Dict[str, Any]:
+```
+- All functions are `async def`
+- `synapse_client` is always the last parameter, keyword-only (after `*`)
+- Use `Synapse.get_client(synapse_client=synapse_client)` to get the client instance
+- Use `TYPE_CHECKING` guard for `Synapse` import — avoids circular dependencies between `api/` and `client.py`
+
+### REST call pattern
+```python
+client = Synapse.get_client(synapse_client=synapse_client)
+return await client.rest_post_async(uri="/endpoint", body=json.dumps(request))
+```
+Available methods: `rest_get_async`, `rest_post_async`, `rest_put_async`, `rest_delete_async`. Pass `endpoint=client.fileHandleEndpoint` for file handle operations; omit for the default repository endpoint. Use `json.dumps()` for request bodies — not raw dicts.
+
+### Return values
+- Most functions return raw `Dict[str, Any]` — transformation happens in the model layer via `fill_from_dict()`
+- Some return typed dataclass instances (e.g., `EntityHeader` from `entity_services.py`) when the data is only used internally
+- Delete operations return `None`
+
+### Pagination
+Use helpers from `api_client.py`:
+- `rest_get_paginated_async()` — for GET endpoints with limit/offset. Expects `results` or `children` key in response.
+- `rest_post_paginated_async()` — for POST endpoints with `nextPageToken`. Expects `page` array in response.
+Both are async generators yielding individual items.
+
+### Entity factory (`entity_factory.py`)
+Polymorphic entity deserialization via concrete type dispatch. Maps Java class names from `core/constants/concrete_types.py` to model classes. When adding a new entity type, register the type mapping here.
+
+### Adding a new service file
+1. Create `synapseclient/api/new_service.py`
+2. Add all public functions to `api/__init__.py` imports and `__all__` — every public function must be re-exported
+3. Use `json.dumps()` for request bodies (not dict)
+4. Reference `entity_services.py` for CRUD pattern, `table_services.py` for pagination pattern

synapseclient/core/CLAUDE.md

@@ -0,0 +1,62 @@
+<!-- Last reviewed: 2026-03 -->
+
+## Project
+
+Infrastructure layer — authentication, file transfer, retry logic, caching, OpenTelemetry tracing, and the `async_to_sync` decorator that powers the dual sync/async API.
+
+## Conventions
+
+### async_to_sync decorator (`async_utils.py`)
+- Scans class for `*_async` methods and creates sync wrappers stripping the suffix
+- Uses `ClassOrInstance` descriptor — methods work on both class and instance
+- Detects running event loop: uses `nest_asyncio.apply()` for nested loops (Python <3.14), raises `RuntimeError` on Python 3.14+ instructing users to call async directly
+- `wrap_async_to_sync()` for standalone functions (not class methods) — used in `operations/` layer
+- `wrap_async_generator_to_sync_generator()` for async generators — must call `aclose()` in finally block
+- `@skip_async_to_sync` decorator excludes specific methods from sync wrapper generation (sets `_skip_conversion = True`)
+- `@otel_trace_method()` wraps async methods with OpenTelemetry spans. Format: `f"{ClassName}_{Operation}: ID: {self.id}, Name: {self.name}"`
+
+### Retry patterns (`retry.py`)
+- `with_retry()` — count-based exponential backoff (default 3 retries), jitter 0.5-1.5x multiplier
+- `with_retry_time_based_async()` — time-bounded (default 20 min), exponential backoff with 0.01-0.1 random jitter
+- Default retryable status codes: `[429, 500, 502, 503, 504]`
+- `NON_RETRYABLE_ERRORS` list overrides status code retry (currently: `["is not a table or view"]`)
+- 429 throttling: wait bumps to 16 seconds minimum
+- Sets OTel span attribute `synapse.retries` on retry
+
+### Credentials chain (`credentials/`)
+Provider chain tries in order: login args → config file → env var (`SYNAPSE_AUTH_TOKEN`) → AWS SSM. Credentials implement `requests.auth.AuthBase`, adding `Authorization: Bearer` header. Profile selection via `SYNAPSE_PROFILE` env var or `--profile` arg.
+
+### Upload/download
+- Both use 60-retry params spanning ~30 minutes for resilience
+- Upload determines storage location from project settings, supports S3/SFTP/GCP
+- Download validates MD5 post-transfer, raises `SynapseMd5MismatchError` on mismatch
+- Progress via `tqdm`; multi-threaded uploads suppress per-file messages via `cumulative_transfer_progress`
+
+### concrete_types.py
+Maps Java class names from Synapse REST API for polymorphic deserialization. When adding a new entity type, add its concrete type string here AND in `api/entity_factory.py` type map AND in `models/mixins/asynchronous_job.py` ASYNC_JOB_URIS if it's an async job type.
+
+### Key reusable utilities (`utils.py`)
+- `delete_none_keys(d)` — removes None-valued keys from dict. MUST call before all API requests — Synapse rejects null values.
+- `id_of(obj)` — extracts Synapse ID from entity, dict, or string
+- `concrete_type_of(entity)` — gets the concrete type string from an entity
+- `get_synid_and_version(id_str)` — parses "synXXX.N" strings into (id, version) tuples
+- `merge_dataclass_entities(source, dest, ...)` — merges fields from one dataclass into another
+- `log_dataclass_diff(obj1, obj2)` — logs field-by-field differences between two dataclass instances
+- `snake_case(name)` — converts camelCase to snake_case
+- `normalize_whitespace(s)` — collapses whitespace
+- `MB`, `KB`, `GB` — byte size constants
+- `make_bogus_data_file()`, `make_bogus_binary_file(n)`, `make_bogus_uuid_file()` — test file generators (in production code, used by tests)
+
+### Exception hierarchy (`exceptions.py`)
+`SynapseError` base with 14+ subclasses: `SynapseHTTPError`, `SynapseMd5MismatchError`, `SynapseFileNotFoundError`, `SynapseNotFoundError`, `SynapseAuthenticationError`, etc. `_raise_for_status()` and `_raise_for_status_httpx()` handle HTTP error responses with Bearer token redaction via `BEARER_TOKEN_PATTERN` regex.
+
+### Rolled-up subdirectories
+
+**`core/models/`** — Internal dataclasses for ACL, Permission, DictObject (dict-like base class), and custom JSON serialization utilities. `DictObject` (`dict_object.py`) provides dot-notation access to dict entries.
+
+**`core/multithread_download/`** — Threaded download manager with `shared_executor()` context manager for external thread pool configuration. Uses `DownloadRequest` dataclass. Default part size: `SYNAPSE_DEFAULT_DOWNLOAD_PART_SIZE`.
+
+## Constraints
+
+- Bearer tokens must never appear in logs — use `BEARER_TOKEN_PATTERN` regex for redaction.
+- `delete_none_keys()` must be called on all dicts before sending to the API — Synapse rejects null values.

synapseclient/core/constants/CLAUDE.md

@@ -0,0 +1,22 @@
+<!-- Last reviewed: 2026-03 -->
+
+## Project
+
+Centralized constants used across the codebase — concrete type mappings, API limits, collision modes, and config file keys.
+
+## Conventions
+
+### concrete_types.py — 3-way registration required
+Maps Java class name strings (e.g., `org.sagebionetworks.repo.model.FileEntity`) for polymorphic entity deserialization. When adding a new entity or job type, register in THREE places:
+1. `concrete_types.py` — add the constant string
+2. `api/entity_factory.py` — add to the type dispatch map
+3. `models/mixins/asynchronous_job.py` `ASYNC_JOB_URIS` — add if it's an async job type
+
+### limits.py
+`MAX_FILE_HANDLE_PER_COPY_REQUEST = 100` and other API batch size limits.
+
+### method_flags.py
+Collision handling modes for file downloads: `COLLISION_OVERWRITE_LOCAL`, `COLLISION_KEEP_LOCAL`, `COLLISION_KEEP_BOTH`.
+
+### config_file_constants.py
+Section and key names for the `~/.synapseConfig` file. `AUTHENTICATION_SECTION_NAME` identifies the auth section.

synapseclient/core/credentials/CLAUDE.md

@@ -0,0 +1,23 @@
+<!-- Last reviewed: 2026-03 -->
+
+## Project
+
+Authentication credential providers implementing a chain-of-responsibility pattern for token resolution.
+
+## Conventions
+
+### Provider chain order (priority)
+1. **UserArgsCredentialsProvider** — explicit login args passed to `syn.login()`
+2. **ConfigFileCredentialsProvider** — `~/.synapseConfig` file (profile-aware via sections)
+3. **EnvironmentVariableCredentialsProvider** — `SYNAPSE_AUTH_TOKEN` env var
+4. **AWSParameterStoreCredentialsProvider** — AWS SSM Parameter Store (via `SYNAPSE_TOKEN_AWS_SSM_PARAMETER_NAME` env var)
+
+### Profile selection
+Select profile via `SYNAPSE_PROFILE` env var or `--profile` CLI arg. If username provided in login args differs from config file username, config credentials are rejected — prevents ambiguity.
+
+### Token handling
+`SynapseAuthTokenCredentials` implements `requests.auth.AuthBase`, adding `Authorization: Bearer` header. JWT validation failure is silent (logs warning, does not raise) — allows tokens with unrecognized formats to attempt API calls.
+
+## Constraints
+
+- Bearer tokens must never appear in logs — redact with `BEARER_TOKEN_PATTERN` regex before logging.