Improve inefficient retrieval of zarr chunks from the API

[jjnesbitt]

We are receiving many requests to the API, from the CLI, of the following format:

GET /api/zarr/<zarr_id>/files/?prefix=0/0/0/13/14/97&download=true

Often times, the path provided (0/0/0/13/14/97) is itself the only file returned, and as such, could just be retrieved by querying for the "level" above (0/0/0/13/14). Really though, every response from that endpoint is itself an object, so there should never be a case where the CLI is trying to determine if a path returned is a directory or a file.

I believe the code generating these requests lives here See the comment below.

It's possible the requests we're receiving are from a modified version of the CLI, in which case this issue can be closed (if we can truly determine that to be the case).

[jjnesbitt]

It actually seems to be originating from here:

dandi-cli/dandi/dandiapi.py

Lines 2085 to 2096 in ccdb659

	@property
	def download_url(self) -> str:
	"""
	.. versionadded:: 0.67.0
	The URL from which the entry can be downloaded
	"""
	return str(
	URL(self.client.get_url(f"/zarr/{self.zarr_id}/files/")).with_query(
	{"prefix": str(self), "download": "true"}
	)
	)

Which is being called here.

This results in one request to the archive per zarr chunk.

---

We should instead update the CLI to handle OPEN zarrs (zarrs within OPEN dandisets) by directly navigating to S3. The zarr can then be downloaded either by listing out files under the bucket prefix that corresponds to the zarr root, or by reading the zarr metadata files at the root of the zarr, which are in a known location. The zarr-python library could even be used, if that makes sense.

For embargoed zarrs, this behavior can remain. This is much less of an issue, as authentication would be required to access these zarrs anyway.

[yarikoptic]

but shouldn't we also optimize as to batch request of URLs in such a code so it would become more efficient for embargoed as well?

edit: as soon as we get ETag (dandi/dandi-archive#2215) , we could then persistently cache those on drive as well.

[jjnesbitt]

but shouldn't we also optimize as to batch request of URLs in such a code so it would become more efficient for embargoed as well?

That's true, although that would come as a secondary objective, since public zarrs account for the vast majority of use.