Az prototype design phase run out of token

[KietNhiTran]

I gave az prototype a tried with my project. I set the MVP scope. It stop after discovery phase. I rerun the design with skip discovery. Unfortunately, it is still run out of token. Error message saying it hit the 168000 token limit.
I wonder, can it generate design artifacts on its way rather than accumulate them till the end?

[a11smiles]

@KietNhiTran v0.1.2b5 is what's currently available from the Azure CLI library, but I'm working on v0.1.2b6, and this issue should be fixed. I am targeting a push of this latest beta version to the Azure CLI library later this week. Once I do, please try again.

If it still doesn't work, there's a new feature added to this new beta version that allows you to set the environment variable DEBUG_PROTOTYPE=true. If this is set during a run, a debug log file is generated. Send the log file to me at joshuadavis@microsoft.com and I'll troubleshoot further.

[KietNhiTran]

@KietNhiTran v0.1.2b5 is what's currently available from the Azure CLI library, but I'm working on v0.1.2b6, and this issue should be fixed. I am targeting a push of this latest beta version to the Azure CLI library later this week. Once I do, please try again.

If it still doesn't work, there's a new feature added to this new beta version that allows you to set the environment variable DEBUG_PROTOTYPE=true. If this is set during a run, a debug log file is generated. Send the log file to me at joshuadavis@microsoft.com and I'll troubleshoot further.

I did a fixe locally in my cloned repo. Output the architect, allow resume later. It passed through the design step. Currently in build phase. Few more issues found in build phase: QA review does not persist the findings into disk, those findings gone when the build phase encountered exception. Perhaps another inconvenient is: can't rebuild a phase with the command line unless changing state in build.yml and deleted the generated files.

[a11smiles]

@KietNhiTran all of these issues are fixed in v0.2.1b6 (which I'm working on merging by the end of the week). You can see a list of all the changes, which includes the ones you're listing, here: https://github.com/Azure/az-prototype/blob/build-deploy/HISTORY.rst#021b6

Once I merge to main, I'll close this issue. But, feel free to reopen if they still exist. If you find new issues, then please continue to create new issues. Thank you for your feedback--it's much appreciated!

[a11smiles]

@KietNhiTran thank you for reporting this. This was a "bug" in two places...in the design process (sending all artifacts to the API to generate a build plan) and in the final build stage when the architecture and deployment docs needed to be created.

Two things:

1. This is actually an API bug and not associated with the tool. Unfortunately, there are many cases reported where the GHCP api is rejecting payloads of over 168,000 tokens even though many of the models (e.g., sonnet, opus, and others) allow 1M tokens. This issue also affects other consumers of Copilot API (the SDK, CLI, and VSCode to name a few). This has been reported and the Copilot API team are aware.
2. With that said, I've built in mechanisms to alleviate the issue in both the design stage and build stage. Those fixes were merged in PR 0.2.1b6 - Hardening Build Pipeline #13 and are in the 0.2.1b6 release. I have opened a PR for the release to be merged into the public CLI library. You can follow that here.

I'm going to close this for now. But please feel free to reopen after you have installed the new release. Thanks again.