Test backwards compatibility of analyses with DB data

[kasbah]

This adds a test for backwards compatibility of analyses. We will run this test on production data in a private Github repo. I added an SSH key for that private Github repo to access production.

[jmoggr]

Code looks fine and does what it sets out to do. Some notes/comments:

• It might be good to document somewhere in this code how this is meant to be run. It's not obvious looking at the code that this will be run in a separate repo. Someone could look at this and reasonably come to the conclusion that it is dead code.

• infrastructure/scripts/dump-notebook-fixtures uses bash and the JSON document snapshots in the DB. We'd like to be moving away from both. It feels like something that ought to be re-written in rust, since that would get rid of the bash and allow us to operate on the automerge documents instead of the snapshots.

• I don't love how this complicates our build and testing pipeline, since we are now splitting stuff across multiple repositories. The only idea I've had:

  • don't print the test logs and instead create private artifacts. This adds the additional step of needing to download the artifacts to view the logs when tests fail.
    • It's not obvious to me if this is better or worse than a separate repo. I lean towards better, but am sympathetic to the other perspective.

[epatters]

Thanks Kaspar. It will be important for us to improve our testing infrastructure as the system grows in complexity and I'm glad you've started to work on that.

Taken together, though, I think what my comments below show is that our system is not currently architected to make this testing easy or convenient. I suspect the problem is that too much of the pipelines model -> analysis and model -> diagram -> analysis are stuck in components or are otherwise not reusable in a standalone script. We should fix that before proceeding because as is I fear that this code will itself become a maintenance problem. I'd be happy to help think through what refactoring is needed if you like.

[epatters]

Jason wrote:

infrastructure/scripts/dump-notebook-fixtures uses bash and the JSON document snapshots in the DB. We'd like to be moving away from both. It feels like something that ought to be re-written in rust, since that would get rid of the bash and allow us to operate on the automerge documents instead of the snapshots.

I won't block the PR on it, but I agree with Jason's comment. Another advantage to writing it in Rust is that we'd check that the SQL queries are valid at compile time.

[epatters]

If we're going down this route, then rather than adding a duplicative, optional field that people have to know about and remember to set, we should refactor the analyses to have a uniform interface based on run or similar. After all, every analysis we have does run something and we already have a semi-standardized set of verbs like simulate and check.

This might also help with integrating web workers in a uniform way (#1105) but I'm less sure about that.

[epatters]

I don't like that we've had to duplicate all this logic for migrating and elaborating the model here. That logic has often changed in the past and it's likely to continue to change in the future.

[epatters]

Ditto here about duplicating logic for elaborating diagrams.