Unity Downloader: Powering the Editor as a continuous integration dependency
Unity has been using Katana over the years for continuous integration (CI). Katana is used for building the releases and running all our internal test suites before we ship a release to you. It’s been working well for a number of years since most of Unity was in the same mercurial repository.
Whenever anyone at Unity wanted to run the editor, it was then possible to either:
- go to an internal website to download the upcoming builds
- build the editor from source code locally
- Or more recently, use the Unity Hub to get the latest published builds
All three available solutions were covering our internal needs very well until recently.
Then came packages
Then, we decided to evolve and start using packages. More and more of Unity features are being moved from the main Unity repository and stored into separate packages repositories. This has allowed us to speed up the delivery process and help take some Unity features and extensions to the next level. It was also a much-needed step on our roadmap to DOTS.
But like any transition, this new technical paradigm came with a set of challenges.
We, of course, wanted to be able to verify and publish these packages and make sure that each of them would play nicely with the different editor versions, and satisfy our internal quality requirements. Packages have their own repositories with a new CI system running on them.
For a package’s CI to be successful, the most important thing is its main dependency: the Unity Editor. We have had no seamless and unified way for the package CI to request a specific Editor build.
So last year we ended up with a lot of the package repos having their own custom way of grabbing the editor. Some of them ended up running CI on physical machines that would just have the editor permanently installed on them with hard-coded paths to each version needed.
Others created custom tools which would trigger Katana builds and download them to the CI machines.
All the solutions were quite hacky and not particularly generic.
So a couple of us sat down at the start of September 2018, using our 20% time that we get in R&D for freeform experimentation, which is also known as Fridays Are For Fun, and set about to try and create a generic way of getting editors for CI that would work for as many people as possible.
And thus, a couple of weeks later, the Unity Downloader was born.
What is the Unity Downloader?
The Unity Downloader is a command-line tool made in Python.
It allows our employees to download any editor (5.0+) for MacOS, Windows and Linux with any combination of components (WebGL, Android, etc).
The Unity Downloader supports this for any Unity version, branch and revision we have.
It’s obviously not just as simple as that, there is some heavy lifting going on in the background that you can’t see, like:
- What is the latest release of 2019.1?
- What revision is it?
- Do we have a build for it?
- How do we reduce unnecessary waiting when people request popular things?
So basically the downloader is a combination of these things:
- The command-line interface (CLI) itself
- which can request, download and extract editor installers and combine them into a valid editor installation.
- Caches for previously requested editors
- We have three of those right now, in locations where it’s easier for our build systems to download them quickly.
- It also uses the public download links as a 4th source in the situation where someone requests a published build.
- The backend
- Which has a rest API the CLI communicates with to request new editor builds to be triggered in Katana.
- It also monitors the triggered builds and communicates progress back to the CLI. Then it puts the finished build in our caches so the CLI can download them.
- A small script that runs in a cron job which monitors our most popular branches (like trunk) and will ensure builds for these are triggered when a new batch lands. This is to minimize the waiting for when someone requests to download a trunk editor since it’s very likely to be requested frequently.
- We can’t store the builds for all of eternity. They take a lot of space and at the same time, we tend to stop using the older revisions a couple of days after something new comes along. So this script just cleans up our caches, deleting files that haven’t been downloaded in the past 30 days.
With all this in place, we could now suddenly run CI on ephemeral VMs which get destroyed and recreated in a clean state after a job ends.
It also easily lets our teams set up their package CI and run it against any supported Unity version they want to test against.
If they need to do a bug fix in the editor itself for their package and ensure it will work in their package CI when it lands, then they can easily tell CI to run using that branch by modifying the Unity Downloader command line to something like this:
The flow for the above request would be as follows, behind the scenes:
- Since we requested a branch we need a revision. So the CLI will ask the backend for what the current latest revision is for the branch 2019.1/myFix, which might return abc123456789.
- The CLI now queries the caches to see if it has the Editor and the il2cpp build target for that revision. If it does then it would simply start downloading it and it would be done. However, if there’s a cache miss, it would continue.
- It would now ask the backend to get the editor and il2cpp component for current OS for the resolved revision.
- The backend will now start by looking among the build artifacts of Katana to see if there already exists a build of it. If it doesn’t, then it will simply trigger it and communicate with the CLI what the estimated time is for the build to complete as well as giving it a Katana build URL in case a user would want to see why it’s taking so long.
- When the build is done, the backend will upload it to the caches and then inform the CLI. that it can now retry its request from step 2.
- The CLI downloads and extracts the builds and the user can launch it at their leisure.
That’s a simplified version of what it is and does.
All internal package CI is now using the unity downloader in some way. Most of our package repo owners don’t know about this since it’s seamlessly handled by our CI tooling to invoke the downloader before it starts building and running tests. The repo owners only need to indicate what version or branch they want and they get it.
Some statistics for the month of May 2019 is:
- 42,000 editors downloaded
- 18,000 downloaded components for above editors
- 80% of usage came from VMs in our build farm, the remaining 20% is from employee machines
In total 300,000 files downloaded since we started using metrics in November.
The graph shows the peaks and valleys of usage throughout a month. You can easily see the nightly jobs we have in CI (peaks) and the weekend lull (flat sections).
So for CI, the use case is pretty obvious. However, a significant chunk of the traffic is coming from actual humans. Some of us have augmented our workflows heavily around its usage.
One situation that the Unity Downloader has been useful for is when tracking down causes of bugs by means of bisection. Since we already tend to find out which published Unity version a bug was introduced in we can trivially get the range of revisions between when it was introduced and the release preceding it. With the Unity Downloader, you can then upfront trigger building as many intermediate revisions as your conscience will allow with something like this.
Setting the —skip-download flag and omitting the — wait flag is the quickest way of just making sure that things are cached and builds are triggered. So a quick succession of those calls will ensure anything that hasn’t been built will be triggered. That ensures that as the bisection progresses after the first couple of revisions tested, the remaining ones are most likely built and ready for download. Doing this in a more traditional sense and building each revision locally would be a very time-consuming task.
Another scenario is to quickly test a Unity project locally. To download the Editor for the project and launch it, I can just do the following:
Which will get the exact editor revision the project was made for and won’t end up triggering the script updater and so on.
So, given that it is now widely used in our automation and has helped our own workflows, I would say it has been a success so far.
However, as with all solutions concocted in a couple of weeks, there are some issues with the design of the system which causes problems.
These problems are mainly:
- The CLI has a lot of business logic in it, it knows about all known components and where they should go. This makes it so that we need to release a new version pretty frequently.
- We also didn’t do a good job with the command line arguments themselves. Choosing a subcommand approach would have made adding new features without muddying the experience a lot easier.
- The download speeds of the CLI for our non-European offices is suboptimal since all the caches are currently located in Europe.
- We can’t share any of it with our users!
Now that the Unity Downloader has reached enough maturity to be widely used internally (especially in CI), we would like to investigate options for making it available for Unity users. It improved our workflows a lot, and we’re confident that it could play a role in helping solve other problems, both internally and externally.
The plan is, over the next weeks and months, to do a complete refactoring of the Unity Downloader.
So with that in mind, and with all the lessons learned along the way, we would like to take a step back and take a moment to involve our users, in order to make the new version of the tool more fit for external needs in case we find a way to expand its usage.
To make sure that the new Unity Downloader fits the needs of our users, especially regarding CI, we would love to get some feedback from you all.
Do you feel like you would have a need for being able to download any or all published editor versions from a terminal of your choice?
Do you have your own CI solution where this might come in handy, and if so then what technical requirements might you have to make it viable for you?
Even if you don’t see yourself using it we would like to know why you think you wouldn’t.