Search Unity

Unity Downloader: Powering the Editor as a continuous integration dependency

, julio 15, 2019

Unity has been using Katana over the years for continuous integration (CI). Katana is used for building the releases and running all our internal test suites before we ship a release to you. It’s been working well for a number of years since most of Unity was in the same mercurial repository. 

Whenever anyone at Unity wanted to run the editor, it was then possible to either: 

  • go to an internal website to download the upcoming builds
  • build the editor from source code locally
  • Or more recently, use the Unity Hub to get the latest published builds

All three available solutions were covering our internal needs very well until recently. 

Then came packages

Then, we decided to evolve and start using packages. More and more of Unity features are being moved from the main Unity repository and stored into separate packages repositories. This has allowed us to speed up the delivery process and help take some Unity features and extensions to the next level. It was also a much-needed step on our roadmap to DOTS

But like any transition, this new technical paradigm came with a set of challenges. 

We, of course, wanted to be able to verify and publish these packages and make sure that each of them would play nicely with the different editor versions, and satisfy our internal quality requirements. Packages have their own repositories with a new CI system running on them. 

For a package’s CI to be successful, the most important thing is its main dependency: the Unity Editor. We have had no seamless and unified way for the package CI to request a specific Editor build.

So last year we ended up with a lot of the package repos having their own custom way of grabbing the editor. Some of them ended up running CI on physical machines that would just have the editor permanently installed on them with hard-coded paths to each version needed.

Others created custom tools which would trigger Katana builds and download them to the CI machines.

All the solutions were quite hacky and not particularly generic.

So a couple of us sat down at the start of September 2018, using our 20% time that we get in R&D for freeform experimentation, which is also known as Fridays Are For Fun, and set about to try and create a generic way of getting editors for CI that would work for as many people as possible.

And thus, a couple of weeks later, the Unity Downloader was born.

What is the Unity Downloader?

The Unity Downloader is a command-line tool made in Python.

It allows our employees to download any editor (5.0+) for MacOS, Windows and Linux with any combination of components (WebGL, Android, etc).

The Unity Downloader supports this for any Unity version, branch and revision we have.

It’s obviously not just as simple as that, there is some heavy lifting going on in the background that you can’t see, like:

  • What is the latest release of 2019.1?
  • What revision is it?
  • Do we have a build for it?
  • How do we reduce unnecessary waiting when people request popular things?

So basically the downloader is a combination of these things:

  • The command-line interface (CLI) itself
    •  which can request, download and extract editor installers and combine them into a valid editor installation.
  • Caches for previously requested editors
    • We have three of those right now, in locations where it’s easier for our build systems to download them quickly.
    • It also uses the public download links as a 4th source in the situation where someone requests a published build.
  • The backend
    • Which has a rest API the CLI communicates with to request new editor builds to be triggered in Katana.
    • It also monitors the triggered builds and communicates progress back to the CLI. Then it puts the finished build in our caches so the CLI can download them.
  • Pre-fetcher
    • A small script that runs in a cron job which monitors our most popular branches (like trunk) and will ensure builds for these are triggered when a new batch lands. This is to minimize the waiting for when someone requests to download a trunk editor since it’s very likely to be requested frequently.
  • Janitor
    • We can’t store the builds for all of eternity. They take a lot of space and at the same time, we tend to stop using the older revisions a couple of days after something new comes along. So this script just cleans up our caches, deleting files that haven’t been downloaded in the past 30 days.

With all this in place, we could now suddenly run CI on ephemeral VMs which get destroyed and recreated in a clean state after a job ends.

It also easily lets our teams set up their package CI and run it against any supported Unity version they want to test against.

If they need to do a bug fix in the editor itself for their package and ensure it will work in their package CI when it lands, then they can easily tell CI to run using that branch by modifying the Unity Downloader command line to something like this:

The flow for the above request would be as follows, behind the scenes:

  1. Since we requested a branch we need a revision. So the CLI will ask the backend for what the current latest revision is for the branch 2019.1/myFix, which might return abc123456789.
  2. The CLI now queries the caches to see if it has the Editor and the il2cpp build target for that revision. If it does then it would simply start downloading it and it would be done. However, if there’s a cache miss, it would continue.
  3. It would now ask the backend to get the editor and il2cpp component for current OS for the resolved revision.
  4. The backend will now start by looking among the build artifacts of Katana to see if there already exists a build of it. If it doesn’t, then it will simply trigger it and communicate with the CLI what the estimated time is for the build to complete as well as giving it a Katana build URL in case a user would want to see why it’s taking so long.
  5. When the build is done, the backend will upload it to the caches and then inform the CLI. that it can now retry its request from step 2.
  6. The CLI downloads and extracts the builds and the user can launch it at their leisure.

That’s a simplified version of what it is and does.

Present

All internal package CI is now using the unity downloader in some way. Most of our package repo owners don’t know about this since it’s seamlessly handled by our CI tooling to invoke the downloader before it starts building and running tests. The repo owners only need to indicate what version or branch they want and they get it.

Some statistics for the month of May 2019 is:

  • 42,000 editors downloaded
  • 18,000 downloaded components for above editors
  • 80% of usage came from VMs in our build farm, the remaining 20% is from employee machines

In total 300,000 files downloaded since we started using metrics in November.

Downloader usage

The graph shows the peaks and valleys of usage throughout a month. You can easily see the nightly jobs we have in CI (peaks) and the weekend lull (flat sections).

So for CI, the use case is pretty obvious. However, a significant chunk of the traffic is coming from actual humans. Some of us have augmented our workflows heavily around its usage.

One situation that the Unity Downloader has been useful for is when tracking down causes of bugs by means of bisection. Since we already tend to find out which published Unity version a bug was introduced in we can trivially get the range of revisions between when it was introduced and the release preceding it. With the Unity Downloader, you can then upfront trigger building as many intermediate revisions as your conscience will allow with something like this.

Setting the –skip-download flag and omitting the — wait flag is the quickest way of just making sure that things are cached and builds are triggered. So a quick succession of those calls will ensure anything that hasn’t been built will be triggered. That ensures that as the bisection progresses after the first couple of revisions tested, the remaining ones are most likely built and ready for download. Doing this in a more traditional sense and building each revision locally would be a very time-consuming task.

Another scenario is to quickly test a Unity project locally. To download the Editor for the project and launch it, I can just do the following:

Which will get the exact editor revision the project was made for and won’t end up triggering the script updater and so on.

So, given that it is now widely used in our automation and has helped our own workflows, I would say it has been a success so far.

However, as with all solutions concocted in a couple of weeks, there are some issues with the design of the system which causes problems.

These problems are mainly:

  • The CLI has a lot of business logic in it, it knows about all known components and where they should go. This makes it so that we need to release a new version pretty frequently.
    • We also didn’t do a good job with the command line arguments themselves. Choosing a subcommand approach would have made adding new features without muddying the experience a lot easier.
  • The download speeds of the CLI for our non-European offices is suboptimal since all the caches are currently located in Europe.
  • We can’t share any of it with our users!

Future

Now that the Unity Downloader has reached enough maturity to be widely used internally (especially in CI), we would like to investigate options for making it available for Unity users. It improved our workflows a lot, and we’re confident that it could play a role in helping solve other problems, both internally and externally. 

The plan is, over the next weeks and months, to do a complete refactoring of the Unity Downloader.

So with that in mind, and with all the lessons learned along the way, we would like to take a step back and take a moment to involve our users, in order to make the new version of the tool more fit for external needs in case we find a way to expand its usage. 

To make sure that the new Unity Downloader fits the needs of our users, especially regarding CI, we would love to get some feedback from you all.

Do you feel like you would have a need for being able to download any or all published editor versions from a terminal of your choice?

Do you have your own CI solution where this might come in handy, and if so then what technical requirements might you have to make it viable for you? 

Even if you don’t see yourself using it we would like to know why you think you wouldn’t.

We are all ears.

Ya no se aceptan más comentarios.

  1. This would be perfect for us.
    We’ve been trying to get a CI/CD pipeline setup, and the current CLI has been rather tricky to get right.
    We tried U3D as someone else mentioned, but our organisation is very restrictive in terms of downloads etc, so it didn’t work.

    Having an offline Unity we can use in something like Jenkins would be brilliant. Unfortunately, we aren’t allowed to send our builds to Unity Cloud, due to the nature of our work.

  2. Tristan Bellman-Greenwood

    julio 18, 2019 a las 10:06 pm

    This would be very useful for creating Docker images with specific Unity versions. However, I would prefer that it be added to Hub instead of a standalone tool.

  3. Looking forward you guys to share this awesome tool. There’s available docker images of unity editor, but I found most of time I have to create my own image to suit my own needs which is painful experience. A tool like this should be very helpful to build an image from scratch.

  4. Very interesting
    It would be great to use this to easily build assetbundles on any bare-bones vm and do other CI task but how much will our small teams use it, i’m not sure since usually we stick to a specific unity version and installing in on the build machine is not a huge pain but still automating it in this way (assuming it will not cause licensing issues) is interesting

  5. There is an open source CLI tool we have been using that does this task for downloading unity versions and platform modules – https://github.com/DragonBox/u3d

    Maybe worth taking a look at.

    1. Thanks for the tip.
      I will take a look this week for sure.

  6. Good work Devs!! You guys are inspiring millions. I love this alot If this is integrated with Hub.

    1. Thanks. Glad you liked it.

  7. Command Line… what is this 1984?

    1. It tends to be useful for automation :)

    2. Command lines are flexible and fast. Every professional environment I’ve worked it heavily relies on it.

      1. Agreed, I and most of my coworkers are only using Command Line. We almost all have Linux except some scientists. It’s way faster and reliable than any GUI.

    3. Completely agree, when it comes to web development or mobile hybrid development command line is king.

      It can be a little scary at first but it allows you to do things rapidly where it would normally take several clicks / seconds on a gui.

  8. I understood earlier that a CLI was going to be added to Unity Hub in 2.1. Did this change? Was that idea abandoned in favor of this downloader? Or will the downloader be merged with the Hub once ready? In my opinion:

    1- Build this into the Hub. I don’t see why we need to separate them (Hub and downloader) since they do very similar things, might as well be one fancy tool that can do both.
    2- The option to download a Unity version for a specific project/package is great. We want that! (makes CI a little easier)
    3- We want good/informative logging. If something failed, we want to know why because we will need to fix our CI asap.

    Thank you!

    1. The unity downloader was started about a year ago now and was needed to solve a problem we were having.
      It turned out to be successful and introduced some useful workflows for us internally.
      So I wanted to reach out and see if this is functionality which is useful for our users or if we should just keep it internal.
      Making a niche tool for internal use is a lot easier than rolling something out to the world, so I want to make sure it would be useful enough to invest that amount of time and energy into it :)
      So regarding the Hub nothing has changed. If there would be value in adding some of these concepts to the Hub then that is where I will spend the effort.
      It’s up to the users at this point to let me know what the road ahead is.
      Regarding number #3 I assume you mean things like compilation errors and test failures etc so you can see it without having to go into the Editor.log in CI?

      Thanks for your comment!

      1. Thanks for the clarification. As a feature, we do need a tool to just correctly install a specific version of Unity, with specific addons, etc.
        What I don’t understand is, if the Unity downloader will be released to the public, why wouldn’t be integrated with the Unity Hub? After all it is THE tool to handle any task related to fetching and installing any Unity version and players, whether it’s through a GUI or a command line. It’s in the name «Hub», as in I don’t need any other tool to manage Unity installations.
        And the reason I had to mention #3, is that for years people has been managing their own CI (and part of it is hacky, as mentioned in the post), and if a tool was to exist to facilitate this task, it needs to be precise for any kind of failure, fetching a Unity editor, installation, etc. For example, it needs to provide informative logs about why a task failed (e.g. Network issues/Insufficient disk space/etc) as opposed to generic errors that would cost countless hours to figure out why this build didn’t work. That would be very nice and I imagine that it would simplify a part of CI/build machines work.

        1. I agree that it would make sense to have these things be a part of the hub, since it is indeed very important.
          What was left out of the blog post is that I do suspect that most, if not all, of the things the unity downloader is doing will find it’s way into the hub. It doesn’t make a ton of sense for us internally to keep using an internal tool when there is a public one which aims to do similar things. I still wanted to lay out what the current downloader is and what problems we would like to solve in the future incarnation of it.
          Also regarding #3 I do agree that this is a very important and is a huge time sink for people to try and understand obscure CI failures.
          Thanks for your comments so far. It has been useful.