Categories & Tags
Archive

All about the Unity networking transport layer

June 11, 2014 in Technology by

This blog is the third in a series about our upcoming networking technology known internally as UNET. We want UNET to be a system that all game developers can use to build multiplayer games for any type of game with any number of players as easily as possible. We’ll be launching UNET in the 5.x cycle.

Get a general overview of the UNET system here, or read on to learn about the UNET transport layer foundation!

When we started to design the new network library for Unity, we want to understand what an ideal library would look like. We realized that we have (roughly) two different types of users:

1.   Users who want networking tools that will give them an out of the box result with minimal effort (ideally without any effort at all).

2.   Users who develop network-centric games and want very flexible and powerful tools.

Based on these two user types, we divided our network library into two different parts: a HLAPI (high-level API) and a LLAPI (low-level API).

If you’re interested in learning more about the Syncvars we use in the high level API you can read more here. The following discussion relates to the low level API and our library design which was based on the following principles:

Performance, performance, performance…

The LLAPI is a thin layer on top of the UDP socket, most of the work is performed in a separate thread (hence LLAPI can be configured to use the main thread only). There’s no dynamic memory allocation and no heavy synchronization (actually most of the library uses memory barrier synchronization with some atomic increment/decrement operation).

If something can be done using C# it should be

We decided to only expose what we felt our users would need to use. Like BSD sockets, the LLAPI supports just one abstraction – exchanging raw binary messages. There are no tpc-like streams, serializers or RPC calls in the LLAPI; only low level messages.

Flexibility and configurability? Yes please…

If you take a look at TCP socket implementation you can find tons of parameters (timeouts, buffer length etc.) which you can change. We chose to to take a similar approach and to allow users to change almost all of our library parameters so they can tune it to their specific needs. Where we faced a choice between simplicity and flexibility we sacrificed simplicity to flexibility.

Nice and easy

We tried to design the LLAPI to resemble the BSD socket API wherever possible.

Network and transport layers

Logically, the UNET low level library is a network protocol stack built on top of the UDP, containing a “network” layer and a “transport” layer. The network layer is used for creating connections between peers, delivering packets and controlling possible flow and congestion. The transport layer works with “messages” belonging to different communication channels:

communication channels

Channels serve two different purposes, they can separate messages logically and they provide different delivery grants or quality of service.

Channels configuration is a part of configuration procedure, something we’ll discuss in more detail in an upcoming post. For now, lets just consider the configuration part as “My system will contain up to 10 connections, each connection will have 5 channels, where channel 0 will have this type, and channel 1 will have other type and so on”. The last part of this sentence is defined by:

The second parameter is channel number and last is channel type, or channel qos (delivery grant).

UNET (so far) supports the following QOS:

  • Unreliable: An unreliable message which can be dropped due to network conditions, or internal buffers overflow, similar to UDP packet.  Example: Short Log Messages
  • UnreliableFragmented: Maximum packet length is fixed, but sometimes you will probably want to send “big” messages. This channel type will disassemble your message to fragments before sending, and assemble it back before receiving. As this qos is unreliable, delivery is not granted.  Example: Long  Logs
  • UnreliableSequenced: Channel grants delivery order, as this qos is unreliable, the message can be missed. Example: voice, video
  • Reliable: Channel grants delivery (or disconnect) but not grant order.  Example: Sending Damage
  • ReliableFragmented: Same as UnreliableFragmented, but additionally it will grant delivery.  Example: Group Damage
  • ReliableSequenced: Same as UnreliableSequenced, but additionally it will grant delivery. This QOS is analogous to TCP stream.  Example: File Delivery/Patching
  • StateUpdate: An unreliable channel type plus channel with this qos will force drop messages which are older than sending/receiving. If when sent, send buffer contains more than one message, only the youngest will send. If the receiving buffer, when reading, contains more than one message only the youngest will be delivered.  Example: Sending Position
  • AllCostDelivery: Fairly similar to Reliable qos, but there is a difference. The reliable channel will resend undelivered messages based on round trip time value (RTT) which is a dynamic parameter while AllCostDelivery will automatically resend messages after a period of time (configured value). This can be useful for small important messages: “I shot player A in the head” or “Mini-game starts”.  Example: Game events such as firing bullets

If you have a use case you feel doesn’t fit conveniently in these categories, we’d love it if you could bring it up in the comments!

Let’s review typical LLAPI function calls:

1.       Initialize library

2.       Configure network: topology, channels amount, their types, different timeouts and buffer sizes (we discuss this in a later post)

3.       Create socket:

This function will open the socket on port 5000 on all network interfaces, and will return an integer value as a descriptor of this socket

4.       Make connection to other peer:

This function will send a connection request to “other peer” at address 127.0.0.1/6000. It will return an integer value as a descriptor of this connection for this host. You will receive the connection event when the connection is established or the disconnect event if the connection cannot be established

5.       Send message:

This last function will send binary data contained in the buffer via socket described by hostId for peer-described connectionId using channel #1 (in our case this is “reliable channel”, so for this message delivery will be granted.

6.       Receiving network events:

For receiving network events, we chose a poll model. User should poll UTransport.Receive() function to be acknowledged about network events. Note that this model is very similar to ordinary select() call with zero timeout. This function receives 4 different events:

  • UNETEventType.kConnectEvent - somebody connects to you, or connection requested by  UTransport.Connect() call has been successfully established
  • UNETEventType.kDisconnectEvent - somebody disconnects with you, or connection requested by UTransport.Connect() call cannot be established for some reason (error code will report what this reason was)
  • UNETEventType.kDatatEvent  - New data has been received
  • UNETEventType.kNothing - Nothing interesting happened

7.       Send disconnect request:

This function call will send a disconnect request to a connection defined by connectionId on host defined by hostId. The connection will close immediately and can be re-used in the future.

That’s it for now! Thanks for reading, and don’t forget to check back (or subscribe to our blog) to catch the next post in the UNET series in which we’ll be discussing network topology configuration.

Share this post

Comments (54)

Comments are closed.

Jes
11 Jun 2014, 9:07 pm

Looks great :)

How can I communicate between UnityClient and pure .Net Lobby, LoadBalancer or RoomServer using UNET?

In most cases Unity on both sides occurs in communicate of Client and battle server in ither cases server is pure .Net

Alexey Abramychev
11 Jun 2014, 9:55 pm

Most probably we will publish standalone library for server platform (Win, Linux, Mac). It is not easy (as UNET is deeply integrated to UNITY) but it is not very difficult too:)

11 Jun 2014, 10:15 pm

Connection descriptor is an int? Why limit yourself to only 2,147,483,647 connections?

… just kidding

11 Jun 2014, 10:21 pm

Naming is very important (at least to me).

Enums whose values start with kXXXX are not very .NET-ish and seem weird.
Building a new library from scratch, i like having everything as perfect as possible (no legacy crap from the past forcing me to do anything). I would change those names if it were possible :)

Jes
11 Jun 2014, 10:32 pm

yeah change to:
UnetEvent.Connected
UnetEvent.Disconnected
UnetEvent.DatatReceived
UnetEvent.Nothing

Jes
11 Jun 2014, 10:34 pm

Or even:
ReceiveResult.Connected
ReceiveResult.Disconnected
ReceiveResult.DatatReceived
ReceiveResult.Nothing

Alexey Abramychev
11 Jun 2014, 10:40 pm

@LIOR TAL @JES
Yes it makes sense, will change

Alexey Abramychev
11 Jun 2014, 10:47 pm

>Connection descriptor is an int? Why limit yourself to only 2,147,483,647 connections?
:)
inside connection described as uint16_t ~65535 connection… We decided to leave int in C# API for the reason do not bore people with casting. If you will over limit this value you will receive error.

11 Jun 2014, 10:49 pm

@Alexey – is there or will there be an early access program? I was already bugging some dude on linkedin to get access but he never replied :)

Alexey Abramychev
11 Jun 2014, 10:56 pm

As far as I remember we published unet for alpha group. No idea for early access program. I guess Erik will able to answer.

Jes
11 Jun 2014, 11:20 pm

What about games with short messages ( like http ):
- connect
- send request
- receive response
- disconnect

Will Unet provide efficient api for such behavior?
Can we use TCP with Unet if we need for some reason?

Alexey Abramychev
11 Jun 2014, 11:31 pm

Sorry, do not understand:
>What about games with short messages ( like http ):
>- connect
>- send request
>- receive response
>- disconnect
It have been supported: you can connect, send request, receive response and then disconnect. But, it is quite slow operation: connect+ack connect = rtt request response = rtt, disconnect rtt/2 + possible timeout if disconnect packet loss. so with 100ms ping time, this procedure will take about 250ms (it is only transmitting expenses). (hence with http it should be slower).

TCP is scheduled as nice to have on the first release, we put our efforts on udp first…

if you ask about something like http – i don’t think so that we will support it on low-level (and we have www class). http is quite slow and quite expensive, so I do not see any reason why we need to support it on low level instead to use any .net library. (Actually i can tell almost the same about tcp, tcp support on c# level won’t affect game performance a lot as the most performance degradation will related to tcp itself)

–Just speculation around your question :)

11 Jun 2014, 11:35 pm

Looks very interesting and I’m absolutely looking forward to it :)

and promissing what you have in the pipe there. Reminds on concept level of what I’ve seen in some of my favorite neworking technologies in the past (Torque and NetDog, a pretty cool C based threaded multiplayer library that sadly got killed by surreal pricings)

Hope to get my hands on it more sooner than later as I’ve a project that would be far less of a pain with UNet ;)

ikriz
11 Jun 2014, 11:42 pm

Good to see low level access also being made available through unity.
Wondering if there is any further documentation of the api available, or do we really need to wait for unity 5?

MadMan
11 Jun 2014, 11:46 pm

This sound good, seems like UNET could meet my needs after all.

1: Right now I use a single Unreliable channel and timestamp/ id each message for my RTS type game, I am limited to 32 Sequenced channels which would not be enough to allocate a channel for each unit’s position. I look forward to the post about how many channels you can have. I am guessing that the number is quite limited? I assume the channels are per connection?

2: If you use uint16, keeping it a uint16 is clearer I don’t mind casting. I like to know what data types I am working with.

3: The way I read the description of StateUpdate is that it’s not guaranteed to be sequential. Am I reading that right? Or are you saying that it does not try to resequence the messages in buffer?

4: It sounds like you will support multiple topology peer to peer? Even for authoritative servers moving below the level of client/server to sender/receiver seems like a good idea.

5: You are using a BinaryFormatter have you thought of using a Protocol Buffer? protobuf-net v2 seems stable.
http://code.google.com/p/protobuf-net/

Alexey Abramychev
12 Jun 2014, 12:18 am

@MARC SCHÄRER, it is not surprising, all network libraries have pretty similar ideas under the hood :)
@IKRIZ Heh, we’re trying to keep reference docs up-to-date with code, but you should already know that docs written by developers is usually sux :) (to prove ref R. Martin “Clean code” code itself is the best document :)) Seriously speaking, yes, before publishing we will polish documentation and examples.

@MADMAN
1. max 255 channels. (it is char coding) yes channels are configured per connections, or other words: I want to create network topology which will contain up to 32000 connections each of them will have 255 channels.
2. We provided all limitation in reference docs, I left this information for now as it is not quite important for design understanding.
3. Sorry, this means that I was unclear, It is granted to be sequenced, all message delivered not sequenced will be dropped: algorithm is quite easy: on sender side: how many messages do I have in channel? 10? ok drop 9 and send last one. On receiver: how many messages do I have in channel? 5? what are they numbers? 7,8, 9, 6, 4? ok I will deliver only 9 and drop others.
4. I guess I will discuss this in the next blogs. Some decision was under discussion in our team, so, I guess it will be interesting to talk about.
5. No/No/Yes :) First no, hlapi uses different protocol, Second No, llapi doesn’t concern about data marshaling and leave this part for library user, third yes, personally, I periodically use this library for different tests :)

MadMan
12 Jun 2014, 1:12 am

From what I have seen so far I like the architecture of the LLAPI. 255 channels meets my needs.

Richard Fine
12 Jun 2014, 1:23 am

How come we have to call UTransport.Init() ourselves? Can’t you do this as part of engine startup?

If we have to call it ourselves, what’s the best place to do that? Note that it’s a bit of a pain having this kind of ‘subsystem’ init attached to GameObjects as it means having to either put the object in each level that we might enter playmode with, or do shenanigans with singletons or something (and similar problem with e.g. third-party audio plugins like WWise). It’d be nice if there was some way to write code that always ran when entering/exiting playmode, without having to put stuff into the scene…

Alexey Abramychev
12 Jun 2014, 1:30 am

:) Hm, they are different. Consider children which want to make a network game, consider user which do not want care about network but want to see network interaction in his games. Different users – different use cases. I’m not guru in .net, but i was (and I am) really impressed with hlapi. While LLAPI is a mostly “straightforward” implementation, HLAPI is sort of art, and it is good example how high level library should be implemented (abstraction, programming model and so on). And again, you can consider these library how two different layers, direct analogy, BSD socket, and library which you created for games :) Except this fact that llapi is layer on top of sockets (which is abstract term too ) :)

Alexey Abramychev
12 Jun 2014, 1:39 am

@RICHARD
>How come we have to call UTransport.Init() ourselves? Can’t you do this as part of engine startup?
No, and the reason is quite easy, from my point of view, user shouldn’t pay for things which he isn’t going to use.

Good place? initial menu, or (“headband”) first screen of the game. Let’s consider formal consequence of the game: game has start, gaming and the end. (as minimum game start is when user turn on his computer and end is when user turn his computer off). It is not kidding, formally speaking this means that you always can find the place where you can set start up and finalize code. Anyway, except Init() you should configure your network layer as well (how many connection do you want, how many channel, what will be channel qos, what will be flow control thresholds, what will be packet size, what will be fragment size and so on, will talk about this later…) So you will need a place somewhere for initialization…

12 Jun 2014, 6:46 am

Really Good And helpful post. i really don’t know about unity networking and transport layer and how its work. i’m learning unity from last 8 months so i daily read all article to learn more.
Thank you :D

12 Jun 2014, 7:09 am

@Alexey

Why was polling choosing as the method of getting data? was it because you are trying to make it resemble the socket API ?

Will the HLAPI provide event based (callback) API ? btw – is there going to be a following post on the HLAPI?

Alexey Abramychev
12 Jun 2014, 8:25 am

@LIOR TAL It is inter domain call. From my point of view all other methods will be not such universal, fast and flexible.

take a look on game loop:
while() {
get_user_input()
get_network_input()
simulate_world()
send_output()
render_world()
}
in this model you will poll network layer anyway… And sure, by adding thread to this model you can create “pure” event based system, or you can hide pool by some sort dispatcher and callback system.

HLAPI as far as I remember is event based. (user provided callback will be called)

Tenebrous
12 Jun 2014, 10:29 am

Perhaps also needed is a ‘ReliableStateUpdate’ mode, whereby it operates like StateUpdate, but with resends, and any resent messages will use the latest state rather than the one that was lost.

Also, I think you mean guarantee/guarenteed instead of grant/granted throughout :)

Alexey Abramychev
12 Jun 2014, 10:36 am

@TENEBROUS — ReliableStateUpdate :) it has been already in the to-do list :)
-Also, I think you mean guarantee/guarenteed instead of grant/granted throughout
yes, you’re right :)

Skyblade
12 Jun 2014, 11:24 am

Will UNET support IPv6?

Hannibalov
12 Jun 2014, 12:28 pm

@ALEXEY
@LIOR TAL

I understand Alexey’s point, but is there any chance we could subscribe to an event when new data arrives? I do it quite frequently with pure c# sockets. And ideally the callback would receive a filled object with the data, the sender, the code, the channel and maybe other stuff, so the callback receiver could filter.

I hope you can still do that because it’s really useful. And it would be even better if it was synced with Unity’s main thread so it would be thread safe.

MadMan
12 Jun 2014, 3:40 pm

Like ALEXEY said you are going to be in the game loop anyway. Also you are going to be doing more than just polling. You should write scripts that handles all this stuff, connection facilitation etc.

“synced with Unity’s main thread so it would be thread safe.”

It’s a little unclear what you are getting at. If you want events it’s trivial to hide the implementation. ALEXEY said that the LLAPI will not be thread safe. I don’t think that many people will be spawning their own threads for networking. Since even at the Low Level API level you will be interacting closely with gameObjects. Thus thread safe events are not needed in my opinion, unless the whole design of unity changes.

The harder part of game networking is actually the client side prediction, dead reckoning, rewinding time, high fire rate weapons prediction etc. Which you will need to handle yourself getting that part done took me months and will be different for each game.

12 Jun 2014, 5:42 pm

The HLAPI is (mostly) C#, so it runs in the Unity main thread. Since it uses the polling mechanism of the LLAPI, all user script code that responds to network events will be run in the Unity main thread, so you don’t have to worry about threading issues there.

The HLAPI main loop would look something like:
– poll LLAPI for events
– generate corresponding HLAPI event
– run the registered event handler

Alexey Abramychev
12 Jun 2014, 7:21 pm

@skyblade ipv6 support scheduled so far as nice to have. I will try to support them but cannot promise

@hannibalov i agree with madman, hence event based model makes sense for server (like lobby) as server is driven by network events only.

Daniel
13 Jun 2014, 12:14 am

Don’t mind my ignorance here, but is it feasible (for a programmer but new to networking and multiplayer) to use both hlapi and llapi? Perhaps have syncvars for certain variables and objects, but for some events to use a separate channel to send unreliable data or some combination of the above?

13 Jun 2014, 10:49 am

I once tried UDP on mobile devices and had to realize that a lot of the cell phone providers block using UDP. How do you plan to overcome this.

Alexey Abramychev
13 Jun 2014, 4:36 pm

@stephan as far as i know, it is history now. Mobile providers blocked it to prevent voip using. With skype on each phone it doesn’t make sense anymore… Let me know please if you have fresh info about udp blocking. Symmetrical nat is a different story, but we are using relay for this

Victoria
16 Jun 2014, 2:04 pm

The new UNET technology seems to give a lot of new possibilities for game developers, but I wonder if there’ll be more documetation on it

Mike
16 Jun 2014, 2:08 pm

Will you provide HTTP/websocket support?

Alexey Abramychev
16 Jun 2014, 9:13 pm

@DANIEL so far no, but it looks like “nice to have” feature.
@VICTORIA I do not have any idea when UNET will be accessible for public, right now it is mostly discussion to be sure that we are going in the right direction. We will publish UNET with full API documentation:)
@MIKE good question, it is still under discussion, (other option is webrtc) probably not for a initial release

Trond
16 Jun 2014, 10:55 pm

@RICHARD FINE:

FWIW, it’s possible to use PostProcessSceneAttribute to inject custom objects into your scene(s) at “play” in the editor and when a build is made. These objects could hold components that run your init code automatically, without having to clutter up your scenes at edit-time, and also eliminating the risk of forgetting to set it up in scenes.

21 Jun 2014, 2:10 am

One thing that will really be nice is a .net DLL. You know so we can host our own master server on a dedicated windows service or (preferably) a .net MVC web application.

Alexey Abramychev
21 Jun 2014, 7:28 am

@Nicholas We know :) and will try to publish. I cannot promise that we will publish this dll on start, but this task in my to-do list has high priority.

Hence, I do not understand you thing about web application, what do you mean? Ability talk with web service using udp? (UNET has udp nature) I meant standalone dll which will provide the same API but can work without Unity…

Gregorius Soedharmo
22 Jun 2014, 4:00 pm

Will you also provide a way to do message broadcasting?

Alexey Abramychev
22 Jun 2014, 7:22 pm

@GREGORIUS We should define first what is broadcasting is? If you mean send the same message to different users, yes it is supported. If you mean classical socket broadcasting, It is not supported (is it important?). Or may be you mean something else? Could you clarify your question?

Gregorius Soedharmo
23 Jun 2014, 1:22 pm

Well, I was thinking more in the line of UDP socket broadcasting. It is mighty useful for WLAN multiplayer peer to peer discovery

Alexey Abramychev
23 Jun 2014, 6:05 pm

@GREGORIUS gotcha, not supported but you idea makes sense, thanks a lot

27 Jun 2014, 8:23 pm

I’m extremely inspired together with your writing skills as
well as with the layout for your weblog. Is that this a paid topic or did you customize it yourself?

Anyway stay up the excellent high quality writing, it is rare to see
a great weblog like this one today..

Mike
29 Jun 2014, 1:48 am

Is that a blocking send? Based on there not being a completion callback, I’m assuming it is. Ideally the LLAPI would support io completion ports on platforms that can support them. I’m using lidgren right now and it is fairly complete for my usage (and covers the same things as LLAPI from the sound of it), but it also includes message pooling/reuse. Unfortunately it still doesn’t support async I/O with completion ports. I’ve considered rewriting it with that support but its quite a task.
How do you handle socket notifications? Would block, send buffer full, etc?

Alexey Abramychev
29 Jun 2014, 9:30 pm

@Mike
> Is that a blocking send?
No, price is 1 memcpy + couple of if()
>How do you handle socket notifications? Would block, send buffer full, etc?
So far there are 3 different models to choose:

1 threaded model where user need call NetworkUpdate from main thread (this model suitable for one core devices and good for debug),

fix rate model, where network thread periodically awakes and does its send/receive job, good for mobile devices

select model, where network thread sleep on select and awakes to do read job, or awakes by timeout to do send job. changing in timeout frequency does flow control to prevent receiver to be overfull with sending messages. Good for pc and consoles

io completion port in the situation when you have only one io thread and only couple of udp sockets doesn’t get a lot of performance (select with one socket 10% worse than iocp with one thread) with select ~150,000 datagrams per second which (i guess) more than enough for client; big plus is that select supported on all platforms…

Server library is totally different task, hence design would be tricky, if we will support iocp with thread pool on single udp socket, some sort of synchronization should be taken into account (and user will need take care about this).

>Would block, send buffer full, etc
if you cannot send message, you will receive false after call Send function and will need to resend message later or drop it.

Mike
30 Jun 2014, 3:17 pm

@Alexey, Thanks for detailed reply!

afpro
1 Jul 2014, 9:57 am

it mentioned that all unet is built on top of udp. i hope i could write my own network transfer layer (eg tcp, bluetooth or my custom hardware like i2c)

Alexey Abramychev
1 Jul 2014, 10:16 am

@AFPRO sorry I do not understand your question. Sure You can write your own network layer using tcp or bluetooth or custom hardware protocols as (and if) those protocols are supported by OS.

afpro
15 Jul 2014, 1:27 pm

i want to use UNET features like ‘SyncVar’, but i want connect players with bluetooth, is that possible?

Tuti
28 Jul 2014, 5:18 am

Will matchmaking make possible to connect players P2P?

Alexey Abramychev
28 Jul 2014, 7:18 am

@AFPRO right now – no, hence i will take a look what we can do :)
@TUTI – I guess that is a topic of other blog, transport layer itself supports p2p

Nice
2 Aug 2014, 9:20 pm

Great news, I am wondering if server can handle huge number of concurrent client in case of making instant messaging server, something like 100k clients or even more

Taugeshtu
4 Aug 2014, 12:51 pm

@ALEXEY ABRAMYCHEV
Hi. Like where this is going. +1 vote for LLAPI + HLAPI working in pair. I could really go wild using LLAPI to lay out network topology and then let HLAPI care about serialization and sending stuff :)
Also, want p2p blog post sooner :)

Leave a Reply

Comments are closed.