• nessus42

How Networking Works in Destiny 1 and How It Will Differ in Destiny 2 (According to Bungie)


Caveats

Every time I use the word "console" in this article, please infer this to mean "console or PC" when talking about Destiny 2.

All my information in this articles comes from the following two official online Bungie publications, and this is my attempt to summarize them. To the extent that these sources are not correct, my write-up here may not be correct:

If you have reliable sources or evidence that this write-up is not accurate in some respect or another, please let me know and I will amend it.

Additionally, Bungie's documentation only indicates communication between consoles and servers. It does not indicate that there is any communication directly between consoles. (Other than between the consoles in a zone and the Physics Host for that zone, which in Destiny 1 runs on a console in the zone.) There are people, however, who claim to have monitored the network traffic to and from their console while playing Destiny, and they say that they see traffic between all the consoles in the zone, not just between consoles and servers. I.e., they claim that contra Bungie's implicit claims on the matter, there is P2P communication going on of some sort between all the consoles in a zone.

Here is a Reddit thread that discusses this issue.

I know of no documentation, however, on the nature of this P2P traffic, or what it accomplishes. If anyone has any information about what is going on here, I'd very interested in hearing about it.

Activity Hosts

In Destiny 1, there are dedicated servers, but they only keep track of a small amount of very important information called "Mission Critical State". They also run "Activity Scripts", which do things like open doors when you shove a Tomb Husk in them, etc.

These servers, which are located in Bungie's server cloud, are called "Activity Hosts" and there are two kinds of Activity Hosts: "Bubble Activity Hosts" and "Mission Activity Hosts".

(Note: Bungie doesn't like to use the term "dedicated server" for these dedicated servers. I'll explain why in a little while.)

A "Bubble" is Bungie's name for what people might typically call a "zone". (I'd like to think that with Bungie's attraction to the term "Bubble", Defender bubbles will be back in Destiny 2, but, alas, it's not looking so good at the moment, despite the continued existence of Bubble Activity Hosts....)

A Bubble Activity Host keeps track of some important information about the state of a Bubble (i.e., a zone). For instance, how many adds there are in the Bubble. If there are plates that players can stand on to trigger certain events, the state of each plates is tracked by the Bubble Activity Host, etc.

A Mission Activity Host is like a Bubble Activity Host, only instead it keeps track of certain important information related to the progress of a "mission". E.g., a coop story mission, a strike, a raid checkpoint, or a patrol mission, etc.

It is often the case that your console will be communicating with both a Bubble Activity Host and a Mission Activity Host. For instance, when you start the Vault of Glass, you start off in a public space that random players on Venus can wander through. They can even help you out if they want to, even though they are not participating in the raid.

For every player in that public Bubble, their console will be communicating with Destiny's Bubble Activity Host for that Bubble. For every player who is participating in the raid, their console will also be communicating with Destiny's Mission Activity Host that is responsible for tracking that part of their Vault of Glass mission. [See Footnote 1]

Physics Hosts

There is also a Physics Host for each Bubble. The Physics Host keeps most of the authoritative state for the Bubble. Or rather it keeps most of the authoritative state of all of the myriad little details about the Bubble that are not considered important enough to be stored on the Bubble Activity Host. (The Bubble Activity Host is more authoritative than the Physics Host for the information that it tracks.)

The Physics Host isn't really all that special. Everyone's console is doing exactly the same calculations that are being done by the Physics Host. The only difference is that if your console is not the Physics Host, its state is not authoritative. (Or rather your console is not authoritative about anything other than your character's location and some other information about your character.) If your console has a disagreement with the Physics Host about the state of the Bubble (with the aforementioned caveat), the Physics Host wins, and your console's Bubble is forced to reconcile itself eventually to the agreed upon "reality".

If the player whose console is running the Physics Host for a Bubble leaves that Bubble, the Bubble Activity Host will select a different console in that Bubble to be the new Physics Host. When this happens, the new Physics host will immediately need to reconcile with its Activity Hosts, which are considered authoritative about the information that they track.

When a reconcile happens between the new Physics Host and its Activity Hosts, they may not agree on certain things. They might not even agree on how many adds are alive in the Bubble, for instance. If the new Physics Host has more adds in it than are in the Bubble Activity Host, then some of the adds on the new Physics Hosts will suddenly drop dead. If the new Physics Host has fewer adds than are on the Activity Hosts, then the new Physics Host will suddenly spawn new adds. Etc.

Every other console in the Bubble will now be forced to reconcile its state with the new Physics Host. This process of Physics Host migration typically happens seamlessly without you ever noticing, and in public Bubbles, it tends to happen very frequently. [See Footnote 2]

One bit of voodoo that I'd like to dispel here, now that you understand Destiny's networking model, is that changing the fireteam leader does not have any effect on who is the host. Changing the fireteam leader is voodoo.

Going to orbit from a raid and back into the raid, however is not voodoo. When you do this, you will likely end up with different Activity Hosts, and you may have a better connection to these different Activity Hosts or end up connected to more lightly loaded hosts.

Some people say, however, that you can control to some degree who the Physics Host is in a raid, for instance. If one player enters the raid first, his console will of course be the Physics Host for the raid while he is the only player in the raid, since his console is the only console in the raid Bubble. If other players then join on him, there are people who claim that this player will remain as the Physics Host.

It's not clear from Destiny's official articles on the topic if it's actually the case that you can rely on the Physics Host remaining the same for any significant period of time. There are certainly circumstances that would cause the Physics Host to change, but it's not clear just what those circumstances are. Except if the player who is on the Physics Host leaves the raid. Then, of course, someone else's console will become the Physics Host.

It was using this technique (having one person enter the raid first and then having everyone join on him or her) that allowed, I am told, players to exploit Crota back in the day by disconnecting the Physics Host's network cable. Whether or not you can rely on the Physics Host remaining the same for a long period of time is unclear. Using this technique, however, you could apparently kill Crota quickly enough most of the time, that there wasn't enough time for a Physics Host migration to occur before unplugging the network cable.

Whither Dedicated Servers?

Bungie has stated in the previously cited article This Week At Bungie – 05/25/2017 that they are not implementing "dedicated servers". But it sure seems that they have dedicated servers! For each player there is a Bubble Activity Host, a Mission Activity Host, and a Physics Host dedicate to just them (and the other members of their fireteam and Bubble.) That sure sounds as if they have dedicate servers. In fact, it sound as if they have three of them for each mission that you engage in!

In the face of this, why does Bungie insist that they are not using dedicated servers? One answer could be that there is significant P2P mesh networking going on too, and that Bungie doesn't like to use the term "dedicated server" if there is also a P2P mesh, even if there are no P2P "hosts".

Another reason Bungie doesn't like the term "dedicated server" is explicitly mentioned by Bungie in the above citation. This is what they say:

Matt: We don't use that term, because in the gaming community, “dedicated servers” refers to pure client-server networking models. Destiny 2 uses a hybrid of client-server and peer-to-peer technology, just like Destiny 1. The server is authoritative over how the game progresses, and each player is authoritative over their own movement and abilities. This allows us to give players the feeling of immediacy in all their moving and shooting – no matter where they live and no matter whom they choose to play with.

I think the salient part of this is that they don't implement a "pure client-server networking model". In a pure client/server model, all of the calculations about the game are done on the "dedicated server" and your console is relegated to being mostly a 3D rendering engine.

This is not how things work in either Destiny 1 or 2. In both games, every console is calculating all of the physics for an entire Bubble all of the time. The reason for this is that you want a first-person shooter to be very snappy. You want the bullets that you shoot from your gun to hit the target now and not half-a-second from now. This is very important for an FPS game to feel visceral, real, and snappy. I.e., for it to feel the same way that it would feel if you were playing the story mode of a single-player FPS game.

And Bungie has explicitly stated that this was one of their primary goals in making Destiny: They wanted it to "feel like a single-player shooter".

With a pure client/server model, on the other hand, every time a bullet is fired from your gun, to see the damage done to your enemy, you have to wait for notification of that bullet to travel across the Internet to the server. The server then needs to determine if the bullets intersected with the target of the bullets. Damage from those bullets then needs to be calculated on the server. Then the server needs to send information back across the Internet to your console telling your console the new health of your enemy. Once all of this has happened, finally your console can render the damage. All of this needs to happen before you will see any damage register. Likewise, when you push on your thumbstick to move, you have to wait for that thumb-stick information to be transmitted to the dedicated server, then you have to wait for the server to calculate the effects of that thumbstick movement on your game character's body. And then you have to wait for that information to be received by your console. And then you need to wait for your console to render that information to your TV.

Unless your network connection to the dedicated server is very, very fast, this will not feel as immediate and visceral as playing a non-networked FPS game.

(It should be noted, however, that no popular first-person shooter has implemented this pure of a client/server model in probably more than 15 years. Client/server implementations for first-person shooters have used techniques like "client-side prediction", etc., for quite some time, in order to mitigate the problems just described.)

What happens instead with Bungie's networking, is that your console calculates the damage you do to an enemy immediately or it calculates your character's movement resulting from your controller actions immediately, without consulting with the Physics Host. (Assuming that your host is not the Physics Host.) You see the damage or movement reflected immediately. Periodically, your console and the Physics host reconcile with each other. The Physics Host is taken to be authoritative regarding damage, etc., but your console is taken to be authoritative about your "movement and abilities". Upon reconciliation, the state of your console-specific Bubble will updated to reflect what the Physics host says it should be, taking into account the information about your character that is authoritative to your console.

If you want to know more about the trade-offs between "dedicated servers" and having more physics simulation done by each console, you might be interested in this article: What every programmer needs to know about game networking.

John Carmack (of Doom & Quake fame) is quoted in this article as saying the following about the pure client/server model:

While I can remember and justify all of my decisions about networking from DOOM through Quake, the bottom line is that I was working with the wrong basic assumptions for doing a good internet game. My original design was targeted at < 200ms connection latencies. People that have a digital connection to the internet through a good provider get a pretty good game experience. Unfortunately, 99% of the world gets on with a slip or ppp connection over a modem, often through a crappy overcrowded ISP. This gives 300+ ms latencies, minimum. Client. User's modem. ISP's modem. Server. ISP's modem. User's modem. Client. God, that sucks.

Ok, I made a bad call. I have a T1 to my house, so I just wasn't familliar with PPP life. I'm addressing it now.

To rectify the problems with the pure client/server model, Carmack implemented a more sophisticated approach, and this is what he had to say about this more sophisticated approach:

I am now allowing the client to guess at the results of the users movement until the authoritative response from the server comes through. This is a biiiig architectural change. The client now needs to know about solidity of objects, friction, gravity, etc. I am sad to see the elegant client-as-terminal setup go away, but I am practical above idealistic.

The article goes on to say:

So now in order to remove the latency, the client runs more code than it previously did. It is no longer a dumb terminal sending inputs to the server and interpolating between state sent back. Instead it is able to predict the movement of your character locally and immediately in response to your input, running a subset of the game code for your player character on the client machine.

Now as soon as you press forward, there is no wait for a round trip between client and server – your character start moving forward right away.

The quotes from Carmack are quite old at this point, and so you may think that they are no longer relevant. E.g., no one is trying to play Destiny using a dial-up modem. (Or so I hope.) But many sources of lag still exist. In Destiny, for instance, your fireteam may consist of players from all over the world. I routinely play with people who live in England or Australia or Hong Kong, etc., while I live near Boston, MA. Having a fireteam distributed all over the world is a distinct source of lag.

Additionally, many players opt for slow DSL Internet connections because they are very cheap. And probably worst of all, most people probably connect their console to the Internet using WiFi, which is prone to packet loss, which is often more problematic than lag. I have even played with a few people whose only Internet connection is via a WiFi hotspot coming from their cell phone! One would think that playing Destiny through a cell phone hot-spot could not possibly work well at all! And yet, from my perspective, this didn't cause any problems. The people who do this have also claimed that it works mostly without issue for them.

So, going back a paragraph or two, we see that there's a spectrum between having a completely authoritative "dedicated server" that does all the calculations and relegates your console to being basically just a dumb terminal that is only responsible for 3D rendering, to having a server that delegates more of the authority to each console and lets each console do more sophisticated calculations, so that the console can render the player's actions with less lag.

Destiny has gone to the extreme end with this second approach. So much so that each console does all the very same physics calculations that the server does.

The upside of this is that your personal actions are rendered just as immediately as if you were playing a single-player first-person shooter. The downside of this approach is that if packets representing damage from your gun unfortunately never make it from your console to the Physics Host for some reason, or other network traffic is delayed or goes missing, you may end up seeing glitches, such as enemies that inexplicably heal, or respawn after dying, etc.

When Bungie says that Destiny 2 does not have "dedicated servers", this is what they mean. Even though all of the servers for your shared-world-experiences will run on Bungie servers in Destiny 2.

But What about Cheating?

Counter Strike: Global Offense is the #1 e-sport game being played today, by dollars earned. [Source] It is also a first-person shooter. They care a lot about cheating. Consequently, they went in the opposite direction that Destiny did. Their servers are much closer to the pure client/server model, and the server is completely authoritative about everything. [Source]

The difference between these models means that in Counter Strike, a player suffering lag is always at a disadvantage to a player who has a fast connection. This is what Valve has to say about their client/server model:

Network latencies and lag compensation can create paradoxes that seem illogical compared to the real world. For example, you can be hit by an attacker you can't even see anymore because you already took cover. What happened is that the server moved your player hitboxes back in time, where you were still exposed to your attacker. This inconsistency problem can't be solved in general because of the relatively slow packet speeds.

In the case described by Valve in Counter Strike, the player suffering lag is able to be hit, even though they are behind cover from their point of view, because according to the server, which is taken to be authoritative, they are not yet behind cover.

Destiny PvP, on the other hand, is not an e-sport at all. (At least not if you measure that by money earned by professional players.) Consequently, Destiny doesn't care as much about cheating. In Destiny PvP a player with lag is often at an advantage, rather than at a disadvantage. The reason for this is that in Destiny, a player's console is authoritative about the player's location, rather than the Physics Host being authoritative about this. (Or at least that's what Bungie says.)

When another player's console is authoritative about their position, but their position is being relayed to your console with significant lag or lost packets, they may seem stuck in a certain location, and yet invulnerable to your bullets. Then they can "miraculously" teleport behind you and shoot you in the back.

From their point of view, which is considered to be authoritative, they just walked around you and shot you in the back. But because the packets detailing their travel path from being in front of you to being behind you, did not make it to your console, you console can't render this movement, and instead can only render an apparent sudden change in location.

This vulnerability in Destiny's networking model is why "lag switches" work in Destiny. These are just simple mechanical switches that let a player disconnect their out-bound network wires for a short period of time, without disconnecting the in-bound network wires. Consequently, their console will still be informed of your movements, while your console is not being informed of theirs, until they are suddenly behind you with a gun to your head.

So, there's a clear trade-off here. You can have immediate, authoritative, responsiveness about your position in the Bubble (like in a single-player FPS) but which allows for certain forms of cheating and for potentially glitchy rendering of other players in the face of lag, or you can have a unreliable and potentially glitchy rendering of your own position in the Bubble, but which prevents certain kinds of cheating, but you can't have both.

For better or for worse, Bungie chose the former. For PvE, this is the clearly superior approach, since AI opponents are not going to cheat. For PvP, the best approach is debatable. But at least in my opinion, what's unique about Destiny is its PvE experience, and if they have to make design trade-offs between better PvE or better PvP, they should always prioritize their PvE experience.

One might argue, considering the different requirements of PvE and PvP, that Bungie should have implemented different networking models for PvE and PvP. Fair enough; perhaps they should have. The only real counter-argument to this is that everything that Bungie does has to be done with a given budget and an aggressive schedule, and the more complicated Bungie makes one thing, the fewer resources they will have to devote to other things. I.e., the more effort they spend on their networking code, the fewer resources they will have to develop content. Content like better missions, strikes, and raids. So, what would you rather have in Destiny? More PvE content? Or a PvP experience that is more like Counter Strike? And if the latter is the case, perhaps playing Counter Strike is really what you should be doing. (And if you get really, really, really good at Counter Strike PvP, you could make more than $350k in a year doing so professionally!)

Note: It also may very well be the case that Bungie does have some differences in the networking implementations for PvE and PvP that are appropriate for the two different modes of play. But we probably all know from personal experience that Bungie's PvP networking implementation is prone to lag exploits and/or giving unfortunate glitchy advantages to players suffering from lag, so whatever these differences might be, it is almost certainly the case that the PvP and PvE networking implementations share a large common core.

One should also note that Counter Strike is not immune from cheating either. Cheating in Counter Strike isn't quite as easy as splicing a toggle switch into your network cable, like you can do with Destiny. But as discussed above, a player suffering from lag in Counter Strike is at a serious disadvantage compared to those who have fast, reliable connections to the server.

Consequently, there are players who cheat in Counter Strike by, for instance, running denial-of-service attacks on the other players in a game in order to cause the other players to lag. The players suffering from the DoS attacks will now be at a serious disadvantage.

TLDR: If Bungie implemented more pure client/server networking for PvP a la Counter Strike, it wouldn't eliminate cheating, nor necessarily result in a profoundly better PvP experience. Players who can't get a good connection to a Counter Strike server, or who suffer from various cheats that are used in Counter Strike, also complain bitterly at times.

If WoW Can Do Huge Zones, Why Can't Destiny?

So some naysayers may point out at this point, that World of Warcraft (and games like it) supports huge zones with hundreds of players in them at any given time. If WoW can do this, why can't Destiny?

Well, basically I've already answered this question. WoW doesn't have the snappy, visceral response time of a single-player first-person shooter. Maybe you haven't noticed this, however, because the game-play styles are very different. In WoW, you don't have guns that shoot 15 bullets per second. You're not wearing a jet-pack that lets you jump in the air while turning and bouncing off of other players, while shooting machine guns at the same time.

In WoW, you generally don't have to aim. You select a target, and then all your subsequent spells will be automatically directed at this target. When you cast a spell, it's not an assault rifle spitting out many rounds per second. Instead, maybe you cast a lightning bolt. When you do so, you wave your hands dramatically for a second and then a lightning bolt shoots out of your hand to the pre-selected target. If that lightnight bolt is delayed by a quarter or half of a second, no one is ever going to notice.

The game-play styles are different enough that the technology required to implement them convincingly is also quite different.

I'm not aware of any multiplayer first-person shooters with huge zones. Even games like Counter Strike that have "dedicated servers" as discussed above, require that every client simulate much or all of the physics that the server does. They are implemented this way for the sake of responsiveness, just like with Destiny, and so they suffer from similar limitations that Destiny does in this regard.

Of course each multiplayer FPS game may support zones of different sizes and different degrees of complexity. Each game may also support a different maximum number of concurrent players within a zone, depending on the design trade-offs made by each game implementation. But none of them (that I am aware of) support the truly huge zones many MMORPGs do.

Destiny 2 Does Away with P2P

So, what is changing between Destiny 1 and Destiny 2 in terms of networking?

Well not much at all, really... And yet something really big at the same time: Everything is going to work the same as before, except that now the Physics Host for a Bubble is no longer going to be a randomly-selected console in the Bubble. In Destiny 2, there's no more migrating P2P "host". The Physics Host will always be located in Bungie's server cloud.

In order to keep the visceral snappiness of a single-player FPS game, however, each console will still be doing all of its own physics simulation for an entire Bubble all of the time. And just like with Destiny 1, each console will still be periodically reconciling with the Physics Host. The only difference is that now the Physics Host for a Bubble will always be a server dedicated for this purpose living in Bungie's server cloud, rather than running on a random console in the Bubble.

So, effectively this means that Bungie has implemented the "dedicated servers" that everyone has been clamoring for. Except for the fact that Bungie prefers not to use the term "dedicated server" for this particular type of networking.

Caveat: It's quite likely that Bungie has made other networking tweaks that they haven't told us about. (They might be tweaking the network code with every patch, for all we know.) Also, please recall from some discussion above that Destiny may do a lot more P2P mesh networking than Bungie has described.

Conclusions

It may very well be the case that the networking implementation in Destiny is not the Last Word in first person shooter networking technology. But especially for snappy and seamless cooperative PvE play, it is unparalleled in my experience in providing a game-play experience that rivals the immediacy of a single-player shooter.

I hope that this article has shed some light on how Bungie has been able to accomplish this, and why they have made some of the trade-offs that they have.

I know that in many people's minds, one of their biggest complaints has been lag artifacts in PvP play. And also sometimes at very inopportune moments in PvE raids. Usually, for example, while just on the verge of trying to detonate sixteen bombs simultaneously, or make that final discharge and slam needed for the challenge mode win.

By moving the Physics Hosts into Bungie's server cloud, and thus eliminating the dependency on, for instance, a flaky wifi connection for a Crucible Physics Host or a raid Physics Host, hopefully we will suffer significantly less frustration caused by these poor connections as we rescue the remains of Humanity from the continued and unrelenting onslaught of the forces of Darkness.

Footnotes

Footnote 1: Your console communicates with the Activity Hosts ten times per second. When you are in an instanced Bubble—e.g., when in a Darkness Zone or in a non-public raid checkpoint—I do not know if your console still communicates with both a Bubble Activity Host and a Mission Activity Host, or if in these situations, the two Activity Hosts are merged into a single Activity Host. I suspect that they are, since Bungie states that "Mission Hosts fully control Private Bubbles".

Footnote 2: The typical seamlessness of Physics Host migration in Destiny is in stark contrast to many other multiplayer games of the past. For instance, I have not-so-fond memories of host players quitting in Modern Warfare 2 (even though I loved Modern Warfare 2 PvP). The game would stop for a while, locking up with a message about "host migration", and then eventually the game would continue. Until the next time the host player quit, and then the game would lock up for a while again.

In contrast, Destiny was architected so that the vast majority of the time that a Physic Host migration occurs, you'd never even notice at all. In fact, Bungie boasts that in a public Bubble, a Physics Host migration happens on average every two minutes and forty seconds, and that this happens seamlessly, without you noticing. (I do not know whether Destiny forces these host migrations, or if that's just how often the current Physics Host tends to come and go from a public Bubble.)

Note that this is one of the reasons that Destiny cannot support huge WoW-like zones. A zone (or rather a Bubble) must be completely runnable by every console in a Bubble. (One reason for this is so that every console is always in a state that is suitable for immediately becoming the new Physics Host when a host migration is required.) Consequently, you should be very happy that Destiny dropped support for PS3s and Xbox 360s, since their weaker CPU and RAM capabilities limited the size and complexity of Bubbles in Destiny.

In Destiny 2, host migrations are a thing of the past. Despite this, each console still simulates the physics of an entire Bubble, and so the capabilities of the slowest console supported by Destiny 2 still limits the size and complexity of a Bubble in Destiny 2. (In Destiny 2, why does every host still simulate the physics of an entire bubble if a console will never be tasked with the job of becoming a Physics Host? The answer to this question is discussed in detail elsewhere in this article.)

In fact, the reason that Destiny 2 will still be limited to 30 frames/sec on consoles rather than being upgraded to 60 frames/sec on PS4 Pros and Xbox Scorpios, is due to the limited CPU capabilities of the PS4 Pro and the Xbox Scorpio. One would naively think that any frame rate limitation would be due to the graphics cards in these consoles, but for Destiny 2, this is not the case. The Destiny 2 physics engine cannot simulate the physics in the complex Bubbles that Bungie would like to implement at a rate much higher than 30 ticks (i.e., 30 Bubble-states per second) using the CPUs that come with even the souped-up versions of the current generation of consoles. If Bungie wanted to support 60 frames/sec on consoles, they would have had to make the Bubbles smaller and/or less complex.

You might object here and point out that World of Warcraft (or your favorite MMORPG) supports huge zones with zillions of active players and is able to do so at high frame rates because WoW doesn't require that each gamer's computer duplicate all the work of Blizzard's dedicated servers. Why doesn't Bungie take the same approach as WoW?

SOURCE

#destiny2 #destiny

109 views

TRUE GAMING