DevDiary #16 Part 2 – “Squeezing the world through a pipe” – March 2021
In part 1 of this DevDiary we provided you with a brief overview of the architecture of the Multiplayer mode of The Guild 3. We also explained how the communication between Host and Client can fail when things go wrong.
If you haven’t read part 1 yet you can find it here.
Today, we want to continue with another issue that is difficult to solve: the raw amount of data that needs to be sent over the network to keep all players up to date about what is happening in the game world. Whenever a character executes an action, the Host needs to broadcast the changes that occurred to all Clients. The more characters there are actively running around in the world and executing actions, the more network traffic will be generated. When too much traffic is being generated, the Clients will start to lag behind which means that there is an increasing delay between requesting an action to be executed and seeing the character actually doing so.
To diagnose this type of problem, we have a tool that records and visualizes all the traffic that is sent. As you can see in the image, the tool visualizes in a graph how much data was sent over time. The height of a bar indicates the total amount of data. The colored sections in the bar show which type of objects (characters, buildings, items, etc.) are associated with the data. The numbers next to the labels indicate how many of these objects were updated.
In the graph above the buildings stand out for producing too much traffic. Buildings are in second place regarding the total amount of traffic generated with only 29 updated instances. In the same timeframe it took 315 updated characters to produce a similar amount of traffic. This means that buildings produce much more traffic per update than characters. After digging deeper, we were able to find out that buildings are sending needlessly data about locally stored paths to other nearby buildings along with each update. We were able to fix this issue in patch EA 0.9.13 and significantly reduce the amount of network traffic that is generated by buildings.
Additionally we noticed that on large maps, the number of supply caravans could become very high. The large number of characters moving around was also generating a problematic amount of network traffic. Because of that, we reduced the number of supply caravans that restock markets in regular intervals. Each supply caravan carries more goods, so fewer caravans are required to keep the markets stocked.
With these improvements the Multiplayer mode should run much more smoothly on larger maps. Nevertheless, the total amount of network traffic is still the biggest technical issue for the Multiplayer mode. We will do our best to further improve it in the future.
We also want to shine a spotlight on how character movement works in Multiplayer. It is a good example to demonstrate how certain features that are straight forward in a Singleplayer mode can be hard in a Multiplayer mode. In part 1 of this DevDiary, we have described the usual approach for executing an action in Multiplayer. It is to have the Host perform the action and broadcast the resulting changes to the Clients. When applied naively to character movement, the Host would calculate the path for the character and, over time, update the position as the character walks along the path. In this scenario, each change in the character’s position needs to be broadcasted to each Client. This approach has two problems. For one, it requires a very large number of position updates that need to be sent over the network. And secondly the character’s movement would appear to stutter on the Client because all data that is sent over the internet is always subject to an inconsistent delay.
There are different ways to solve this problem. The Guild 3 approaches it by transmitting primarily the movement destination. The Host will send a message that says “Alice is now running towards position X” or “Alice is now running towards Bob”. Each Client will then independently perform all the steps necessary to move Alice to her destination. That includes calculating a path and moving Alice along that path while trying to avoid collisions with other characters on the street. Once Alice has reached her destination, the Host will send a final message: “Alice is done moving. Her final position is X”. When this message is received on the Client, they will accordingly finish the movement process and place Alice on that position. With that approach, the characters will move smoothly on the Client with only a minor correction at the very end that ensures that the positions of standing characters remain accurate.
Recent Problems aka “Where did my friends go?”
Many of you have noticed and reported connection issues when trying to join multiplayer games in the past couple of weeks. In order to properly explain those problems and how we tackled them, we have to take a little detour into how our matchmaking system works.
The first important part of this system is the matchmaking backend, which enables us to connect between Steam players with players from GOG. The backend is developed, hosted and maintained by another team at THQ Nordic. Secondly, The Guild 3 uses a software library, which communicates with this backend to create, maintain and make your games visible and joinable from all around the world. When you host a game, we send a command to create a so-called lobby for you in the backend, which in turn takes care of making it visible (or invisible) to other players and also enforces player limits. Other players can then see your game in our game listing window, which retrieves this information from the backend. When you decide to join a session, we send a command to the backend that you intend to join this specific lobby. Once this is successful, the backend then notifies the hosting player’s game that someone wants to join and their game instance will then initiate direct communication between your game and the hosting player to make your session possible.
Now let’s have a look at the recent connectivity issues many of you have experienced. We received reports from you and our internal testers, that joining of multiplayer games would sometimes not work or behave in a weird way. With the help of your feedback, the work of our testers and our own experiments we could analyse the issue and see that the underlying problem was in the backend and was most likely introduced with a recent update. As with every piece of software, bugs often surface when implementing new features and this was exactly the case here. We then provided the backend team with all the information we could gather about the problem and they then worked tirelessly to resolve it. In the meantime we also did some small improvements to make the connection process smoother. Patch EA 0.9.13.5 then included an increased connection timeout and a little fix for disconnecting from lobbies. The increased timeout was necessary, because the backend is also used by other games from THQ Nordic and may experience heavy load from time to time.
For the upcoming version of the The Guild 3 we are currently working on some more improvements, which should make the multiplayer experience a bit smoother. E.g. we are upgrading the library we use to interact with the backend (the new version contains a couple of bug fixes and improvements) and we are also remodeling the multiplayer connection process a bit to provide for a nicer experience.
We hope that this DevDiary was able to give you another interesting insight into the multiplayer mode of The Guild 3 and why it is so difficult to solve all problems quickly. We will do our best to continuously improve it.