Developer Diary: How the 0.25 crashes were fixed

Commanders!

One of the most common questions you ask us on Discord as well as on the forums are what are we doing about the critical issues that seriously interfere with gameplay – such as game crashes. After the introduction of Update 0.25, we received a considerable number of reports about such problems, which led to a large investigation and to the solutions introduced in the recently applied hofixes. In today’s article, we’d like to tell you more about these issues as well as the process that led to their correction.

Crashes are, generally speaking, one of the most serious issues a game can have, making their correction absolutely vital. The problem with crashes is, however, that they can happen for vastly different reasons and to identify their reason, one must collect a large amount of data because these crashes do not generally happen on the developer side and do not appear during our internal tests (if they did, we wouldn’t have released the affected update). This data is collected via the automated crash report system, the prompts for which appear upon a crash – it’s extremely important and essential to finding the solution for such issues.

The information gathered using this system falls into several different categories, each of which is being handled by a different team. But let’s talk about some specific examples a bit.

These graphs show how the amount of the Russian client crashes changed between the two major updates – the Caribbean Crisis and the Black Sea Incursion, depending on the type of player operating system. Note how the amount of crashes for 32bit systems decreased with the introduction of the Black Sea Incursion update while the amount of crashes for 64bit systems increased. The colors represent each specific update.

Let us digress a bit and talk about the 32bit systems first.

Optimizing the game for 32bit systems might not seem like a big deal for the western markets, but the further east you go, the more relevant it becomes due to the overall level of hardware. The problem with 32bit systems is, however, that their application-assigned memory is limited to 2 GB. In other words, if the game tries to gobble up more than two gigabytes, it will crash. In reality, the crash will actually happen a lot sooner due to memory fragmentation – even at 1.5 to 1.6 GB, there isn’t a free bit of memory left for the game to use.

A vast majority of 32bit system crashes is tied to the lack of memory and to improve the situation, we are working hard on optimizing the client by reducing its memory needs and the fragmentation effect. In any case, we really recommend upgrading to a 64bit system because 32bit systems are inherently less stable with large applications such as Armored Warfare due to the abovementioned restrictions and hardware developers are gradually stopping supporting 32bit systems in general – Nvidia, for example, already announced that it would stop updating 32bit system drivers.

But let us return to the graphs above. You can see that the last hotfix reduced the number of crashes significantly. We are still continuing to monitor the situation to make sure the number of crashes does not overstep the acceptable limits.

To find out more about the crashes that haunted the game in Update 0.25, we have to reach out to the two developers who are best suited to provide us with some answers – the leader of the game client team Aleksei Petranovsky and the lead game mechanics programmer Marat Radchenko.

Let’s take a model situation where a player suffers from a crash in battle and the game sends out a crash report. Naturally, such a player would want to return to the battle as soon as possible and fires up the game again without offering additional details in the crash report comment section. That’s fine. The most important part is to actually allow the report to be sent by pressing the “Send” button in the crash report window.

Such submitted reports are then aggregated into a bin that sorts them out and sends them to different teams. To understand how the sorting works, you must know the following:

• The client calls various so-called “functions”
• When collecting the reports, a list of these called functions is passed to the developers
• Depending on this list, an ID that describes the issue at hand is generated. From this point onwards, all issues of this type (with similar function interactions at the point of the crash) receive this ID. When the developer the issue is assigned to goes through the reports submitted by players, he can identify them by using this ID. This allows him or her to identify the number of such issues that are currently happening.

This graph shows the most typical crashes that are happening for various reasons, sorted by their frequency of occurrence. Some of them are connected to each other (even though the reasons behind them are a bit different) and can be separated into several groups:

Group 1 consists of crash issues 1 and 3 (as per the graph designation). These were caused by old network code problems that subsequently led to a client crash. These issues were fixed in the most recent hotfix and were handled by the team that deals with battle mechanics and their backend. As a result, the number of crashes on the live server dropped by 30 percent.

Group 2 consists of crash issue 2 (as per the graph designation). This crash was caused by a situation where DirectX at some point “loses” its connection to the GPU, causing the client to shut down. This crash can have multiple reasons – problems with graphics card hardware, problems with drivers, GPU overheating or issues with the game’s rendering system. We can unfortunately only deal with the last point and even in that case, finding a solution to this particular type of crash based on crash reports only is very difficult since we need to find a way to replicate the problem on our end. Solving this issue is one of the highest priority tasks the client team currently has.

Group 3 consists of crash issues 4, 5 and 7 (as per the graph designation). These are the battle loading screen crashes. When loading a battle, the main game thread is busy loading up resources and uses and additional thread to update the screen, but certain parts of the game do not expect to be handled by a separate thread and crash. This issue was handled by the User Interface team and was fixed by the last hotfix.

Group 4 consists of crash issue 6 (as per the graph designation). Ironically, this crash is caused by a system that is used to monitor game crashes (this system is called Watch Dog). It monitors the running game processes and if they do not respond within a certain time frame, it artificially and intentionally causes a crash. These crashes are actually just a symptom of another issue, not the issue itself – they are necessary to relay the information about non-responsive components of the game so that we can analyze it. Such non-responsiveness can be caused by multiple reasons – processor overload, disk overload, insufficient memory or a game bug. These issues are also handled by the client team.

As you can see, your feedback and crash reports help us determine which crashes are the most numerous and assign them the highest priority. That does, however, not mean that we are not working on other issues as well – those run in parallel to the abovementioned crash fix works.

It’s very important to note that for those crashes that are rare, submitting a crash report is even more important – please help us by sending them, it makes the crash fixing process much faster. Relying on other players to send their crash reports and ignoring yours is actually something that can seriously hinder our operations – please, don’t be that guy. Every crash deserves to be reported and every report will be investigated and every bit of data is helpful.

You can, naturally, also report any issues to the amazing ladies and gentlemen from our Support Service who note them down and pass them to us in their periodical reports.

We’ll see you on the battlefield!