Author: Richard Taylor, Armored Warfare Lead Developer
Hey everyone,
I know there's been a lot of discussion about the 'skill based Matchmaking' we've had in Armored Warfare this week. I wanted to explain a bit more about where it came from, what it's supposed to do, and what future it may or may not have in Armored Warfare.
First off, let's consider the basic goal of the system:
Provide matches where both teams have roughly the same average Winrate across the whole team.
Why? The idea was that if Matchmaking could deliver teams with a close average Winrate across the board, the outcome of the match would be determined by the actions of the players and not by the luck of the draw in what players ended up on the two teams.
In making these Winrate balanced teams, the system is also supposed to obey a few other constraints:
- Balance the tiers evenly across the teams (It is not doing as well at this as it is intended to yet)
- Balance the 'class groups' (not 'classes') evenly across the teams
The gist of this goal came about from evaluating teams where the Winrates of the players are lopsided one way or another. Let me provide a couple examples to showcase what we were aiming to avoid:
Team A versus Team B
- 3 65+% Winrate players versus 1 55% Winrate player
- 4 55% Winrate players versus 6 52% Winrate players
- 5 50% Winrate players versus 4 50% Winrate players
- 3 45% Winrate players versus 4 45% Winrate players
Put against each other, the match outcome is nearly an absolute certainty. It's almost pointless for a match like that to take place. Nothing is being proven other than the fact that a team of obviously better players is going to smash the team of obviously worse players.
Just how extreme is the influence of average team Winrate on match outcome?
There was a period of time where we ran the game without skill influenced Matchmaking. Pulling the data from 140,000 matches during this time, we evaluated the correlation between average team Winrate and match outcome. Here is a look at some of the numbers we discovered.
When Matchmaking did not take player skill into consideration at all (only class/tier were factored in), the following distribution was discovered:
- 38% of the matches had both teams within 1% of each other (50/49, 48/47, etc.)
- 30% of the matches were within 2% of each other (51/49, 48/46, etc.)
- 32% of the matches were 3% or higher between the two teams (53/50, 54/48, etc.)
Now while a 3% spread doesn't seem like a big spread, it does actually take a fairly large skill gap between teams to end up with an overall average gap of 3% or larger.
When the Winrate average between the two teams was 3% or greater, the better team won 70% of the time.
Our intention behind bringing the team average Winrates closer together was to eliminate that bias to the match outcome ahead of time, giving both teams a more even chance of succeeding.
When skill based Matchmaking is active, 99.7% of the matches are within 1% average Winrate. There are some issues with the system admittedly:
Player global Winrate is used, not Winrate with the specific vehicle
The issue here is that we all have vehicles we're good with and ones that we're not so great with, but we're treated the same for team balancing in Matchmaking. It would be easy to switch to vehicle Winrates instead of global, but the sample sizes for a player's Winrate on an individual vehicle are too small to be statistically significant so we haven't gone that route as of yet.
Winrate is not a good indicator of skill
Truthfully, Winrate was never intended to be the only stat factored in. It was intended to be a starting point for building the system under the premise that we could always adjust the actual 'rating' once we felt the basic concept was functional. But I don't think there's another single stat that would do any better, except potentially 'Average Reputation per match'. We would need to develop a more complex rating formula and those are pretty debatable too.
Ultimately, the best players are the ones that are able to skew their teams into wins more often over a large sample size and that would be reflected with a Winrate that is higher than the averages. The worst players are the ones that pull in a Winrate at the low end over a large sample size, and the rest of the stats available to rate them would probably reflect the same trend anyway.
Once you reach extremes (extremely high Winrate or extremely low Winrate), then it starts to get fuzzier in terms of who the best players are, since there's questions such as 'platoon frequency, stat padding with specific vehicles, etc.' on the high end and 'deliberate trolling' on the low end, but for the majority of the players, player ability correlates to an approximate Winrate range pretty consistently.
Good players get stuck with bad players to balance them out during the team balancing pass, while average players get left alone to be average
This is an intrinsic side effect of using skill in any way in balancing team compositions. To be clear, we don't wait for equally good players to queue to place them into teams against each other. We create a group of 30 players with the right tiers/classes to satisfy valid team requirements on a first come/first serve basis, then divide them across the two teams using vehicle/tier first, then massage the team compositions to even out the Winrates. As of the recent hotfix, players with a Winrate over 55% are treated as if they have a Winrate of 55% for purposes of averaging the team Winrates, which means players with over 55% Winrates are being treated as 'worse' than they actually are for team balance, which should work in their favor to a degree.
Now one crucial piece of information I don't have on hand that would be pertinent to this discussion is what difference the 'best' player in a match has on the outcome (i.e., if Team A's best player is X% better than team B, what tends to happen with the match outcome?) Knowing that could provide further analysis as to whether the team balancing effort is actually resulting in fewer pre-determined match outcomes.
With all of this text out of the way, I do want to say that using skill to influence the team balance is not an essential component of Armored Warfare. It was implemented with the goals I outlined above and in response to the numbers that we pulled from live matches.
Even without any skill based Matchmaking, 68% of the matches had teams within 2% of each other's average Winrate, which is overall a pretty close range to pit teams against each other (a 2% spread favors the better team 60% of the time).
It could very well be that a purely random distribution of players across teams (while respecting tier and vehicle) is simply the fairest way to go about team creation. With this approach, the unlucky draws would balance out with the lucky draws over a large enough pool of matches and your individual skill as a player would average out to more wins/losses for you provided your sample size of matches was large enough. You would end up in some matches with pretty biased outcomes ~32% of the time (still, even in the worst of conditions, teams have at least a 27% chance to win statistically), but 2/3's of your matches would have teams that are pretty close to each other in rating.
I do not think you would notice a drastic difference even if we disabled the influence skill has on Matchmaking which is certainly a point in favor of removing the mechanic all together. The goal was to try and eliminate that underlying random element of which players landed randomly on which team and I believe we accomplished that with the system. But if succeeding in that goal isn't providing an overall better experience for everyone, then the system doesn't need to stay.
A number of you have made very compelling arguments for and against the mechanic and I appreciate your reasoning. We'll discuss this system further internally as well.
