Welcome

Please register for Total War Access to use the forums. If you're an existing user, your forum details will be merged with Total War Access if you register with the same email or username. For more information please read our FAQ’s here.

Categories

Modelling ranged units power - for fun!

13

Comments

  • Disposable HeroDisposable Hero Posts: 3,777Registered Users
    eumaies said:

    Well ok... I just think if you have a theory and univariate comparisons (eg range, dps etc) which you do that theory can stand on its own without arbitrary numbers built around it.

    Modeling is useful when you have data that is comparable. Ranged units in this game are so complex you have no chance of modeling it well without a lot of work and debate about assumptions which turns it back into a theory anyway.

    A healthy skepticism is fine, but I think you are overly eager to dismiss this completely and that you're doing it for the wrong reasons.

    There is absolutely no need to build a model on theory. You can do it by explicitly solving every equation describing the process. For 100% theoretic calculations I think the limit is like a dozen electrons. Everything else is different degrees of approximation. You can have a 99% accurate prediction model based on 0% theory and 100% statistics if you have a good enough dataset to base it on. For example weather prediction or molecular property prediction are very empirical. Any type of neural network simulation relies 100% on a training dataset which the networks connections are trained on by changing the weighting of all connections. There is 0% theory involved and such a prediction becomes a black box that works depending on how well it was trained.

    That's basically what we have here, a crude model that is perhaps based on 5% theory, based on how the game treats some things and how they are supposedly knit together. The rest is parametrization on the training dataset. The result is am extremely crude and wonky perhaps 70% accurate underfitted prediction model.

    It does not have 0% predictive power, I have actually shown that. The question is what it fails to predict and why. For this there are some major sources of error and noise, and the fitting method is faaar from robust. There are the two major weaknesses. Dismissing it completely is however not quite fair. Interpreting it with a very open mind on the other hand is very advisable. :smile:
  • eumaieseumaies Senior Member Posts: 4,497Registered Users
    My point is an extremely biased (ie fitted on the wrong variables or poorly parameterized) is worse than no model at all. Under the no brainer that every stat is positively correlated with power you can predict some true variation in unit power. But everything else just follows from your omitted variables and assumptions.

    For example there is no scaling factor for the diminishing returns of being a hybrid unit. So hybrid units will stand out. There is no leadership in your model, so low leadership units will stand out. Artillery could be wrong in various ways yet data from artillery pieces stats is biasing every coefficient in your model.

    Theory predicts all these breakdowns, so why even model? Model where you have valid assumptions and Conparable and complete data.
  • Disposable HeroDisposable Hero Posts: 3,777Registered Users
    eumaies said:

    My point is an extremely biased (ie fitted on the wrong variables or poorly parameterized) is worse than no model at all. Under the no brainer that every stat is positively correlated with power you can predict some true variation in unit power. But everything else just follows from your omitted variables and assumptions.

    For example there is no scaling factor for the diminishing returns of being a hybrid unit. So hybrid units will stand out. There is no leadership in your model, so low leadership units will stand out. Artillery could be wrong in various ways yet data from artillery pieces stats is biasing every coefficient in your model.

    Theory predicts all these breakdowns, so why even model? Model where you have valid assumptions and Conparable and complete data.

    I partly agree, but how much does it really matter why it works to the extent it works? Missing elements is one thing, but only if the unit in question stands out in that particular element. For example, this would fail horribly to predict the value of skirmish cav because it's not parametrized on speed above 50 and doesn't account for cb, ws and ma. Ws and ma should be included here too but it's not done yet and I probably won't bother, but we could safely ignore cb. It's partly but not entirely compensated for by over estimating the value of 360 for example. All in all it does a somewhat crappy prediction of the value of the container. As long as all key stats are well represented in the training set the predicted value is not random.

    If you look at the outcome, is there any specific one you think is completely bs wrong? If so we could speculate why it's off.
  • eumaieseumaies Senior Member Posts: 4,497Registered Users
    hmm i think i responded but it's not working.
  • eumaieseumaies Senior Member Posts: 4,497Registered Users
    I just don't see any reason why I would look at the model results at all. You could I guess label it as "power of units when accounting for x,y, and z, and ignoring a,b, c," but I don't need a model to tell me the answer to that. It would just a function of what x, y, and z are and what a, b and c are.

    I didn't say the predicted values are random. I said they're biased (technical term, not related to mindset). Biased is worse than random. If I was going to manually adjust post-hoc for the biases then I don't really need a model in the first place.
  • Disposable HeroDisposable Hero Posts: 3,777Registered Users
    eumaies said:

    I just don't see any reason why I would look at the model results at all. You could I guess label it as "power of units when accounting for x,y, and z, and ignoring a,b, c," but I don't need a model to tell me the answer to that. It would just a function of what x, y, and z are and what a, b and c are.

    I didn't say the predicted values are random. I said they're biased (technical term, not related to mindset). Biased is worse than random. If I was going to manually adjust post-hoc for the biases then I don't really need a model in the first place.

    Depends on what you are using it for and how systematic any bias is. Our application is "for fun" and not for diagnosing a stroke... And if the sum of bias is random it appears as noise and just adds uncertainty.

    But I'll put it to a test later for fun, without changing anything I will add a few units I forgot and see how they are predicted.
  • eumaieseumaies Senior Member Posts: 4,497Registered Users

    eumaies said:

    I just don't see any reason why I would look at the model results at all. You could I guess label it as "power of units when accounting for x,y, and z, and ignoring a,b, c," but I don't need a model to tell me the answer to that. It would just a function of what x, y, and z are and what a, b and c are.

    I didn't say the predicted values are random. I said they're biased (technical term, not related to mindset). Biased is worse than random. If I was going to manually adjust post-hoc for the biases then I don't really need a model in the first place.

    Depends on what you are using it for and how systematic any bias is. Our application is "for fun" and not for diagnosing a stroke... And if the sum of bias is random it appears as noise and just adds uncertainty.

    But I'll put it to a test later for fun, without changing anything I will add a few units I forgot and see how they are predicted.
    Have fun! :) Sorry to be so negative about it. I personally wouldn't reference in a balancing debate but to each his own.
  • Cukie251Cukie251 Posts: 920Registered Users
    I think it is a valid point that without knowing your exact weights and exactly what you accounted for, than its difficult to really ascertain the validity of the model. For instance its one thing to know that you modeled range, but its equally important to know the range compared to the weight on DPS, etc. The potential dps of a gun unit may be 2-4x that of an artillery piece, but depending on the weight it may be less valuable than it actually is.

    And thats not including things you didn't model in, an example is in the defensive category you have MD but didn't mention MA, a decent MA helps a great deal when fending off units like dogs, and not accounting this could put a unit like sister's of avelorn lower on the list defensively than they should be.

    When you combine all of these little factors its impossible to know if your model is accurate, even 70% of the way. Maybe you could link a google doc? It would be interesting to play with some of the values and see what it spits out.
  • Disposable HeroDisposable Hero Posts: 3,777Registered Users
    Cukie251 said:

    I think it is a valid point that without knowing your exact weights and exactly what you accounted for, than its difficult to really ascertain the validity of the model. For instance its one thing to know that you modeled range, but its equally important to know the range compared to the weight on DPS, etc. The potential dps of a gun unit may be 2-4x that of an artillery piece, but depending on the weight it may be less valuable than it actually is.

    And thats not including things you didn't model in, an example is in the defensive category you have MD but didn't mention MA, a decent MA helps a great deal when fending off units like dogs, and not accounting this could put a unit like sister's of avelorn lower on the list defensively than they should be.

    When you combine all of these little factors its impossible to know if your model is accurate, even 70% of the way. Maybe you could link a google doc? It would be interesting to play with some of the values and see what it spits out.

    The impact of range I think is the single most interesting part of this exercise. You can see it directly on page 2 here, the dps is multiplied by the value returned but that exponent. That is how heavily you have to weigh range in order to get artillery on the same balance base line as missile infantry. Artillery have so many drawbacks compared to hand gunners for example, both defensively and offensively (dps and accuracy both). The only upside is range basically. There is model killing, but there is also over killing. It's mostly range. That's why the newer monstrous archers and weapons teams are so powerful, they tap into that range without paying the premium. That is one of the few things I think this actually shows, it shows that it's very unlikely that both artillery and the newer long ranged missile infantry units are both balanced. I won't say the model shows which one of them is off but I can assure you it's not possible to get them all on the same base line with any reasonable tweaks.

    In principle I wouldn't mind sharing but I would need to rework it to be readable for anyone else and then proof read it like a million times because any typo would be used against me.... Let's see how much work I end up investing. :)
  • GeneralConfusionGeneralConfusion Posts: 936Registered Users
    So, one point that I'll drag up as an example of why I don't think it's valuable to put a lot of weight on this model; you said you've disposed of artillery damage by, essentially, counting 'overkill' damage as about equivalent in effect to killing models outright, and so just counted listed damage values as being applied.

    I think that's a big issue, and it partially explains why artillery tends to rate weirdly low in your model. The fact that artillery projectiles kill models on contact and pass through is hugely important; a cannon firing on a cavalry unit and killing 8 models is a very different thing, functionally, from a unit of handguns firing on a unit and doing equivalent damage but killing only 1 model outright, for example.
  • ParmigianoParmigiano Posts: 750Registered Users

    So, one point that I'll drag up as an example of why I don't think it's valuable to put a lot of weight on this model; you said you've disposed of artillery damage by, essentially, counting 'overkill' damage as about equivalent in effect to killing models outright, and so just counted listed damage values as being applied.

    I think that's a big issue, and it partially explains why artillery tends to rate weirdly low in your model. The fact that artillery projectiles kill models on contact and pass through is hugely important; a cannon firing on a cavalry unit and killing 8 models is a very different thing, functionally, from a unit of handguns firing on a unit and doing equivalent damage but killing only 1 model outright, for example.

    WLC wouldn't be used at all going by his graphs. I don't think there is anything wrong with making a graph to highlight the strengths of Jezzails. He could add some weaknesses of Jezzails to the model to try to even it out.

    They only have 18 of them, so they would die faster than units with 75 with slightly worse stats.

    Their collateral damage is a limitation.

    Their total damage from the ammo they have may not be worth their cost compared to Deck Gunners.
  • Disposable HeroDisposable Hero Posts: 3,777Registered Users
    eumaies said:

    eumaies said:

    I just don't see any reason why I would look at the model results at all. You could I guess label it as "power of units when accounting for x,y, and z, and ignoring a,b, c," but I don't need a model to tell me the answer to that. It would just a function of what x, y, and z are and what a, b and c are.

    I didn't say the predicted values are random. I said they're biased (technical term, not related to mindset). Biased is worse than random. If I was going to manually adjust post-hoc for the biases then I don't really need a model in the first place.

    Depends on what you are using it for and how systematic any bias is. Our application is "for fun" and not for diagnosing a stroke... And if the sum of bias is random it appears as noise and just adds uncertainty.

    But I'll put it to a test later for fun, without changing anything I will add a few units I forgot and see how they are predicted.
    Have fun! :) Sorry to be so negative about it. I personally wouldn't reference in a balancing debate but to each his own.
    Here's the test I talked about doing. I found 5 units that I had not added before that sort of fits the criteria of being ranged units: skink skirmishers and chameleons, shades, orc arrers and zombie gunnery mobs. They are just plugged into the old model.



    Skinks and chamels slightly undervalued could be attributed to poison, and chams have some hybrid character too which I didn't take into account yet.

    Also checked the average relative error in estimated value of the training set, and it's quite big, it's 21%. Represented graphically like this:



    So maybe I'll just chip in a little more time and add MA/WS too just for completeness. By just looking at it it seems to over all underestimate a lot of units with some hybrid character. I have avoided to select the obvious hybrid units though as the model did set out to look at specialized ranged units. If I do this it should in principle be possible to add in GW variants, corsair handbows etc. Maybe even the bone giant, but it's way outside the parametrization set so I'd be extremely surprised if that one falls within.
  • Disposable HeroDisposable Hero Posts: 3,777Registered Users

    So, one point that I'll drag up as an example of why I don't think it's valuable to put a lot of weight on this model; you said you've disposed of artillery damage by, essentially, counting 'overkill' damage as about equivalent in effect to killing models outright, and so just counted listed damage values as being applied.

    I think that's a big issue, and it partially explains why artillery tends to rate weirdly low in your model. The fact that artillery projectiles kill models on contact and pass through is hugely important; a cannon firing on a cavalry unit and killing 8 models is a very different thing, functionally, from a unit of handguns firing on a unit and doing equivalent damage but killing only 1 model outright, for example.

    Well, I just have to choose a way that makes it not depend on the target being fired upon. Killing cav models is great on a fresh unit, but leads to lots of lost damage on a half-health unit because many fewer models will be clipped. Same with accuracy scaling, I don't scale it to give full benefit of having the best accuracy, if I did that all archer type units would appear useless and only handgunners would be good... so I have to assume that the target is a rather large single entity. Anything else and it becomes very circumstantial....
  • ystyst Posts: 6,202Registered Users
    What is this total meta really. Its simply plotting wherever u want. Pretty much throwing the units u want nerf to the outlier for fun
    https://imgur.com/a/Cj4b9
    Top #3 Leaderboard on Warhammer Totalwar.
    Unit stats compare courtesy of Seal62 https://total-war-unit-compare.herokuapp.com/
  • Disposable HeroDisposable Hero Posts: 3,777Registered Users
    yst said:

    What is this total meta really. Its simply plotting wherever u want. Pretty much throwing the units u want nerf to the outlier for fun

    The stats of the units all go through the same equations and produces a meta score based on how much the stats "add" up to. It's described to some length on page 2 I think.

    Let's discuss Organ guns instead? The only clear outlier below the graph.
  • eumaieseumaies Senior Member Posts: 4,497Registered Users

    yst said:

    What is this total meta really. Its simply plotting wherever u want. Pretty much throwing the units u want nerf to the outlier for fun

    The stats of the units all go through the same equations and produces a meta score based on how much the stats "add" up to. It's described to some length on page 2 I think.

    Let's discuss Organ guns instead? The only clear outlier below the graph.
    Except you chose the stats to include or exclude. Why is leadership not included again?
  • eumaieseumaies Senior Member Posts: 4,497Registered Users
    and again, you don't need a model (which is incorrectly estimating artillery damage/value) to make an inference about organ guns, jezzails, or anything else. You can just compare stats and argue why they do or don't matter in game performance. This modeling just hides the underlying stats and produces "outliers" in terms of mystery meat inputs.
  • Disposable HeroDisposable Hero Posts: 3,777Registered Users
    eumaies said:

    yst said:

    What is this total meta really. Its simply plotting wherever u want. Pretty much throwing the units u want nerf to the outlier for fun

    The stats of the units all go through the same equations and produces a meta score based on how much the stats "add" up to. It's described to some length on page 2 I think.

    Let's discuss Organ guns instead? The only clear outlier below the graph.
    Except you chose the stats to include or exclude. Why is leadership not included again?
    If it makes you happy I can add it, it will not be very influential though. Not saying it irrelevant, but not very influential in relation to range, dps, ap-ratio, accuracy etc. It's also not really intuitive how it factors in so it will mostly be noise.
    eumaies said:

    and again, you don't need a model (which is incorrectly estimating artillery damage/value) to make an inference about organ guns, jezzails, or anything else. You can just compare stats and argue why they do or don't matter in game performance. This modeling just hides the underlying stats and produces "outliers" in terms of mystery meat inputs.

    You're allowed to be skeptic. There is no mystery of selective disfavoring of units going on here though. There is no stats on these units that are unique and not present in several other units, which happen to end up on the line together with the rest. The reason things end up as outliers here is that they have "too much" or "too little" value contained in their collected stats relative to their cost. The accuracy is not great, as expected, but it is systematic.
  • eumaieseumaies Senior Member Posts: 4,497Registered Users
    How would you even know if leadership is systematically correlated with cost to a meaningful degree?
  • eumaieseumaies Senior Member Posts: 4,497Registered Users
    There is no reason at all to include artillery in your model. They can only screw up every coefficient because you know you aren’t modeling what they cost relative to their actual real world damage. Their speed will be misleading as well.

    the fact that you would include data from units you already know you can’t directly compare with the stats you have, while also including in your model in general only those stats (from among all stats and factors that ca considers in costs) that you find most relevant makes this an enourmous case of completely inappropriate and out of context comparisons.

    This is the balance forum on steroids ;).

  • ParmigianoParmigiano Posts: 750Registered Users

    yst said:

    What is this total meta really. Its simply plotting wherever u want. Pretty much throwing the units u want nerf to the outlier for fun

    The stats of the units all go through the same equations and produces a meta score based on how much the stats "add" up to. It's described to some length on page 2 I think.

    Let's discuss Organ guns instead? The only clear outlier below the graph.
    They nerfed the Organ Gun 100g, your graph fails to show why that unit might have been effective.
  • ystyst Posts: 6,202Registered Users
    Cant work that way at all. If its about stats, sisters will go thru the roof, they basically murders saurus spear pretty decisively in melee.

    Jezz dmg is the worst of its kind. No one does so little dmg, basically when u reach 150m range every missile from basic to elite is going to do whooping +50% more dmg per volley than them. Until they keep firing beyond 200-250-300m they aint gonna be that useful.
    https://imgur.com/a/Cj4b9
    Top #3 Leaderboard on Warhammer Totalwar.
    Unit stats compare courtesy of Seal62 https://total-war-unit-compare.herokuapp.com/
  • Disposable HeroDisposable Hero Posts: 3,777Registered Users
    a unit.
    eumaies said:

    There is no reason at all to include artillery in your model. They can only screw up every coefficient because you know you aren’t modeling what they cost relative to their actual real world damage. Their speed will be misleading as well.

    the fact that you would include data from units you already know you can’t directly compare with the stats you have, while also including in your model in general only those stats (from among all stats and factors that ca considers in costs) that you find most relevant makes this an enourmous case of completely inappropriate and out of context comparisons.

    This is the balance forum on steroids ;).

    Not really, there's a lot of abstracting going on, but nothing wrong in principle. Perfect accuracy is impossible, but to some extent you can estimate value from stats. It's what we all do in our heads when we look at stats.

    There is absolutely no reason why artillery would not be on the same line as all other unit types. We plot cost vs value, and the units are supposedly all cost effective (balanced), they are bought for the same currency (gold) with the same limits (12400). If we can produce a half-arsed estimate of value from the stats they should all end up on the same line.

    In order to even try to do that there is abstracting going on. The influence of target must be abstracted, so we calculate a "potential" damage rather than exact damage vs every possible target, and the influence of calibration and accuracy is abstracted to reward good accuracy but not over estimate it. It's nothing strange, it's the only way you can do it. It's basically assuming a somewhat optimal target for every unit. In reality cannons waste damage on units while units waste damage on single entities due to accuracy. Powder do so less, but also over-kill more. It's all some sort of balance, but at the end of the day we deal with potential damage and trust the player uses his unit as efficiently as he can. There's nothing fundamentally wrong with that.

    There is also nothing inappropriate with this. I do a model, I am open with how I make this model and show the results. If you must interpret this as an attempt to make @CA_Duck without a job, maybe this discussion is not aimed at you. :smile: There is no such ambition. I am playing with numbers and plotting the results, the some responsibility lies with the observer not to go nuts about me trying to shove something down his throat. I am not. If you get provoked by this little exercise maybe just ignore it would be an option? I don't understand why you get so eager to dismiss this in its entirety. I am the first one to admit there is no perfection to be found here. It's all about approximations, generalizations and abstractions.
  • CanuoveaCanuovea Posts: 13,368Registered Users, Moderators
    Its all weird stats voodoo to me.

    However I would be interested in actually seeing what all the other nodes actually are. Especially that one at the bottom, wow.

    Seems mostly decently behaved in there at the moment.
    -Forum Terms and Conditions: https://forums.totalwar.com/discussion/172193/forum-terms-and-conditions#latest
    -Using all caps is the equivalent of shouting. Please don't.
    -The "Spam" flag is not a "disagree" flag. Have a care.
    -...No, no the "Abuse" flag isn't a "disagree" flag either!
    -5.7 Summon a moderator if someone seems to be out of line, or use the report button. Do NOT become another party to misbehaviour
  • Disposable HeroDisposable Hero Posts: 3,777Registered Users
    yst said:


    Jezz dmg is the worst of its kind. No one does so little dmg, basically when u reach 150m range every missile from basic to elite is going to do whooping +50% more dmg per volley than them. Until they keep firing beyond 200-250-300m they aint gonna be that useful.

    That is true for all artillery as well. They pay a big premium for having such long range, while Jezzails and Bowshapti don't. That's actually the one thing that this model has really showed me, it's how much value you have to assign range in order to compensate for the weaknesses that artillery has in general. If you don't assign a lot of extra value for extreme range, then all artillery in this game appears severely under-powered, because they don't have the raw damage output to compensate for lack of mobility, armor, MD, shields, 360, loose formation, stalk and whatever else the missile infantry in general has going for them, in addition to good dps. The only thing they don't have that artillery has is range (and the small leadership debuff if we're picky). I don't think model killing in its own right makes up for this since both cannons and catapults cost about the same, and one kills models and one kills units so to say, and then we have helblasters and organs that fire more like a unit with their multiple shots. What they have in common is much more range than the traditional missile infantry. So, no matter how I twist and turn this I have to conclude that range is very highly valued, otherwise artillery has no chance of being cost efficient.
  • ystyst Posts: 6,202Registered Users
    edited June 11
    Range is simply what it is, I mean hell if u wanna put a huge value on it, why not try blessed treb. Its $850 with 440m range. If u can prove them having a chance, ever, of being cost efficient, great.

    Plenty of useless units that u dont get it cost efficient, trash tank, majority of catapults, trash guards, even hell blaster, unless u can wipe out 2 grp of elite inf, deleting 2 archers still net u a loss trade.

    I mean sure if u wanna play art and plot charting game.

    Put trash tank and luminark on the chart.

    -edited
    https://imgur.com/a/Cj4b9
    Top #3 Leaderboard on Warhammer Totalwar.
    Unit stats compare courtesy of Seal62 https://total-war-unit-compare.herokuapp.com/
  • CanuoveaCanuovea Posts: 13,368Registered Users, Moderators
    @yst

    I just said that things were doing okay here, but some of these comments are borderline. Please be careful.
    -Forum Terms and Conditions: https://forums.totalwar.com/discussion/172193/forum-terms-and-conditions#latest
    -Using all caps is the equivalent of shouting. Please don't.
    -The "Spam" flag is not a "disagree" flag. Have a care.
    -...No, no the "Abuse" flag isn't a "disagree" flag either!
    -5.7 Summon a moderator if someone seems to be out of line, or use the report button. Do NOT become another party to misbehaviour
  • Disposable HeroDisposable Hero Posts: 3,777Registered Users
    Canuovea said:

    Its all weird stats voodoo to me.

    However I would be interested in actually seeing what all the other nodes actually are. Especially that one at the bottom, wow.

    Seems mostly decently behaved in there at the moment.



    This is the best I can do without manually putting all labels. :smile:
  • Disposable HeroDisposable Hero Posts: 3,777Registered Users
    yst said:

    Range is simply what it is, I mean hell if u wanna put a huge value on it, why not try blessed treb. Its $850 with 440m range. If u can prove them having a chance, ever, of being cost efficient, great.

    Plenty of useless units that u dont get it cost efficient, trash tank, majority of catapults, trash guards, even hell blaster, unless u can wipe out 2 grp of elite inf, deleting 2 archers still net u a loss trade.

    I mean sure if u wanna play art and plot charting game.

    Put trash tank and luminark on the chart.

    -edited

    It's limited to "ranged power", so I have selected units that have a strong focus on dealing direct ranged damage and not being hybrids. I don't include things with explosion damage since I don't know how I would estimate their damage properly. Luminark could possibly fit one day, but is has outlaying properties like cost and being a chariot, so basically that's just trying to break it. There's a balance, I can't just add more and more correction factors, because it just becomes more and more under-fitted, so it can be better to just conclude that for example gutter runners are well blow the graph, partly because there is no correction for the snare ability. There is no point adding it, because it's unique and whatever value I assign it will completely determine the result. Poison I could add because many units share the poison ability, but I didn't do that yet.
  • ExarchExarch Posts: 575Registered Users
    I think Eumaies makes a good point about the artillery, which exists in a very different regime to the rest of the units (long ranged, but very weak defensively, low shot count, but with high damage and splash damage). When you exclude the 'monstrous' data points, the only long range (<190m) units left are the artillery.

    When you exclude the monstrous units, all the high ranged data points will be artillery, so the exponential term will be heavily biased towards this unit type. When the long-ranged monstrous infantry are introduced, they appear in a region of the training space conditioned only on artillery, which makes comparison difficult.

    I really like the general approach. I would especially like to see the missile infantry only plot I think, as most of the units within that regime behave much more similarly- the main split there being bow vs powder.
Sign In or Register to comment.