Exactly. My gut tells me that they are doing a good job with the tools they have, but there is a bit of a black hole. Specifically where they think the networking tests are adequate, therefore detailed reporting for the consumer, limitations to players, and mechanisms for the developer to help ensure the good code is not twisted into something else as it spans the internet are simply not in play to the extent they should be.

They have to realize that "quality" testing on a LAN is experience as the connection provided P2P. I'm not a networking guru... But play testing on a LAN with simulated lag.. Does that imply they are using a server? And their testing isn't p2p at all?

Ultimately Bungie would have to answer. Based on my limited experience in a similar field from many many moons ago...a simulation is a simulation. The hardest part about the simulation is that it is typically not dynamic but more static. e.g. you would set it to simulate x or y condition and test. The internet simulations of today would need to simulate a to z conditions and randomly and sometimes wildly fluctuate in order to get a 'better' simulation. This would still not accurately reflect real world conditions as the network hardware of the internet that the code passes through may react or have 'bugs' that affect how the data flows which is almost impossible to simulate. I'm not genius or expert on this stuff, but my QA and TQM background tell me perceptions of information and facts presented are not matching the real world experience.

Their test set up is a bog standard LAN with P2P hosting and up to 100ms (tester selectable) latency. Reading between the lines the test latency setting is static not dynamic. [i][b]Source[/b] : Sage Merrill Bungie Sandbox Lead Crucible Radio Dec 2015[/i]

That. Right. There. Definitely part of the issue but not all of it. Based on just that simple fact, I'm willing to go on a limb and suggest that there is no method to simulate anything more than latency let alone the missing method to collect end user data/loss/corruption issues.

Bungie game test process (pieced together but pretty much industry standard) is; 1. Iterative bench testing of components pre-assembly 2. Build version bench testing in lab (simulated latency) and in house play testing. 3. Multiple Builds emerge and are tested per item 2 to get Alpha build 4. Alpha build testing in lab, I house play test and user group play test (7th Column) 5. More builds and testing to get to Beta 6. Beta undergoes testing per item 2 7. Beta is released into the wild - millions of users and every aspect of game data including latency trace routes etc are recorded and analysed 8. Beta is adjusted using data at 7 further testing per item 4 occurs to get release version. NetCode and internet "navigation" receive an extraordinary amount of attention at multiple steps through the build process. You are probably aware of the ground breaking architectural Bungie developed for Destiny so you will probably have a sense of the work required on NetCode and Internet "navigation" inherent to develop the group up system. Further evidence for this abounds but can be seen in the level and speed of work done by Bungie with various global ISP and tertiary institutions to resolve extreme performance issues experienced over tertiary networks when Destiny was first played in the dorms of US and Europe Oct/Nov 2014. Trust this sheds light - there is much about this matter available in the public donation. You may enjoy researching the matter for your self. Your research will help you further develop your hypothesis. Good luck and thank you again for a fascinating thread. [b]EDIT[/b] : a further snippet regarding your sense that Bungie may not measure jitter. This is one of many fundamental network matters and well understood by all network engineers. A quick review of the Bungie.net help sub forum for example confirms this is so as do many publicly available sources such as interviews with Bungie sandbox, pvp leads and studio heads, positions vacant adverts under experience required sections and by reviewing various GDC presentations Bungie has given over the years. I think what you are zeroing in on is less "Bungie don't understand how to design NetCode to deal with Internet and user setup vagaries" and more "are Bungie test processes simulating real player conditions closely enough to get the NetCode setting just right" - is my understanding correct ?

Excellent response. Thank you. It helps highlight the need for more advanced testing tools as more and more people use the internet in a online playing experience. I can only hope they take their networking code & testing to the next level (what ever that might be), although the demand for dedicated servers to handle some of the know issues has been discussed by the destiny population.