Really exciting week; the whole team was on-point this week if I do say so myself! Quite some major developments With any luck I won't spent so much time on the log this week, since there's valuable work to be done and I'm under a lot of time pressure here with this whole concept of 'getting certain things done by a certain time' -- a strange and foreign concept that Adam has convinced me has substantial value to it :V
In the last log, I talked about how we had to go back to the drawing board with our entity component system, since, unfortunately, things were starting to break down in LuaJIT, necessitating pulling work over to C. Most of my time has been spent this week nailing this problem. On Thursday, a turning point came when I had finished enough of the system to start running initial tests in LJ. For me, it was one of the most exciting moments in months when I saw the results. Basically, our ECS design works really, really well AND interfaces with LuaJIT REALLY well. The C itself runs faster than I expected. But the LJ <-> C interface? Multiple orders of magnitude faster than I thought possible. It blew my expectations clear out of the water. So, let me lay down some fun facts about where we are with respect to the system that will support all of the entity data in LT. First, the C-side:
- Logic that runs on the C-side is stupidly fast...
- ..it is also easy to thread. I'm already threading the C-side logic. Yes, LT is actually going to use your cores
- In C we can perform non-trivial, per-frame logic on >100K entities @ 60 FPS with a good deal of headroom on a modern CPU (using threading).
- Depending on how heavy the logic is, we can perform it on the order of 10ns to 100ns per entity. Kinematics updates, for example, are currently clocking in at ~30ns (3ms for 100K entities).
- All of this is based on using non-hard-coded entity types. That means new entity types added by mods enjoy the exact same performance characteristics.
- New component and entity types can be modified and added dynamically. The ECS was designed from the ground-up to ensure that we were not hard-coding anything that we didn't absolutely have to.
- Creating and destroying entities is virtually instantaneous due to the memory pool design I built for entity data storage. 10K missiles created in one frame is boring work for ECSv2.
- Logic that runs in LJ is 'on the order of' C speed (!!!!).
- LJ logic can update tens of thousands of objects per frame without issue. This will improve when LJ gains threading.
- ..oh yeah, since all data is stored and managed C-side, we can thread the Lua as well
- Lua code accesses and modifies entity data natively, with zero overhead, exactly like C code would. In other words, myShip.Integrity.health = myShip.Integrity.health - damage. We can write that in Lua, and the instructions emitted by LJ are 'basically' the same as what Visual Studio / GCC / Clang would emit.
- Speaking of emitting, all Lua code is currently running as JITed assembly in my tests. The new ECS <-> Lua interface was designed to ensure that LJ is able to JIT everything, and never have to fall back to the interpreter (which is why we can achieve C-like speed.)
A huge weight has been lifted from my shoulders. I can now say with great confidence that C + LJ is our solution. I can now write gameplay code with confidence that it won't be thrown away when I figure out that it performs poorly. Really, this is just a tremendous leap forward Adam and I are absolutely stoked that our design hit the nail on the head.
Adam: BSP Trees
Adam has spent the week becoming the resident master of BSP trees and narrow-phase physics. He's been really, really thorough about implementing them in such a way as to avoid common pitfalls (BSPs are well-known for their robustness issues -- it's easy to get them to 'mostly work,' another thing entirely to get them to 'really work'). At this point, we've got fast and accurate raycasts on complex geometry thanks to his BSPs. But it's not over yet: with robustness locked-in and hammered-out, Adam's going to start working on optimizing the BSP construction algorithm to ensure that we get the best performance possible for collision checks against annoyingly-complex objects like stations.
I'm really happy that someone on the team is taking the time to understand the nuances of BSPs (and, in general, robustness in bounding volume hierarchies). I'm certain it's going to pay dividends. After all, with the ECS now consuming so little of our frame time, the next-biggest performance hog is likely to be physics, and Adam's on track to make sure we cut it down to the same blazing efficiency that we managed to pull out of the ECS.
It's also worth mentioning that Adam is really good at building small testbeds that visualize results and show us when things are off, even if only by just a hair. This is exactly the kind of thing that I wouldn't do if I thought my implementation was 'mostly working,' and exactly the reason I was impressed with his rigor from week 1. Thanks to these visualizations he's caught a lot of edge-cases and robustness issues that would likely have gone unnoticed otherwise.
Sean: Colony Dynamics and Working Market Simulations
Sean continues to delve into all of the high-level simulation workings of LT, still focusing primarily on colonies. This week, however, I sat down with him for quite some time to explain the way the 'real' markets work in LT (bids/asks), the way it relates to AI, how colonies play into that, etc. I implemented a few basic objects for him in Lua to help with his simulation, the most important of which was a quick implementation of a dynamic market that keeps track of bids / asks, resolves them in the correct order, etc.
From there, Sean has been able to set up his first working economy simulations! At the moment, he's looking at a single, on-colony marketplace and how the colony's supply & demand drives the local market, as well as how external factors influence it. He's also experimenting with the pricing/valuation algorithms and how the AI's ability to changes its own idea of how much any given item is worth affects the global simulation. It may be a small-scale simulation, but to me this is a major step: from here, all he needs to do is continue adding elements to the simulation, studying how they affect it, and adjusting his course appropriately. I'm eager to see this work take off!
As you can see, the LT train is showing no signs of slowing down anytime soon! I'm pretty amazed at how much we've burned through in only a few weeks. But the burning must not let up! In the coming week, I'll be attempting to close out ECSv2, with a focus on continued performance studies in LJ. With luck I'll be starting on a new milestone near the end of the week. Adam is looking to have sphere and bounding-box intersection tests working on our BSPs sometime in the coming week, and will hopefully also begin work on optimizing BSP construction. As for Sean, I'm looking to see the market simulation blossom as more and more of the 'real LT' mechanics are factored in and added to the equation!
See you all next week