For the next blog it may be interesting to read about the details of the benchmark and the results of LuaJIT. Additionally, why the previous attempts failed to scale. For instance, why isn't the simulation LOD the answer to the problem? I imagine you want to be able to simulation some set of agents at full resolution, but it's costing too much frame time? Are any parts of the simulation multi-threaded? Why not scale back the simulation requirements for 1.0?
All good questions. Right now the goal is only to get those agents that must be simulated at full resolution to be performant; I do not even count the LOD-simulated ones towards the 'problem,' since the bulk of the work is in the full-detail sims. Multi-threading is difficult in game logic simulation but is being explored currently. It is very possible that simulation size will be scaled back for 1.0 if that makes the difference between working and not-working (no solution has gotten 'close enough' to be at this point, though).
Also, to ask the difficult question; why isn't the shortest path to completion to continue writing the game in c / c++? I know the codebase is large, but modern IDEs can help manage the complexity; code completion and documentation go a long way on large codebases. Too much boilerplate? Compile times too slow? I understand scripting languages are more expressive, but the opportunity cost is pretty high; 2 years in c/c++ or 2 years experimenting with scripting languages. Not trying to criticize, I feel like you needed some one to pull you out of this black hole earlier. Hopefully, LuaJIT yields some fruit.
No I understand, it's a good question to be asking. Frankly I don't know the answer -- could I have spent these two years slimming down the C++ codebase, refactoring to make it easier on myself / make compile times faster (yes, they were a problem)? Maybe so, maybe it would have been enough. Maybe it wouldn't have. As I've explained, my best intuition at the time told me that it wasn't going to happen with that development style. But it's honestly not really worth asking at this point, because I've already gone down this other road. Thing is, though, I've learned what I feel to be a lot
of invaluable lessons with all these attempts and failures. I feel much more equipped to 'handle' LT, even if I don't have the right solution nailed down just yet, I have a much more vast set of tools to work with. I think it's good that I finally came out of my rather narrow world of C++ and only C++.
FWIW, LuaJIT appears to be yielding more fruit every day. Though of course we must not get our hopes up
* ping jblow for an early release of JAI
An interesting language no doubt, but I have very little interest in using it TBH -- I'm sure he's designed it for his needs...but personally, watching him program with it, I think his needs are quite different from mine.
I'm happy someone other than Dino has brought up multi-threading.
Threading is being explored. As I've said before, though, (and I don't think some people are hearing me when I say this), threading game logic is not like threading physics simulations or math computations or what-have-you. Game logic is massively interdependent, a perfect storm of a situation for death-by-mutexes or plain old memory corruption. Throw in a scripting language on top and things get even more complicated. Now we have a script interpreter state that can't just be launched on multiple threads. We have to spawn a new interpreter state for each thread -- any shared data now has to be managed somehow by the underlying engine, since (in the case of Lua) the scripting language provides no facilities for sharing data with other instances (in the case of some other scripting languages, it's not even possible due to the global interpreter lock (GIL)).
But yes. It's being explored.