Hey again everyone! 'Tis once again the season for Josh to apologize for waiting too long to post a new devlog... Honestly, time just flies when you're having fun working on LT. And we've been doing a lot of that.
Talv, in his usual beyond-his-years Talvish wisdom, has advised me that it's more important to be consistent with devlogs than to be exciting, flashy, etc. I feel like I've heard that from you guys before. At some point, it may actually sink in to this thick head of mine At any rate, I'm going to turn over the 1-hour hourglass (that's not redundant because I also have a 30-minute 'hour'glass...) on my desk and force myself to finish this log before the last grain hits the bottom.
To accomodate the expedited writing process, I'm going to bullet-point the list of what I've been working on, then expand on a single element of focus (whichever I feel is the most important at the moment). We'll see how it goes!
Since Adam has already been kind enough to give us quite some detail on what he's been up to, I'm just going to go over my own work. So...over the past month, I've:
- Implemented several new components for entities, most of which are in preparation for / in service to implementing gameplay logic
- Implemented a mod-loading system that performs validation and loading of isolated mod files, which of course means that I also...
- Defined a mod file format and started writing logic in mod form; ideally, most if not all of our actual gameplay logic will be written like this, both for our own convenience in being able to easily plug/unplug pieces of gameplay, as well as to ensure uniformity with respect to 'built-in' content vs. mod-added content
- Implemented an event-handler architecture for effortlessly broadcasting events as well as hooking listeners to respond to said events
- Worked with Adam to implement first pass of 'job' system, which is, essentially, the 'code' analogue of an ECS
- Implemented several jobs / pulled much of the existing LT logic out into jobs, resulting in isolated, efficient, and easy-to-reason-about logic
- Implemented (naive) collision detection and resolution (as a job) -- had lots of fun with large quantities of asteroids bashing into one another
- Tackled many, many performance optimizations accross the board, attempting to ensure that we stay lean-and-mean as we push more content into the game; some examples are LOD meshes & automatic generation thereof, fast font glyph caching, fast vertex AO computation for SDFs, and more...
- Explored building a simple editor program for expediting the process of adding components, entities, and logic + rolling those changes into a mod file and testing it in-game (heavily-inspired by the TES games and the legendary 'TES Construction Set' from back in my Morrowind modding days)
- Began work on version 2 of the jobs system, which will, hopefully, unify how we write the heavy, running-almost-all-the-time gameplay logic and the sparse, happens-only-under-specific-conditions gameplay logic
- Got the 64-bit Windows version of LT built & running
Jobs and Gameplay Logic
With our CTypes system firmly nailed-down and providing an excellent solution to how we handle data in the game, we're now looking to solve the equivalent problem with logic/functionality/code. After all, there is only data and function, nothing else With entities, we asked the question: how can we efficiently, easily, and extensibly define the format and manage the lifetime of the data that keeps track of an entity? How can we do so in a way that will support many diverse types of entities, while also supporting the potential to have hundreds-of-thousands of these entities in play, and perhaps even thousands or tens-of-thousands going into and out of existence in a matter of fractions of a second?
To those questions, we answered and answered with vigor in the form of an ECS, which became an ECS v2, which became CTypes.
Now we ask the analogous questions with respect to logic: how can we efficiently, easily, and extensibly define the format and manage the lifetime of the logic that runs on and creates the characteristic behavior of an entity? How can we do so in a way that will support diverse types of entity behaviors, and potentially hundreds-of-thousands of such behaviors in play at once, and potentially tens-of-thousands of these behaviors stopping, starting, or being replaced with a different behavior at any given moment?
I've touched on partial answers to this question in previous logs. Part of the answer is: exploit the sparsity of game logic to do 'as little work as possible' at every moment of time (note the symmetry with how we store 'as little data as possible' in CTypes by using native memory, pools, static typing, etc.) This means, for example, that an AI player who is travelling to a location does not enter a state where the script constantly checks whether the player has arrived at the destination. This is wasteful and fails to exploit the sparsity of the situation (the probability of arrival is exceptionally low on any given frame). Instead, the script informs the engine of the specific condition(s) upon which it should be resumed: "Dear engine, I am writing to inform you that I no longer require CPU cycles until my client, Sir NotEdisonTrent III, aka 'Trent', arrives within 100 metres of the asteroid @ memory address 0xA513701D. Alternatively, if my client should be the victim of an attack, I would like to be notified in spite of not having arrived at said asteroid. Thank you for your time, see you in a few billion cycles. Love, script/ai/TravelTo.lua." If the underlying logic engine is clever with respect to how it services such a request, then the difference here is the difference between being able to run a few hundred AI units vs. being able to run a few hundred thousand (not that we need to run a few hundred thousand -- but keep in mind that if we consider all the other logic going on in the game, then indeed, we are looking at on the order of a few hundred thousand bits of logic). It's an orders-of-magnitude subject, like many things in computing.
We already have a job system for easily running logic on large batches of entities that is very cache-friendly as well as easily-parallelizable. At the moment, I am working to fill the final piece of this puzzle: enabling said system to accommodate highly-sparse logic like the example above. The good news is that I have already designed (on paper) the architecture for the system. I'm working on the implementation as we speak (well, I mean, by the time you read this I'll be back to working on it)! Interestingly, implementing such a system is, in many ways, symmetric to designing a robust OS kernel, as many of the concepts have parallels in OS design: a piece of logic is a thread, sparse logic is a thread that waits on condition variables (a fundamental synchronization primitive used in low-level kernel / driver implementations), thread groups are woken by condition variable signals / broadcasts, a job is a process, each instance of a job is a thread, job state is thread-local storage (TLS), ...and so-on (of course it's much easier to do for a game than for an OS, don't be alarmed by my comparison!) Lucky for me, I did manage to pass the operating systems course at uni before dropping out (Side-note: our professor was the guy who made one of the first scripting languages, Tcl. He was the best prof I ever had.)
An infrastructure for handling game logic is the last remaining element that I see between where we are now and a time where we can iterate wildly on shippable gameplay code. It's all very exciting
Success! Would you look at that? I finished just in the nick of time (ok, maybe by the time I proof-read and actually post, I will have gone a few grains over ). I realize that you all probably would like more details on some of the other elements, but I'm clearly unable to give 'brief' summaries of things when I'm excited about them. And I'm excited about a lot of those bullet points. At least this way I'm able to write logs in a reasonable amount of time while still providing juicy details + a high-level picture of progress.
I really would have liked to explain a bit more about my work with the proof-of-concept editor, but alas, it would require another turn of the hourglass. For now, I just want to mention that it is being 'mostly shelved,' as the goal was to determine whether the cost-of-implementation of such a tool would be less than or greater than the time saved developing. From my exploration, it seems the cost-of-implementation would not justify the LT-dev time savings, so we will most likely not see continued work on the editor. I can imagine some pragmatists out there cheering and some dreamers out there crying Well, come on, we have to save some things for LT 2, right? That being said, it still has some useful features: we can use it to easily get output from LuaJIT tracing & assembly dumping on Windows, where it's more difficult to do so than on Linux.
Now then. I'm going to hold myself to posting the next devlog next Friday. Period. Next Friday I will do the same thing that I did today: turn the hourglass, and write. Talvieno is right, as always: consistency is the key. And if I just wrote a devlog more often, I wouldn't have to squish so darn much into one anyway!
Oh dear. It appears next friday is the 13th. Ha. Oh well, numbers are just numbers, and, after all, 13 is both a prime AND a member of a twin prime pair, so I don't see how it could possibly be unlucky. See you all on the 13th...
I took a few screenshots to accompany the wall of text: Limit Theory Devlog Gallery ~ October 4, 2017
Wireframe view of an asteroid field, where you can see the LOD mesh optimizations (notice the wireframe density remains approximately-constant in screen-space, despite asteroid size or distance. Constant screens-space detail is 'ideal' LOD.)
I wrote a simple mod to shoot asteroids out of my ship. I used it to test the new collision detection & resolution job. It's fun blowing apart a big sphere of asteroids with a spray of little ones (It's kind of hard to see without an animated GIF, but I'm not nearly as good as Adam at making those..)
Using Plume (the concept LT editor) to get LuaJIT assembly dumps on Windows: