Hey again everyone! Hope you all had a good Thanksgiving (for those to whom it is applicable). I certainly ate my fair share
It's been a productive few weeks since the last log (minus Thanksgiving break), but last week, in particular, saw two major breakthroughs that have me really happy with the current state of affairs
Major Progress Since Last Time:
- Major breakthrough with being able to use Lua data & functions from our C subsystems
- Used the above to implement a C-side scheduler that schedules Lua logic for execution; significantly faster than previous implementation
- Ported planets & atmospheres
- Integrated Loom for advanced inspection of LuaJIT traces and assembly
- Implemented several new algorithms for procedural geometry library, touched on in Lindsey's latest log
- Loads of quality-of-life improvements to how 'game-level' Lua is written
- First-pass design of mod format taking into account all existing support systems
We're All Modders: Hammering out LT Mod Format for Building Gameplay
At this point, we've essentially got all of our core support systems working. By that, I mean that we have either Lua-level or C-level (in some cases a mixture) mechanisms for supporting almost everything gameplay code could want to do: from defining new datatypes that can be used with the utmost efficiency, to adding and/or modifying existing ones, to writing gameplay code that needs to run in response to certain events, or needs to be scheduled to run at certain frequencies, to writing new methods that can be called on the aforementioned datatypes, to defining jobs that batch-process large chunks of entities that match a certain filter, and so on. Where there was once a wild-west of Lua trickery, there is now a solid set of production-ready, battle-tested system. So, what remains? Naught but the orchestration of said systems to compose the symphony of gameplay that is Limit Theory
So, it is time that we once again turn our attention to the format for this orchestration: the Limit Theory Mod Format. I have expressed many times that I wish for our vanilla LT gameplay-level logic to be written in exactly the same manner as any other mod, and I'd like to stand by that -- both for the sake of making it easier on us devs (since we'll have a well-defined, decentralized framework for incrementally adding modular bits to the game), as well as for the sake of ensuring that LT is not just moddable in theory, but is built on the very proof that it's moddable in practice.
Thus far, my solution is unfolding as a two-part mod format, where each part may exist zero or more times within any given mod file. One part is data, the other part is code. Data sections use a very simple 'data description language' to perform the aforementioned orchestration of our system. A data section for implementing an inventory might look like this:
Hopefully it doesn't take too much brainpower to figure out what's going on here: we've opened up some object scope called 'Ship,' declared that it has an Inventory, and then gone on to define Inventory itself. Inside Inventory, we can see three types of 'declarations,' each of which corresponds concretely to a programming concept and each of which will be used in a different way to feed our underlying systems to build the required functionality.
As the programmers among you can likely guess, 'has' declares a field/member variable, 'can' declares a member function, and 'when' declares an event listener (a piece of code that will run in response to a particular 'event' that may occur on the object). The syntax of it all is just a sketch, so let's not concern ourselves too much with quibbling over whether 'has/can/when' is too cute / whether semicolons are the bane of mankind, etc
The point is that we have a portion of the mod that's using a very simple 'data' language to inform all of our heavy-duty systems of how to make inventories work. The 'has' statements inform our CTypes system about fields, address, alignments, etc. The 'can' is informing our more-general type system about the presence of callable member functions. The 'when's are informing our event handler system about events and their listeners.
Then, of course, we must write some Lua to make it all work:
And this looks more like what you would expect from standard Lua code. Here we're just implementing the functions referenced in the data section with standard Lua + some of the extra features added by the LT engine. Easy! All the 'bigger picture' elements that dictate when, where, and how a particular function runs are specified in the data section (e.g. addItem will be a method that can be called on any object with 'has Inventory', so that in other code we may say myShip:addItem(DenseTalvienite, 17) for example). Hence, much of the need for a modder to 'worry about' the internal mechanisms used to make LT performant (you know, the ones I talk about in pretty much every devlog ever?) is alleviated. Granted, they still exist and may still be interacted with at the code level, but for the majority of gameplay functionality, doing so should be unnecessary.
The Lua mod loader that parses all the mods into a format that can then be shipped off to the appropriate engine-level systems is already implemented; what remains is to do the shipping-off bit and to move existing gameplay code into isolated mod files where appropriate. I'd like to have this done by the end of this week
In response to anticipated concerns about 'oh no another language / LTSL / JOSH BE CAREFUL' : Rest assured, the goal is entirely different here. I didn't define a new scripting language; this is just a very simple data-description language that helps us connect all the moving pieces of our mod! It can be parsed with ~20 lines of Lua and the lovely string.gsub function. Anything more complex is unnecessary. I am not rabbitholing, I promise
Side-Soliloquy: Why So Much Concern for Moddability?
This bonus Josh-soliloquy is an optional part of your devlog experience
Spoiler: SHOW
I've left out all talk of a 'breakthrough' that happened last week involving Lua data playing nicely with our C systems. It's technical and I've decided to err on the side of not-so-technical this week, but, briefly, I'll just say that this breakthrough has opened up a lot of fantastic augmentations to our existing systems, as well as tremendous new flexibility in splitting data & logic between Lua and C (using whichever makes the most sense for the task at hand). Previously, Lua->C interop was easy. LuaJIT's FFI makes it basically effortless. But going the other way? Accessing Lua-side data / calling Lua-side functions from C? It was not so easy when that data/function could be arbitrary and needed to live alongside supporting C data (think: sparse logic scheduler). Since Lua is GC'd, we can't get pointers to Lua data, and this is a major annoyance about which I've ranted before. However, there's a function that I overlooked until last week...hidden in the Auxiliary Library API...luaL_ref...and it solves all my problems. *Slams face vigorously for having overlooked this gem.* At least I know now.
Until the 'morrow! (Well, actually I plan to do my next log on Friday. If I'm 'close but not quite' with respect to the mod system, then it will be Monday. Stay tuned!)
PS ~ Sorry for lack of screenies, but most of this work wasn't amenable to it. Besides, you guys just got a whole boatload of them in Lindsey's last log!