Return to “Dev Logs”

Post

Friday, August 18, 2017

#1
Friday, August 18, 2017

Hey all!

As usual, I apologize that these logs are doing a bit of temporal stretching lately -- honestly we're just really, really busy working on LT and it's tough to pull myself away these days, so consider it a good thing! When we last spoke, several major advances had set us up for fruitful weeks to come. I'm pleased to report that they were, indeed, very fruitful.

At the moment, I can summarize our work by saying: we're dangerously close to shifting the majority of our efforts to sprinting on LT gameplay code in Lua. And we're doing everything we can to close the gap.

Seriously, the 'road to (performant) gameplay' has been way longer than anyone wanted. But the end is in sight :)

---

Josh

  • Finished implementing zero-cost Lua wrappers for engine functionality (mentioned in last devlog; means we're getting superb performance when interacting with the LT Core)
  • Implemented enough of the foundational Lua code that we can all work on LT in Lua at the same time
  • Continued careful analysis of LuaJIT debug output; continued to ensure 100% JITing to assembly; continued to learn more about the trace compiler and how to beat it into submission :ghost:
  • Hooked up our ECS to Lua + wrote the convenience wrappers that allow it to be used really easily (note: we now call it the 'CType' system instead of 'ECS', reason being that it is actually more general and flexible than an entity component system, although it can and does serve as one -- I'll explain more below!)
  • Ported Lua deferred renderer; modified to take advantage of all high-performance constructs
  • Made major improvements to our logarithmic z-buffer code while doing the above (this is the code that allows us to render objects at massive scales & distances without issue)
  • Implemented a work-in-progress event system in Lua for use in gameplay code
  • Implemented Lua wrappers for efficiently dealing with SDFs and OpenVDB (the library mentioned in the last log that we now use for extracting meshes from distance fields)
  • Implemented CSG functions on SDFs that are roughly 10x faster than OpenVDB's defaults, allowing complex, detailed meshes to be built very quickly (read: better procedural mesh algorithms coming soon!)
  • Wrote a simple pathtracer for fun since graphics Josh has been repressed :monkey: Not totally irrelevant to LT, though, as we could perhaps use it for nice renders of procedural assets or for having a 'ground truth' reference for our physically-based rendering code
  • Built several Lua 'template' applications that allow us to easily various, specific pieces of functionality really quickly (just like the old LTSL sandboxes!)
  • Loads of other things that I'm forgetting

Adam

  • Started working in the Lua side of the LT codebase!
  • Put in the time to learn about all of LuaJIT's lovely 'idiosyncrasies'; Adam is now 100% capable of helping me deal with the technical side of LJ
  • Used LuaJIT magic to implement the first parametric type in our CType system -- the arraylist! This means that not only can we define and use new, native datatypes in Lua, but we can also have what amounts to native std::vector<T> and so forth of them. I know I've beat this horse to the ground, but seriously, the power we now have to create and use types that are as fast as compiled code simply can't be overstated! Memory management is by far the weakest spot in Lua's performance, and we are almost entirely side-stepping it.
  • Hooked up BSP code to Lua and used it to implement picking (being able to click on an object in the world and do something in response -- a seemingly-trivial task that is not at all trivial for large scenes and complex meshes, since it requires casting a ray through acceleration structures)
  • Got our first star system set up and working using CTypes and all the new high-perf Lua stuff!
  • Made a lot of improvements to our geometry code (BSPs, triangles, rays, etc.)
  • Implemented really fast BSP-Sphere intersection tests
  • Loads of other things that I'm forgetting (again)

Sean

Sadly, last Friday was Sean's final day here as an intern, since he's now headed off to college to study CS and become an unstoppable force of programming. We'll miss him for sure...his time here seemed way too short. Nonetheless, he was able to accomplish an impressive amount in the seemingly-little time that he had.

In his final act of glory, Sean finished implementing a fully-generic, hierarchical, fleet-based AI maneuvering system, capable of representing fleets with any number of sub-fleets, and intelligently directing their motion. To demonstrate it, he showed us a group of several fleets flying together in a V formation, with each fleet comprised of squadrons in a V formation, each of which was comprised of fighters in a V formation...you get the idea! All of the math required to make it happen is rather difficult due to this arbitrary nesting requiring scale-invariance. Sean got it all hammered out and now we have proper, hierarchical fleets. I'm really excited to see this in-game :D I don't have a gif on-hand, but I'll be sure to show off his work as soon as we attach it to the main LT Lua code.

---

High-Performance, High-Convenience: A Peak at Coding in LT

As you all know, Adam and I have worked relentlessly over the past months to devise ways to achieve both extreme performance capable of handling the LT simulation logic, as well as the convenience of being able to write said logic in a language that won't rob us of every last drop of life. I'd like to show just a snippet of code demonstrating what our native type system ('CType' -- the final version of what we used to call the ECS) can do, how it looks, etc. So, have a slice of toy code! (Since it's harder to read without syntax highlighting, I just screenshotted some code in-editor)

Image

This is a massively-simplified battle simulator, intended only to showcase the simplicity of gameplay code. Nonetheless, it demonstrates some important features / fruits of our labor. Note: I cranked this out really quickly and didn't actually test it, there may be typos or slight mistakes :whoops: )

Notable Take-Aways:
  • Native memory types can be defined on-the-fly in Lua; behind the scenes they are 100% as compact as the equivalent type would be in statically-compiled C; accessing the memory is extremely fast due to fancy, custom memory-pool allocation magic done automatically by the engine. All of this complexity is hidden from the user, so code remains very easy to read and write.
  • Types can be nested in other types, enabling 'component-based' design, as demonstrated with Health and Weapon. Of course, since this is a full-blown type system rather than an ECS alone, we can do things like have multiple health components, which ends up being useful sometimes!
  • 'Member functions' can be attached to these types and called in the idiomatic Lua manner; convenience + performance = winning!
  • Generic functions like 'Weapon:damage' can be written without concern for the type of the target object and still 'just work,' despite the fact that we're using direct field access instead of table lookups like most dynamic languages (if you're a programmer and understand what I mean, that should blow your mind a bit :) )
  • (Here's the even-more-mind-blowing magic) Functions that can operate on different types of objects (like Weapon:damage, which could be used to damage stations, hardpoints, etc) will be traced by LuaJIT and end up running as fast as equivalent statically-compiled constructs (in C this would require a switch statement or function pointer, in C++ this would require a template or virtual tables; here it requires nothing extra!!); this is a HUGE win for both performance and simplicity!
  • Although I didn't show it above, types can be built in a piece-meal fashion rather than defined all at once. In particular, mods have access to type definitions and can inject their own fields and methods as required to support extra, mod-related data and functionality (I showed something like this in a devlog a while back). As always, giving modders the ability to add just about anything conceivable to vanilla LT is really important to me!

---

Well, once again I've completely and utterly failed to be brief :ghost: :ghost: It's hard to be brief when there's so much happening and so much about which to be enthused. Speaking of being enthused. I must leave you all now to return to the code :)

:wave:



Hello, everyone! Talvieno here with a non-technical summary. There hasn't been much demand for one for a while, but this devlog seems to be worthy of it. Without further ado, let's get down to business! There are a couple "big things" to take away from this update - namely, Sean is no longer with us, and we're almost ready to start focusing entirely on gameplay. As to Josh's bullet lists, I'll break down what those mean in the spoilers below.

Josh
(tl;dr: made things much faster and more performant)
Spoiler:      SHOW
  • Implemented stuff for better performance
  • Made it easier to write code while the game is running (I think)
  • Implemented more stuff for better performance
  • Hooked up pieces of code we've been working on for a while so we can actually use them, but then called it CType (instead of ECS) because that sounds cooler and makes more sense
  • Some stuff that printed stuff to screen that used to be in Lua is now in the C code core, so it's faster
  • Made major improvements to code that let us draw things that are far away (or very large)
  • Worked some on the foundation for gameplay code
  • Implemented stuff for better performance (Yes, again)
  • Implemented stuff for way better performance (No, really)
  • Got sidetracked and made some really shiny images with what seems to be a homebrewed renderer (which is particularly impressive, if not entirely LT-relevant, but it was done in Josh's free time)
  • Built some programs to let us test small pieces of LT Lua code quickly
  • Lots more where that came from

Adam
(tl;dr: Lots of coding wizardry)
Spoiler:      SHOW
  • Started working on gameplay code and gameplay-related things!
  • Fully educated himself on Lua and LuaJIT, making him roughly on-par with Josh's level of accumulated LuaJIT knowledge
  • Long complicated bulletpoint that essentially means Adam is a wizard and is now quite near god status. Basically he implemented a lot of techy workaround stuff that overcomes Lua's inherent flaws. Neato, right?
  • More wizardry. Like Josh said, Adam implemented stuff that lets you click "on" an in-world object. Trickier than it sounds. It's taken care of now. It seems this may be more robust than the way Josh did it before, too.
  • Created a star system with the new LT code he and Josh have been working on. I basically copied this from what Josh said, it's pretty clear already.
  • Implemented more stuff for better performance (yes, Adam is guilty of this generic task description too)
  • Implemented some really performant collision-related tests, meaning we can check for collisions even faster. (Collisions are actually really hard for computers to detect. Our brains are much better suited to it.)
  • Loads of other stuff that Josh is forgetting.
Okay, now we're to where Josh is telling us that Sean is leaving. :( We'll miss him dearly, and hope to see him around sometime in the future. In his final act of glory, he [technical gibberish that basically means he made ships in fleets fly around in an actually smart way, instead of all flying around haphazardly like in the LT Prototype]. It'll look pretty, believe me.


As to all the last part (the image, the code, the final bulletpoint)... that can be summarized by saying Josh is proudly showing off the fruits of his labor, which unfortunately may not mean much to anyone that isn't a well-established programmer that can at least code on an intermediate level. The main things to take away from it is that it's fast, and flexible, meaning it's the perfect tool for modders. I'm actually understating it here - this is some top-tier wizardry that Josh and Adam have written up. If you're used to games running considerably more slowly when you start adding mods, that shouldn't be the case with LT. For those interested in writing their own mods, all this also means that you'll have a ridiculous amount of control over the functionality you can add to the game.

And, that's it! Josh failed epically at being brief (does he ever not? We love him for it), but I've hopefully condensed it down to something a bit more pleasant to read for the non-technically inclined. :) I'll answer any questions as necessary, and thanks for reading!
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
Post

Re: Friday, August 18, 2017

#4
Sad to read that Sean had to leave. I thank him for the work he did on making LT a better game. :D

I always love the shinies, Josh. :thumbup:

:shifty: Enjoyed the enthusiasm on display in your post...and the colours used in the code you revealed.

Will need a " non-technical version" to enhance my joy concerning your progress.

Glad to read more about Adam's work. :) He posts great mini updates which we appreciate. ;)

Keep up the good work, Josh, Adam and the rest of team LT. :clap: :angel:
Post

Re: Friday, August 18, 2017

#12
The points listed in the notable takeaways are all excellent. You've basically co-opted LuaJIT and C to create your ideal game development framework, with the added upside of being actually well-performing. I reckon that stuff will pay handsome dividends when the time comes to start work on new projects.
Post

Re: Friday, August 18, 2017

#14
1. Thank you again, Josh. Always good to see communication from the Ubermeister. (Adam, I read your comments as well -- I really appreciate your taking the time to keep in touch with those of us supporting this project.)

2. Thank you, Sean! I hoped you enjoyed your time working with Josh and Adam and LT, and I wish you the best in your studies.

3. Yay 4 technical progress! \o/ It sounds like effort is being expended to let modders write what's really just generic Lua with some LT-specific extensions, and that this will deliver the (simplicity) benefits of Lua with the (performance) benefits of C/assembly -- that's amazing, and awesome, and what in the world are you going to do with this miracle of technology you've created (other than make LT)? White papers? Masters' theses? Advice to whoever owns Lua / LuaJIT? Free "performant LuaJIT interface library" on GitHub? Other?

4. It takes nothing -- NOTHING -- away from any of the above to add that I remain focused (as a Limit Theory Kickstarter backer) on seeing evidence that the milestone of "on-the-fly procedural generation of new star systems and dynamic factional AI that spans these multiple star systems in plausible ways" has been achieved.

Eyes on the prize, gentlemen. :)
Post

Re: Friday, August 18, 2017

#15
Easy power. Nice. I'll just say that it will need to be fairly easy and flexible if I'm going to have any hope of coding in some of my ideas. :ghost: :monkey:

All this performant work makes me wonder if there will be some sort of GUI to play with these tools in addition to coding, making more of a "Game Engine" feel to things, giving LT something akin to a Procedural Unity or UE4 where you can generate infinite similar models or scenes. I think it would be really cool if you could have an engine where you build a model, defining a set of core "skeleton" features that are universal, then you could have the engine create iterations on the skeleton to make new, unique-but-similar models. Or apply the same thing to a collection of models/objects. It would be even cooler if you could go through these PCG iterations, prune the ones you dont like, and put the ones you do into a "family" which helps the engine determine the parameters of what you were looking for vs what you weren't. the more members of the "family", the better defined the family and it's traits are.

If that's not really clear, imagine PCG cathedrals
Spoiler:      SHOW
Image
Image
Image
Image
Image
Image
They all have many architectural similarities, but are all subtly or even significantly different in the execution of those similarities. If you could create an archetypal cathedral, generate iterations, select from those iterations and put them into a family of approved iterations that tell the engine "Yes, these are what I want the iterations to look like, make more things that look like them", so that future iterations deviate in specific ways, you could create a near infinite number of unique cathedrals which all maintain that important "cathedralness".

This sort of thing could be used to create and define faction or species-based aesthetics (All Borg ships, while unique are essentially giant cubes, while all Starfleet ships look essentially like a spoon with multiple handles, and all Goa'uld ships are radially symmetric snowflakes with a pyramid in the center).

If you then apply this to multiple models that make up what i'm calling a "scene" you could define the equivalent of a generic castle.
  • All castles have a perimeter ring of walls
  • All castles have a keep
  • Castles come in Irish, English, French, German, Spanish, Japanese, Indian, etc varieties, each variety has it's own traits, but they're all still castles.
In terms of development, I think this would profoundly magnify the output of creative work; Hand define and build a handful of archetypes, select a dozen or so approved random iterations on each archetype, and then you have countless unique objects that all bear a resemblance to each other aesthetically and behaviorally. But also in terms of gameplay, this could offer an extremely easy way to issue construction orders. You would have to issue the order of "Build a 'Castle' in the Hyperion VI System" You wouldn't have to go in and tell the AI "Ok, build walls here, build a keep here, build training grounds here, build x here, y here, z here" you would just tell your subordinate AI "Build a castle in the Hyperion VI system" and it would select from the "Castle" family, generate a design based on your culture and budget, and you come back a while later and boom, there's a totally unique castle that is aesthetically and functionally coherent with the rest of your empire.

Again, sorry if i'm not being terribly coherent, I'm suffering the caffeine shakes at the moment :ghost:
Image
When you're trying to fill an infinite multiverse, if you're not willing to consider the entire creative output of humanity as a starting point, you're wasting your time.
User: JoshParnell is accountable for this user's actions.

Online Now

Users browsing this forum: No registered users and 3 guests

cron