Page 3 of 3

Re: The Limit Theory Email/RSS News Thread

Posted: Wed Aug 16, 2017 9:59 am
by Talvieno
I've decided to include a small post by AdamByrd (one of the two coders Josh hired as help) as a temporary stand-in while we wait for an update from Josh himself. :)
AdamByrd wrote:
Tue Aug 15, 2017 9:59 am
Guys! Gals! Hey. I'm just popping on to check in with a super ultra tiny mini update.

Things are starting to move fast. Lil'T is alive and kicking. And producing pixels. Actual pixels! (Seriously, I've spent so much time staring at text books and data dumps these last three months I almost forgot we're making a game. With nebulas and rocks and space pirates.) I've been working to nail down how we're going to be writing actual gameplay code. That means fleshing out how Lua talks to C to keep heavy bits and allocations in fast code land, how to make that Lua feel 'normal' and not like C grafted onto Lua (metatables are really nice!), making sure LuaJIT actually jits to assembly, iterating on the engine API to remove friction, and building a prototype solar system. I'm insanely excited about the progress we've been making. All the mountainous unknowns are now small piles of exploded rubble. We're moving faster by the day.

The GIF floodgates are opening soon, my friends.

What a time to be alive.


Link to original: viewtopic.php?f=30&p=153664#p153643

Re: The Limit Theory Email/RSS News Thread

Posted: Sun Aug 20, 2017 9:57 am
by JoshParnell
Friday, August 18, 2017

Hey all!

As usual, I apologize that these logs are doing a bit of temporal stretching lately -- honestly we're just really, really busy working on LT and it's tough to pull myself away these days, so consider it a good thing! When we last spoke, several major advances had set us up for fruitful weeks to come. I'm pleased to report that they were, indeed, very fruitful.

At the moment, I can summarize our work by saying: we're dangerously close to shifting the majority of our efforts to sprinting on LT gameplay code in Lua. And we're doing everything we can to close the gap.

Seriously, the 'road to (performant) gameplay' has been way longer than anyone wanted. But the end is in sight :)

---

Josh

  • Finished implementing zero-cost Lua wrappers for engine functionality (mentioned in last devlog; means we're getting superb performance when interacting with the LT Core)
  • Implemented enough of the foundational Lua code that we can all work on LT in Lua at the same time
  • Continued careful analysis of LuaJIT debug output; continued to ensure 100% JITing to assembly; continued to learn more about the trace compiler and how to beat it into submission :ghost:
  • Hooked up our ECS to Lua + wrote the convenience wrappers that allow it to be used really easily (note: we now call it the 'CType' system instead of 'ECS', reason being that it is actually more general and flexible than an entity component system, although it can and does serve as one -- I'll explain more below!)
  • Ported Lua deferred renderer; modified to take advantage of all high-performance constructs
  • Made major improvements to our logarithmic z-buffer code while doing the above (this is the code that allows us to render objects at massive scales & distances without issue)
  • Implemented a work-in-progress event system in Lua for use in gameplay code
  • Implemented Lua wrappers for efficiently dealing with SDFs and OpenVDB (the library mentioned in the last log that we now use for extracting meshes from distance fields)
  • Implemented CSG functions on SDFs that are roughly 10x faster than OpenVDB's defaults, allowing complex, detailed meshes to be built very quickly (read: better procedural mesh algorithms coming soon!)
  • Wrote a simple pathtracer for fun since graphics Josh has been repressed :monkey: Not totally irrelevant to LT, though, as we could perhaps use it for nice renders of procedural assets or for having a 'ground truth' reference for our physically-based rendering code
  • Built several Lua 'template' applications that allow us to easily various, specific pieces of functionality really quickly (just like the old LTSL sandboxes!)
  • Loads of other things that I'm forgetting

Adam

  • Started working in the Lua side of the LT codebase!
  • Put in the time to learn about all of LuaJIT's lovely 'idiosyncrasies'; Adam is now 100% capable of helping me deal with the technical side of LJ
  • Used LuaJIT magic to implement the first parametric type in our CType system -- the arraylist! This means that not only can we define and use new, native datatypes in Lua, but we can also have what amounts to native std::vector<T> and so forth of them. I know I've beat this horse to the ground, but seriously, the power we now have to create and use types that are as fast as compiled code simply can't be overstated! Memory management is by far the weakest spot in Lua's performance, and we are almost entirely side-stepping it.
  • Hooked up BSP code to Lua and used it to implement picking (being able to click on an object in the world and do something in response -- a seemingly-trivial task that is not at all trivial for large scenes and complex meshes, since it requires casting a ray through acceleration structures)
  • Got our first star system set up and working using CTypes and all the new high-perf Lua stuff!
  • Made a lot of improvements to our geometry code (BSPs, triangles, rays, etc.)
  • Implemented really fast BSP-Sphere intersection tests
  • Loads of other things that I'm forgetting (again)

Sean

Sadly, last Friday was Sean's final day here as an intern, since he's now headed off to college to study CS and become an unstoppable force of programming. We'll miss him for sure...his time here seemed way too short. Nonetheless, he was able to accomplish an impressive amount in the seemingly-little time that he had.

In his final act of glory, Sean finished implementing a fully-generic, hierarchical, fleet-based AI maneuvering system, capable of representing fleets with any number of sub-fleets, and intelligently directing their motion. To demonstrate it, he showed us a group of several fleets flying together in a V formation, with each fleet comprised of squadrons in a V formation, each of which was comprised of fighters in a V formation...you get the idea! All of the math required to make it happen is rather difficult due to this arbitrary nesting requiring scale-invariance. Sean got it all hammered out and now we have proper, hierarchical fleets. I'm really excited to see this in-game :D I don't have a gif on-hand, but I'll be sure to show off his work as soon as we attach it to the main LT Lua code.

---

High-Performance, High-Convenience: A Peak at Coding in LT

As you all know, Adam and I have worked relentlessly over the past months to devise ways to achieve both extreme performance capable of handling the LT simulation logic, as well as the convenience of being able to write said logic in a language that won't rob us of every last drop of life. I'd like to show just a snippet of code demonstrating what our native type system ('CType' -- the final version of what we used to call the ECS) can do, how it looks, etc. So, have a slice of toy code! (Since it's harder to read without syntax highlighting, I just screenshotted some code in-editor)

Image

This is a massively-simplified battle simulator, intended only to showcase the simplicity of gameplay code. Nonetheless, it demonstrates some important features / fruits of our labor. Note: I cranked this out really quickly and didn't actually test it, there may be typos or slight mistakes :whoops: )

Notable Take-Aways:
  • Native memory types can be defined on-the-fly in Lua; behind the scenes they are 100% as compact as the equivalent type would be in statically-compiled C; accessing the memory is extremely fast due to fancy, custom memory-pool allocation magic done automatically by the engine. All of this complexity is hidden from the user, so code remains very easy to read and write.
  • Types can be nested in other types, enabling 'component-based' design, as demonstrated with Health and Weapon. Of course, since this is a full-blown type system rather than an ECS alone, we can do things like have multiple health components, which ends up being useful sometimes!
  • 'Member functions' can be attached to these types and called in the idiomatic Lua manner; convenience + performance = winning!
  • Generic functions like 'Weapon:damage' can be written without concern for the type of the target object and still 'just work,' despite the fact that we're using direct field access instead of table lookups like most dynamic languages (if you're a programmer and understand what I mean, that should blow your mind a bit :) )
  • (Here's the even-more-mind-blowing magic) Functions that can operate on different types of objects (like Weapon:damage, which could be used to damage stations, hardpoints, etc) will be traced by LuaJIT and end up running as fast as equivalent statically-compiled constructs (in C this would require a switch statement or function pointer, in C++ this would require a template or virtual tables; here it requires nothing extra!!); this is a HUGE win for both performance and simplicity!
  • Although I didn't show it above, types can be built in a piece-meal fashion rather than defined all at once. In particular, mods have access to type definitions and can inject their own fields and methods as required to support extra, mod-related data and functionality (I showed something like this in a devlog a while back). As always, giving modders the ability to add just about anything conceivable to vanilla LT is really important to me!

---

Well, once again I've completely and utterly failed to be brief :ghost: :ghost: It's hard to be brief when there's so much happening and so much about which to be enthused. Speaking of being enthused. I must leave you all now to return to the code :)

:wave:



Hello, everyone! Talvieno here with a non-technical summary. There hasn't been much demand for one for a while, but this devlog seems to be worthy of it. Without further ado, let's get down to business! There are a couple "big things" to take away from this update - namely, Sean is no longer with us, and we're almost ready to start focusing entirely on gameplay. As to Josh's bullet lists, I'll break down what those mean in the spoilers below.

Josh
(tl;dr: made things much faster and more performant)
Spoiler:      SHOW
  • Implemented stuff for better performance
  • Made it easier to write code while the game is running (I think)
  • Implemented more stuff for better performance
  • Hooked up pieces of code we've been working on for a while so we can actually use them, but then called it CType (instead of ECS) because that sounds cooler and makes more sense
  • Some stuff that printed stuff to screen that used to be in Lua is now in the C code core, so it's faster
  • Made major improvements to code that let us draw things that are far away (or very large)
  • Worked some on the foundation for gameplay code
  • Implemented stuff for better performance (Yes, again)
  • Implemented stuff for way better performance (No, really)
  • Got sidetracked and made some really shiny images with what seems to be a homebrewed renderer (which is particularly impressive, if not entirely LT-relevant, but it was done in Josh's free time)
  • Built some programs to let us test small pieces of LT Lua code quickly
  • Lots more where that came from

Adam
(tl;dr: Lots of coding wizardry)
Spoiler:      SHOW
  • Started working on gameplay code and gameplay-related things!
  • Fully educated himself on Lua and LuaJIT, making him roughly on-par with Josh's level of accumulated LuaJIT knowledge
  • Long complicated bulletpoint that essentially means Adam is a wizard and is now quite near god status. Basically he implemented a lot of techy workaround stuff that overcomes Lua's inherent flaws. Neato, right?
  • More wizardry. Like Josh said, Adam implemented stuff that lets you click "on" an in-world object. Trickier than it sounds. It's taken care of now. It seems this may be more robust than the way Josh did it before, too.
  • Created a star system with the new LT code he and Josh have been working on. I basically copied this from what Josh said, it's pretty clear already.
  • Implemented more stuff for better performance (yes, Adam is guilty of this generic task description too)
  • Implemented some really performant collision-related tests, meaning we can check for collisions even faster. (Collisions are actually really hard for computers to detect. Our brains are much better suited to it.)
  • Loads of other stuff that Josh is forgetting.
Okay, now we're to where Josh is telling us that Sean is leaving. :( We'll miss him dearly, and hope to see him around sometime in the future. In his final act of glory, he [technical gibberish that basically means he made ships in fleets fly around in an actually smart way, instead of all flying around haphazardly like in the LT Prototype]. It'll look pretty, believe me.


As to all the last part (the image, the code, the final bulletpoint)... that can be summarized by saying Josh is proudly showing off the fruits of his labor, which unfortunately may not mean much to anyone that isn't a well-established programmer that can at least code on an intermediate level. The main things to take away from it is that it's fast, and flexible, meaning it's the perfect tool for modders. I'm actually understating it here - this is some top-tier wizardry that Josh and Adam have written up. If you're used to games running considerably more slowly when you start adding mods, that shouldn't be the case with LT. For those interested in writing their own mods, all this also means that you'll have a ridiculous amount of control over the functionality you can add to the game.

And, that's it! Josh failed epically at being brief (does he ever not? We love him for it), but I've hopefully condensed it down to something a bit more pleasant to read for the non-technically inclined. :) I'll answer any questions as necessary, and thanks for reading!



Link to original: viewtopic.php?f=30&t=6271

Re: The Limit Theory Email/RSS News Thread

Posted: Sun Sep 03, 2017 7:55 am
by JoshParnell
Thursday, August 31, 2017

Really happy to report that it's been a tremendously productive two weeks! In the last log I spoke extensively about our various mechanisms for getting bleeding-edge performance out of a magic blend of LuaJIT and C. Since then, we've been building more and more 'real stuff' using those systems, working our way to release-quality code! The general theme as of late has been "wow, it just works!" Of course, the longer version is "wow, after 1000s of hours invested in technobabble and aggressive banging-head-on-desk followed by burning the codebase and restarting, things finally, at long last, just work!" :ghost:

---

(I'm not going to separate by Josh & Adam this time since we were both working all over the place.)

Basic Game Objects & Components: Working!

Last week was the first time that we were actually able to build and use some real components and game objects purely out of our CTypes system (see last devlog if you don't know what that means). It worked beautifully. Right now we've only got System, Asteroid, Ship, and Nebula objects -- we're scaling up slowly as we nail down what we want the gameplay code to look like / how we want to be able to manipulate said objects in Lua. We've also got some basic components -- Motion and Transform, in particular. The former handles kinematics, the latter keeps track of position/orientation, etc. These are your bread-and-butter components for a component-based game engine (along with something like Drawable/Renderable, Inventory, etc.), so naturally we want to test the most ubiquitous components first.

The real stress test of our CTypes system came when we packed thousands of asteroids into a system, doing a rather naive update loop on them, and performing full rigid body dynamics on them (meaning I stuck a motion component in the asteroid definition and allowed every single asteroid to have accurate linear and angular dynamics -- something that we're probably not even going to do for release, because I still contend that dynamic asteroids don't add a whole lot of value..but hey, mod away). The speed was unreal. The bottleneck was, by a huge margin, the rendering (no LOD models yet). The actual update logic is so fast that, if I turn off rendering, I can get up to about 30,000 asteroids before I drop below 60 FPS on my machine. Again, that's a full rigid body dynamics step happening on 30,000 moving, spinning asteroids @ 60 FPS. And that is using very naive update logic, no cleverness about putting objects that aren't moving to sleep, no threading, etc. Adding in cleverness is only going to push that number even higher :shock: Anyway, I guess I can stop bragging about our performance now :oops: It's just that...I spent SO long battling FPLT -- and sometimes seriously doubting that I'd be able to give LT that 'epic' scale that I so badly wanted -- that...surely I deserve a little leeway... :monkey: :angel:

Dynamics: Fast, Accurate

I mentioned the rigid body sim above, but it's worth mentioning it as a stand-alone point. To be perfectly honest, I've never had accurate physics code in LT. I mean, the linear part was accurate. Kind of. But angular dynamics have always been extremely hacky in LT. But those days are over, and we now have a genuine physics integrator with accurate angular dynamics (i.e. applying impulses at various points on objects results in correct torque & force). Honestly it's not really a big point, as you likely won't notice many cases of fancy angular dynamics in LT. But I think the takeaway is that this stuff was written in Lua (yes, the physics step is in Lua...!) and runs well within the acceptable range for us to support a massive simulation.

I suppose it's also worth mentioning that mods could certainly leverage this to 'go further' with the physics in LT. You all know I'm a fan of realism right up until it starts cutting into fun, which, in my experience, happens quite early-on. I don't intend for ship thrusters to use real impulses, because I don't want players to have to spend hours trying to properly mass-balance their ships to get inertial characteristics that aren't impossible to fly with. But for those of you who love that good old abused phrase "real Newtonian physics" -- well, mod away. I already wrote the real physics for you ;)

Adam's BSPs are Stupidly-Fast

Several times over the past week I've had to turn around and tell Adam that his BSPs are just grossly over-performant. In our current LT sandbox, we can click objects to select them, view data about them, orbit them with the camera, etc. Selection requires raycasting the scene, which, in turn, requires raycasts against the BSP trees that provide the acceleration structures for meshes. Currently I am not using a broadphase structure, so we are literally raycasting against every single asteroid / ship in the scene. Really, it shouldn't even work. It should just melt the computer when you have thousands of complex objects in the scene. But it doesn't. In fact, the fps hit is so low on my machine that I forget how negligent I'm being.

The biggest place that raycasts are used (biggest in terms of 'most likely to cause performance problems') is in checking to see if a projectile weapon hits an object. Pulses, beams, railguns all use raycasts. Missiles are slightly more complex but still do some raycasting.

I guess where I'm going with this is: we can probably support like 10000000 million projectiles in the world at once. Or something like that. Basically, REALLY BIG BATTLES. 100 ships is going to be a cakewalk, apparently :shock:

Josh's Nebulae are Stupidly-Fast (and Pretty!)

I spent some time after-hours one night revisiting my most-prized algorithm: the nebula generator. I made it roughly 10x faster while also improving the visual fidelity AND making the generator more consistent, such that it yields less "really bad" systems. It was a fun night. We all know I'd die if I went more than half a year without touching 'nebula.glsl' :roll: But seriously, I'm happy about this, because nebula generation was one of the most expensive parts of system generation (by quite a lot). It's now much more reasonable, especially on less-powerful GPUs.

Render-Time Interpolation of Simulation State

We implemented it. This is a major technical component of games that need to have tight control over the game world simulation (like those that want to simulate universes). I've spoken about this in devlogs long, long ago. The idea is that simulation of the game world happens at a fixed rate, independent of the frame rate. This makes simulation more stable and typically more performant as well. It also brings a major challenge: your simulation state is no longer in-sync with your render loop, so now you have to 'guess' what the simulation state would look like at the current render time. To do so, you simply keep one frame's worth of historical state, and interpolate between the last and current state to obtain a smooth, linear approximation that gives the player the perception that everything is nice and continuous on-screen.

Major game component, done.

Dev UI

We've built (well, mostly Adam) a really handy developer UI to help with poking and prodding the game as we work on various parts of it. We decided that the real UI should be one of the last things we do (this was guided by the realization of how much time I've sunk into UI that later became useless when underlying game mechanics changed). Until then, though, we've got some nice programmer-artsy widgets to see what's going on.

Multiplayer: Just for Fun

Last Friday I implemented multiplayer. Just for kicks :ghost: Don't get excited (or worried), it's not a feature and we're not shipping with it. By multiplayer, I mean I implemented a very naive peer-to-peer network architecture so that Adam & Taylor (a good friend/fellow programmer at the tech park) & I could all throw asteroids around in the same star system. We would find the biggest asteroid that we could in the system, then set our cameras to orbit it, then fight to push/spin it in various directions with impulses. Not exactly MMO-of-the-year material, but it works.

So, the UDP socket API in our engine works, and I've proven that it's possible to implement (some degree of) multiplayer with it. I hope the LTMP modding team is ready! :V (Also, I was on linux and they were on windows, so it works across different platforms, as you would expect of a decent implementation.)

---

Conclusion:

Everything is coming together. The road has been thoroughly and meticulously paved, after oh-so-many setbacks, and we're FINALLY getting to spend almost all of our time in Lua at this point, just pushing forward...which is a really good sign! Game constructs are being built and carefully probed for performance along the way, ensuring that everything is done 'the right way' so that (ideally) we don't end up with any nasty performance-related surprises in the end. That which is being built is being built to last. Thus far, all of that effort we sunk into performance is REALLY paying off, because the work is moving super quickly.

I know it's probably difficult to appreciate how much progress the above points represent -- especially when screenshots don't look appreciably different than they did several months ago in the early days of LT in LuaJIT. But if you've been reading, you know the truth: under the hood, it's a whole different ballgame. This time, the muscle under the hood is enough to get us to release. I'm genuinely excited about it (and let me tell you, it's not easy to excite someone who's spent the last five years doing little more than staring at vim :lol: )

From here, we just keep at it. Every day, more game. Every day, more LT. Then one day...

...you know :)

PS ~ Some eye candy just because. Nothing you haven't seen before -- pretty nebulae, decent asteroids, bad ship algorithms...probably too much bloom :oops: But I know you guys require food every now and then :)

Limit Theory Devlog Gallery ~ August 31, 2017

Image

Image

Image

(More in the gallery)



Link to original: viewtopic.php?f=30&t=6281

((Talvieno's note: There is going to be a Kickstarter update very soon. I was initially going to wait for it and put them both in the same post, but I've opted to post this now, and then I'll be making another post with a copy of the Kickstarter post when it arrives.))