Return to “Announcements”

Post

Re: The Limit Theory Email/RSS News Thread

#31
I've decided to include a small post by AdamByrd (one of the two coders Josh hired as help) as a temporary stand-in while we wait for an update from Josh himself. :)
AdamByrd wrote:
Tue Aug 15, 2017 9:59 am
Guys! Gals! Hey. I'm just popping on to check in with a super ultra tiny mini update.

Things are starting to move fast. Lil'T is alive and kicking. And producing pixels. Actual pixels! (Seriously, I've spent so much time staring at text books and data dumps these last three months I almost forgot we're making a game. With nebulas and rocks and space pirates.) I've been working to nail down how we're going to be writing actual gameplay code. That means fleshing out how Lua talks to C to keep heavy bits and allocations in fast code land, how to make that Lua feel 'normal' and not like C grafted onto Lua (metatables are really nice!), making sure LuaJIT actually jits to assembly, iterating on the engine API to remove friction, and building a prototype solar system. I'm insanely excited about the progress we've been making. All the mountainous unknowns are now small piles of exploded rubble. We're moving faster by the day.

The GIF floodgates are opening soon, my friends.

What a time to be alive.


Link to original: viewtopic.php?f=30&p=153664#p153643
Have a question? Send me a PM! || People talking in IRC over the past two hours: Image
Image
Image
Post

Re: The Limit Theory Email/RSS News Thread

#32
Friday, August 18, 2017

Hey all!

As usual, I apologize that these logs are doing a bit of temporal stretching lately -- honestly we're just really, really busy working on LT and it's tough to pull myself away these days, so consider it a good thing! When we last spoke, several major advances had set us up for fruitful weeks to come. I'm pleased to report that they were, indeed, very fruitful.

At the moment, I can summarize our work by saying: we're dangerously close to shifting the majority of our efforts to sprinting on LT gameplay code in Lua. And we're doing everything we can to close the gap.

Seriously, the 'road to (performant) gameplay' has been way longer than anyone wanted. But the end is in sight :)

---

Josh

  • Finished implementing zero-cost Lua wrappers for engine functionality (mentioned in last devlog; means we're getting superb performance when interacting with the LT Core)
  • Implemented enough of the foundational Lua code that we can all work on LT in Lua at the same time
  • Continued careful analysis of LuaJIT debug output; continued to ensure 100% JITing to assembly; continued to learn more about the trace compiler and how to beat it into submission :ghost:
  • Hooked up our ECS to Lua + wrote the convenience wrappers that allow it to be used really easily (note: we now call it the 'CType' system instead of 'ECS', reason being that it is actually more general and flexible than an entity component system, although it can and does serve as one -- I'll explain more below!)
  • Ported Lua deferred renderer; modified to take advantage of all high-performance constructs
  • Made major improvements to our logarithmic z-buffer code while doing the above (this is the code that allows us to render objects at massive scales & distances without issue)
  • Implemented a work-in-progress event system in Lua for use in gameplay code
  • Implemented Lua wrappers for efficiently dealing with SDFs and OpenVDB (the library mentioned in the last log that we now use for extracting meshes from distance fields)
  • Implemented CSG functions on SDFs that are roughly 10x faster than OpenVDB's defaults, allowing complex, detailed meshes to be built very quickly (read: better procedural mesh algorithms coming soon!)
  • Wrote a simple pathtracer for fun since graphics Josh has been repressed :monkey: Not totally irrelevant to LT, though, as we could perhaps use it for nice renders of procedural assets or for having a 'ground truth' reference for our physically-based rendering code
  • Built several Lua 'template' applications that allow us to easily various, specific pieces of functionality really quickly (just like the old LTSL sandboxes!)
  • Loads of other things that I'm forgetting

Adam

  • Started working in the Lua side of the LT codebase!
  • Put in the time to learn about all of LuaJIT's lovely 'idiosyncrasies'; Adam is now 100% capable of helping me deal with the technical side of LJ
  • Used LuaJIT magic to implement the first parametric type in our CType system -- the arraylist! This means that not only can we define and use new, native datatypes in Lua, but we can also have what amounts to native std::vector<T> and so forth of them. I know I've beat this horse to the ground, but seriously, the power we now have to create and use types that are as fast as compiled code simply can't be overstated! Memory management is by far the weakest spot in Lua's performance, and we are almost entirely side-stepping it.
  • Hooked up BSP code to Lua and used it to implement picking (being able to click on an object in the world and do something in response -- a seemingly-trivial task that is not at all trivial for large scenes and complex meshes, since it requires casting a ray through acceleration structures)
  • Got our first star system set up and working using CTypes and all the new high-perf Lua stuff!
  • Made a lot of improvements to our geometry code (BSPs, triangles, rays, etc.)
  • Implemented really fast BSP-Sphere intersection tests
  • Loads of other things that I'm forgetting (again)

Sean

Sadly, last Friday was Sean's final day here as an intern, since he's now headed off to college to study CS and become an unstoppable force of programming. We'll miss him for sure...his time here seemed way too short. Nonetheless, he was able to accomplish an impressive amount in the seemingly-little time that he had.

In his final act of glory, Sean finished implementing a fully-generic, hierarchical, fleet-based AI maneuvering system, capable of representing fleets with any number of sub-fleets, and intelligently directing their motion. To demonstrate it, he showed us a group of several fleets flying together in a V formation, with each fleet comprised of squadrons in a V formation, each of which was comprised of fighters in a V formation...you get the idea! All of the math required to make it happen is rather difficult due to this arbitrary nesting requiring scale-invariance. Sean got it all hammered out and now we have proper, hierarchical fleets. I'm really excited to see this in-game :D I don't have a gif on-hand, but I'll be sure to show off his work as soon as we attach it to the main LT Lua code.

---

High-Performance, High-Convenience: A Peak at Coding in LT

As you all know, Adam and I have worked relentlessly over the past months to devise ways to achieve both extreme performance capable of handling the LT simulation logic, as well as the convenience of being able to write said logic in a language that won't rob us of every last drop of life. I'd like to show just a snippet of code demonstrating what our native type system ('CType' -- the final version of what we used to call the ECS) can do, how it looks, etc. So, have a slice of toy code! (Since it's harder to read without syntax highlighting, I just screenshotted some code in-editor)

Image

This is a massively-simplified battle simulator, intended only to showcase the simplicity of gameplay code. Nonetheless, it demonstrates some important features / fruits of our labor. Note: I cranked this out really quickly and didn't actually test it, there may be typos or slight mistakes :whoops: )

Notable Take-Aways:
  • Native memory types can be defined on-the-fly in Lua; behind the scenes they are 100% as compact as the equivalent type would be in statically-compiled C; accessing the memory is extremely fast due to fancy, custom memory-pool allocation magic done automatically by the engine. All of this complexity is hidden from the user, so code remains very easy to read and write.
  • Types can be nested in other types, enabling 'component-based' design, as demonstrated with Health and Weapon. Of course, since this is a full-blown type system rather than an ECS alone, we can do things like have multiple health components, which ends up being useful sometimes!
  • 'Member functions' can be attached to these types and called in the idiomatic Lua manner; convenience + performance = winning!
  • Generic functions like 'Weapon:damage' can be written without concern for the type of the target object and still 'just work,' despite the fact that we're using direct field access instead of table lookups like most dynamic languages (if you're a programmer and understand what I mean, that should blow your mind a bit :) )
  • (Here's the even-more-mind-blowing magic) Functions that can operate on different types of objects (like Weapon:damage, which could be used to damage stations, hardpoints, etc) will be traced by LuaJIT and end up running as fast as equivalent statically-compiled constructs (in C this would require a switch statement or function pointer, in C++ this would require a template or virtual tables; here it requires nothing extra!!); this is a HUGE win for both performance and simplicity!
  • Although I didn't show it above, types can be built in a piece-meal fashion rather than defined all at once. In particular, mods have access to type definitions and can inject their own fields and methods as required to support extra, mod-related data and functionality (I showed something like this in a devlog a while back). As always, giving modders the ability to add just about anything conceivable to vanilla LT is really important to me!

---

Well, once again I've completely and utterly failed to be brief :ghost: :ghost: It's hard to be brief when there's so much happening and so much about which to be enthused. Speaking of being enthused. I must leave you all now to return to the code :)

:wave:



Hello, everyone! Talvieno here with a non-technical summary. There hasn't been much demand for one for a while, but this devlog seems to be worthy of it. Without further ado, let's get down to business! There are a couple "big things" to take away from this update - namely, Sean is no longer with us, and we're almost ready to start focusing entirely on gameplay. As to Josh's bullet lists, I'll break down what those mean in the spoilers below.

Josh
(tl;dr: made things much faster and more performant)
Spoiler:      SHOW
  • Implemented stuff for better performance
  • Made it easier to write code while the game is running (I think)
  • Implemented more stuff for better performance
  • Hooked up pieces of code we've been working on for a while so we can actually use them, but then called it CType (instead of ECS) because that sounds cooler and makes more sense
  • Some stuff that printed stuff to screen that used to be in Lua is now in the C code core, so it's faster
  • Made major improvements to code that let us draw things that are far away (or very large)
  • Worked some on the foundation for gameplay code
  • Implemented stuff for better performance (Yes, again)
  • Implemented stuff for way better performance (No, really)
  • Got sidetracked and made some really shiny images with what seems to be a homebrewed renderer (which is particularly impressive, if not entirely LT-relevant, but it was done in Josh's free time)
  • Built some programs to let us test small pieces of LT Lua code quickly
  • Lots more where that came from

Adam
(tl;dr: Lots of coding wizardry)
Spoiler:      SHOW
  • Started working on gameplay code and gameplay-related things!
  • Fully educated himself on Lua and LuaJIT, making him roughly on-par with Josh's level of accumulated LuaJIT knowledge
  • Long complicated bulletpoint that essentially means Adam is a wizard and is now quite near god status. Basically he implemented a lot of techy workaround stuff that overcomes Lua's inherent flaws. Neato, right?
  • More wizardry. Like Josh said, Adam implemented stuff that lets you click "on" an in-world object. Trickier than it sounds. It's taken care of now. It seems this may be more robust than the way Josh did it before, too.
  • Created a star system with the new LT code he and Josh have been working on. I basically copied this from what Josh said, it's pretty clear already.
  • Implemented more stuff for better performance (yes, Adam is guilty of this generic task description too)
  • Implemented some really performant collision-related tests, meaning we can check for collisions even faster. (Collisions are actually really hard for computers to detect. Our brains are much better suited to it.)
  • Loads of other stuff that Josh is forgetting.
Okay, now we're to where Josh is telling us that Sean is leaving. :( We'll miss him dearly, and hope to see him around sometime in the future. In his final act of glory, he [technical gibberish that basically means he made ships in fleets fly around in an actually smart way, instead of all flying around haphazardly like in the LT Prototype]. It'll look pretty, believe me.


As to all the last part (the image, the code, the final bulletpoint)... that can be summarized by saying Josh is proudly showing off the fruits of his labor, which unfortunately may not mean much to anyone that isn't a well-established programmer that can at least code on an intermediate level. The main things to take away from it is that it's fast, and flexible, meaning it's the perfect tool for modders. I'm actually understating it here - this is some top-tier wizardry that Josh and Adam have written up. If you're used to games running considerably more slowly when you start adding mods, that shouldn't be the case with LT. For those interested in writing their own mods, all this also means that you'll have a ridiculous amount of control over the functionality you can add to the game.

And, that's it! Josh failed epically at being brief (does he ever not? We love him for it), but I've hopefully condensed it down to something a bit more pleasant to read for the non-technically inclined. :) I'll answer any questions as necessary, and thanks for reading!



Link to original: viewtopic.php?f=30&t=6271
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
Post

Re: The Limit Theory Email/RSS News Thread

#33
Thursday, August 31, 2017

Really happy to report that it's been a tremendously productive two weeks! In the last log I spoke extensively about our various mechanisms for getting bleeding-edge performance out of a magic blend of LuaJIT and C. Since then, we've been building more and more 'real stuff' using those systems, working our way to release-quality code! The general theme as of late has been "wow, it just works!" Of course, the longer version is "wow, after 1000s of hours invested in technobabble and aggressive banging-head-on-desk followed by burning the codebase and restarting, things finally, at long last, just work!" :ghost:

---

(I'm not going to separate by Josh & Adam this time since we were both working all over the place.)

Basic Game Objects & Components: Working!

Last week was the first time that we were actually able to build and use some real components and game objects purely out of our CTypes system (see last devlog if you don't know what that means). It worked beautifully. Right now we've only got System, Asteroid, Ship, and Nebula objects -- we're scaling up slowly as we nail down what we want the gameplay code to look like / how we want to be able to manipulate said objects in Lua. We've also got some basic components -- Motion and Transform, in particular. The former handles kinematics, the latter keeps track of position/orientation, etc. These are your bread-and-butter components for a component-based game engine (along with something like Drawable/Renderable, Inventory, etc.), so naturally we want to test the most ubiquitous components first.

The real stress test of our CTypes system came when we packed thousands of asteroids into a system, doing a rather naive update loop on them, and performing full rigid body dynamics on them (meaning I stuck a motion component in the asteroid definition and allowed every single asteroid to have accurate linear and angular dynamics -- something that we're probably not even going to do for release, because I still contend that dynamic asteroids don't add a whole lot of value..but hey, mod away). The speed was unreal. The bottleneck was, by a huge margin, the rendering (no LOD models yet). The actual update logic is so fast that, if I turn off rendering, I can get up to about 30,000 asteroids before I drop below 60 FPS on my machine. Again, that's a full rigid body dynamics step happening on 30,000 moving, spinning asteroids @ 60 FPS. And that is using very naive update logic, no cleverness about putting objects that aren't moving to sleep, no threading, etc. Adding in cleverness is only going to push that number even higher :shock: Anyway, I guess I can stop bragging about our performance now :oops: It's just that...I spent SO long battling FPLT -- and sometimes seriously doubting that I'd be able to give LT that 'epic' scale that I so badly wanted -- that...surely I deserve a little leeway... :monkey: :angel:

Dynamics: Fast, Accurate

I mentioned the rigid body sim above, but it's worth mentioning it as a stand-alone point. To be perfectly honest, I've never had accurate physics code in LT. I mean, the linear part was accurate. Kind of. But angular dynamics have always been extremely hacky in LT. But those days are over, and we now have a genuine physics integrator with accurate angular dynamics (i.e. applying impulses at various points on objects results in correct torque & force). Honestly it's not really a big point, as you likely won't notice many cases of fancy angular dynamics in LT. But I think the takeaway is that this stuff was written in Lua (yes, the physics step is in Lua...!) and runs well within the acceptable range for us to support a massive simulation.

I suppose it's also worth mentioning that mods could certainly leverage this to 'go further' with the physics in LT. You all know I'm a fan of realism right up until it starts cutting into fun, which, in my experience, happens quite early-on. I don't intend for ship thrusters to use real impulses, because I don't want players to have to spend hours trying to properly mass-balance their ships to get inertial characteristics that aren't impossible to fly with. But for those of you who love that good old abused phrase "real Newtonian physics" -- well, mod away. I already wrote the real physics for you ;)

Adam's BSPs are Stupidly-Fast

Several times over the past week I've had to turn around and tell Adam that his BSPs are just grossly over-performant. In our current LT sandbox, we can click objects to select them, view data about them, orbit them with the camera, etc. Selection requires raycasting the scene, which, in turn, requires raycasts against the BSP trees that provide the acceleration structures for meshes. Currently I am not using a broadphase structure, so we are literally raycasting against every single asteroid / ship in the scene. Really, it shouldn't even work. It should just melt the computer when you have thousands of complex objects in the scene. But it doesn't. In fact, the fps hit is so low on my machine that I forget how negligent I'm being.

The biggest place that raycasts are used (biggest in terms of 'most likely to cause performance problems') is in checking to see if a projectile weapon hits an object. Pulses, beams, railguns all use raycasts. Missiles are slightly more complex but still do some raycasting.

I guess where I'm going with this is: we can probably support like 10000000 million projectiles in the world at once. Or something like that. Basically, REALLY BIG BATTLES. 100 ships is going to be a cakewalk, apparently :shock:

Josh's Nebulae are Stupidly-Fast (and Pretty!)

I spent some time after-hours one night revisiting my most-prized algorithm: the nebula generator. I made it roughly 10x faster while also improving the visual fidelity AND making the generator more consistent, such that it yields less "really bad" systems. It was a fun night. We all know I'd die if I went more than half a year without touching 'nebula.glsl' :roll: But seriously, I'm happy about this, because nebula generation was one of the most expensive parts of system generation (by quite a lot). It's now much more reasonable, especially on less-powerful GPUs.

Render-Time Interpolation of Simulation State

We implemented it. This is a major technical component of games that need to have tight control over the game world simulation (like those that want to simulate universes). I've spoken about this in devlogs long, long ago. The idea is that simulation of the game world happens at a fixed rate, independent of the frame rate. This makes simulation more stable and typically more performant as well. It also brings a major challenge: your simulation state is no longer in-sync with your render loop, so now you have to 'guess' what the simulation state would look like at the current render time. To do so, you simply keep one frame's worth of historical state, and interpolate between the last and current state to obtain a smooth, linear approximation that gives the player the perception that everything is nice and continuous on-screen.

Major game component, done.

Dev UI

We've built (well, mostly Adam) a really handy developer UI to help with poking and prodding the game as we work on various parts of it. We decided that the real UI should be one of the last things we do (this was guided by the realization of how much time I've sunk into UI that later became useless when underlying game mechanics changed). Until then, though, we've got some nice programmer-artsy widgets to see what's going on.

Multiplayer: Just for Fun

Last Friday I implemented multiplayer. Just for kicks :ghost: Don't get excited (or worried), it's not a feature and we're not shipping with it. By multiplayer, I mean I implemented a very naive peer-to-peer network architecture so that Adam & Taylor (a good friend/fellow programmer at the tech park) & I could all throw asteroids around in the same star system. We would find the biggest asteroid that we could in the system, then set our cameras to orbit it, then fight to push/spin it in various directions with impulses. Not exactly MMO-of-the-year material, but it works.

So, the UDP socket API in our engine works, and I've proven that it's possible to implement (some degree of) multiplayer with it. I hope the LTMP modding team is ready! :V (Also, I was on linux and they were on windows, so it works across different platforms, as you would expect of a decent implementation.)

---

Conclusion:

Everything is coming together. The road has been thoroughly and meticulously paved, after oh-so-many setbacks, and we're FINALLY getting to spend almost all of our time in Lua at this point, just pushing forward...which is a really good sign! Game constructs are being built and carefully probed for performance along the way, ensuring that everything is done 'the right way' so that (ideally) we don't end up with any nasty performance-related surprises in the end. That which is being built is being built to last. Thus far, all of that effort we sunk into performance is REALLY paying off, because the work is moving super quickly.

I know it's probably difficult to appreciate how much progress the above points represent -- especially when screenshots don't look appreciably different than they did several months ago in the early days of LT in LuaJIT. But if you've been reading, you know the truth: under the hood, it's a whole different ballgame. This time, the muscle under the hood is enough to get us to release. I'm genuinely excited about it (and let me tell you, it's not easy to excite someone who's spent the last five years doing little more than staring at vim :lol: )

From here, we just keep at it. Every day, more game. Every day, more LT. Then one day...

...you know :)

PS ~ Some eye candy just because. Nothing you haven't seen before -- pretty nebulae, decent asteroids, bad ship algorithms...probably too much bloom :oops: But I know you guys require food every now and then :)

Limit Theory Devlog Gallery ~ August 31, 2017

Image

Image

Image

(More in the gallery)



Link to original: viewtopic.php?f=30&t=6281

((Talvieno's note: There is going to be a Kickstarter update very soon. I was initially going to wait for it and put them both in the same post, but I've opted to post this now, and then I'll be making another post with a copy of the Kickstarter post when it arrives.))
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
Post

Re: The Limit Theory Email/RSS News Thread

#34
Monday, October 2nd, 2017

Ahoy there! I know it's been a long time since Josh made a post, so I'm stepping in here for a moment with one from Adam, just to show progress is still being made. For those who don't know, Adam is one of the programmers Josh hired, and this is his first major post. We'll get back to you soon with more regular updates from Josh, but in the meantime, hopefully you can enjoy this one. :)
AdamByrd wrote:
Mon Oct 02, 2017 11:31 am
Hey guys. I'm going to try something new for a bit and see how it goes. In between Josh's bigger updates I'm going to start posting smaller updates. These mini-updates will focus on the specific problem I'm solving at the moment. The goal is to go deeper into detail than the bigger updates normally do and is often going to be centered around code architecture since I've been spending a lot of time there lately. They will also be more informal: I'm going to write them without much revision and without the flair that generally comes with Josh's writing. This is mostly just laziness that I'm selling as time savings and pragmatism. I also imagine this being a little more interactive than the usual updates, so feel free to ask specific questions or tell me what you'd like to hear more about and I'll do my best. Off topic stuff is fine as well as I realize you guys don't know a whole lot about me yet. What I won't be covering is what Josh is currently working on. For one I don't want to steal his thunder and two I want to ensure there's still new information in the regular updates. Onward!

Recap
First a bit of history. My primary goal since teaming up with Josh has been to get to gameplay as soon as possible. In a particularly productive team meeting early on this lead to the birth of Lil'T. The original goal was to have a single, playable system in a week. Something basic with player controls, a nebula, and some asteroids. This was back when I had just finished the BSP implementation. We had been working solely in the engine and didn't even have a separate (code) project for LT. I didn't know any LUA or how to start writing gameplay that leveraged the engine so this was a good motivator to start knocking down those barriers. I think it ended up being about 2 weeks before we really got to the point we wanted to be. I have to say, hooking up the BSPs for the first time and being able to click things in the scene felt really damn good.

Now, as much as I want to get to gameplay, the second major goal is making sure the code we write is shippable. It can't be prototype stuff that's going to need to be re-written. It doesn't have to 100% perfect and complete either, but it needs to be written in a way that can expanded without needing to be refactored. In getting the first phase of Lil'T running it became obvious that the infrastructure to support LUA needed to be in before going further. That took another few weeks and you've likely read about some of that in the past couple of updates. During that work we spent a lot of time profiling, testing, and tweaking things and it became pretty obvious that being able to do some of that while the game was running would save us a bunch of time so I set to task on the debug UI.

With the dev UI in a good spot we're shifting back to gameplay again. We're likely going to do this 1-2 combo of sprinting on gameplay for a bit then stepping back to improve the infrastructure supporting it a few more times. Often we can't really tell what the scaffolding needs to look like until we have concrete gameplay as a guide. We spent a couple of days talking through how we wanted to simulate entities. Since they have one-to-many parent-child relationships (a ship is the parent of it's turrets), the game objects form a tree structure. We compared hierarchical updates using depth first and breadth first traversal. We compared job-based (run on all entities) and event-based (build a list of entities that need to be updated). Ultimately we settled on event-based updates with the ability to create jobs within that framework.

Input vs Simulation
Once we nailed down how we wanted updates to be processed, input became the next target. We ended up with a lot of code that looks like this:

Code: Select all

function onUpdate (dt, hasFocus)
  if hasFocus then
    if PHX.Mouse.Pressed(PHX.MouseButton.Left) then
      self:fire()
    end
    if PHX.Mouse.Pressed(PHX.MouseButton.Right) then
      self.camera:setTarget(self:getPick())
    end
  end
end
(Note that I'm being loose with the term 'update' in the text (but not the code). The core game loop is essentially simulate, update, draw. Simulate is the fixed interval simulation step. Update happens right before draw and is for things like animation and interpolation. Draw is for, well, drawing. Also note that handling input in the update step is wrong and will be changed eventually. We aren't currently buffering input and since the simulation rate is much lower than the update rate we miss a lot of input when handling it in update at the moment.)

This isn't great. Having to manually track, pass, and handle focus in every single location that processes input is going to lead to a lot of mistakes and unnecessarily obtuse code flow. We knew this wasn't going to last long as it was being written and it needed to be addressed soon, before enough incorrect code built up that refactoring it would be a big task.

Josh and I talked through this quite a bit and his idea was to treat the game window as a fairly standard UI. The GameView would just be a UI widget that displays the final, rendered view of the simulation. The simulation becomes an autonomous thing that marches through time, updating its own state without any concept of a camera, a player, or input. UI widgets will provide visible controls and input handling that inform the simulation of requested state changes. The GameView decides how it should render the view of the simulation.

It took me a bit of time to understand the implications of this approach. It sounded a bit odd at first, but the more I worked through it, the more I was able to see the large number of subtle benefits.
  • The GameView gets the full benefit of the automatic layout functionality of the UI. Debug and gameplay UI elements can push the GameView out of the way in addition to just overlapping it. If you've noticed how the debug UI overlaps gameplay that's actually quite annoying. It's nicer if the debug window is able to make the gameplay view narrower and slide it to the right so they are side by side instead. In general it means we can allow the game view to be resized, moved, snapped, etc very easily and naturally. We can iterate on UI layout faster, allow players to customize their UI easily, and handle window resizes without any added work.
  • Multiple GameViews can exist (think radar, scanning, minimap, whole-system views, etc).
  • Input focus is already handled cleanly. The UI tracks what your mouse is over, which widget has focus, is being dragged, etc. It's trivial to extend this to have separate mouse and keyboard focus, hard and soft focus, separate onInput functions for letting only the focused widget handle input, and whatever else might be needed.
  • The amount of branching in gameplay code drops significantly. We don't have to check for focus or input because that gets hoisted to the highest level and doesn't need to be repeated all over the place. Even with our tiny system, the amount of code and complexity immediately dropped.
  • The simulation takes on more of a client-server model. Input is handled in the 'client' which then requests state changes on the 'server'. Validation in the simulation all happens in one place. We end up being forced to build a clean API in some sense.
  • Concepts like control schemes become much easier to think about and implement. Since each form of input is going to be a separate UI widget completely changing how the game controls just means creating a new widget and adding it to the UI hierarchy.
There's probably a lot more nuance that I'm leaving out. Anyway, we're currently using the dev UI to handle this. We didn't originally intend for the dev UI to be the final UI system but the performance is good enough that it's actually a possibility. It's powerful and easy to use. However, if the dev UI is swapped out in the future it's not going to affect the way the gameplay code is written so it's natural to move forward with it for now.

Of course, making this change meant a decent amount of refactoring. I had to pull all of our input handling out, decide where it went and reimplement it in the new system. I ended up with 3 new widgets: GameView, DebugWindow, and DebugPicker. I also reimplemented the data inspectors Josh added a while back so we can actually walk through the entire game state from UI windows. I also took the time to add a bit more UI polish: added stretch weights to grid columns/rows, fixed subtle positioning errors in labels, added alignment to labels.

This is mostly under-the-hood work, but I figure I can at least drop a gif of the GameView being animated smoothly: https://i.imgur.com/EkASTsN.mp4


This update is a bit higher level than I'm aiming for in general, but it's not a very involved or probably interesting topic and it's only a few days worth of work. Let me know what you guys want to hear more/less about so I know where to focus in the future posts.

Cheers
Link to original: viewtopic.php?f=30&t=6310
Post

Re: The Limit Theory Email/RSS News Thread

#35
Wednesday, October 4, 2017

Hey again everyone! 'Tis once again the season for Josh to apologize for waiting too long to post a new devlog... :ghost: Honestly, time just flies when you're having fun working on LT. And we've been doing a lot of that.

Talv, in his usual beyond-his-years Talvish wisdom, has advised me that it's more important to be consistent with devlogs than to be exciting, flashy, etc. I feel like I've heard that from you guys before. At some point, it may actually sink in to this thick head of mine :oops: At any rate, I'm going to turn over the 1-hour hourglass (that's not redundant because I also have a 30-minute 'hour'glass...) on my desk and force myself to finish this log before the last grain hits the bottom.

To accomodate the expedited writing process, I'm going to bullet-point the list of what I've been working on, then expand on a single element of focus (whichever I feel is the most important at the moment). We'll see how it goes!



Since Adam has already been kind enough to give us quite some detail on what he's been up to, I'm just going to go over my own work. So...over the past month, I've:

  • Implemented several new components for entities, most of which are in preparation for / in service to implementing gameplay logic
  • Implemented a mod-loading system that performs validation and loading of isolated mod files, which of course means that I also...
  • Defined a mod file format and started writing logic in mod form; ideally, most if not all of our actual gameplay logic will be written like this, both for our own convenience in being able to easily plug/unplug pieces of gameplay, as well as to ensure uniformity with respect to 'built-in' content vs. mod-added content
  • Implemented an event-handler architecture for effortlessly broadcasting events as well as hooking listeners to respond to said events
  • Worked with Adam to implement first pass of 'job' system, which is, essentially, the 'code' analogue of an ECS
  • Implemented several jobs / pulled much of the existing LT logic out into jobs, resulting in isolated, efficient, and easy-to-reason-about logic
  • Implemented (naive) collision detection and resolution (as a job) -- had lots of fun with large quantities of asteroids bashing into one another :D
  • Tackled many, many performance optimizations accross the board, attempting to ensure that we stay lean-and-mean as we push more content into the game; some examples are LOD meshes & automatic generation thereof, fast font glyph caching, fast vertex AO computation for SDFs, and more...
  • Explored building a simple editor program for expediting the process of adding components, entities, and logic + rolling those changes into a mod file and testing it in-game (heavily-inspired by the TES games and the legendary 'TES Construction Set' from back in my Morrowind modding days)
  • Began work on version 2 of the jobs system, which will, hopefully, unify how we write the heavy, running-almost-all-the-time gameplay logic and the sparse, happens-only-under-specific-conditions gameplay logic
  • Got the 64-bit Windows version of LT built & running



Jobs and Gameplay Logic

With our CTypes system firmly nailed-down and providing an excellent solution to how we handle data in the game, we're now looking to solve the equivalent problem with logic/functionality/code. After all, there is only data and function, nothing else :) With entities, we asked the question: how can we efficiently, easily, and extensibly define the format and manage the lifetime of the data that keeps track of an entity? How can we do so in a way that will support many diverse types of entities, while also supporting the potential to have hundreds-of-thousands of these entities in play, and perhaps even thousands or tens-of-thousands going into and out of existence in a matter of fractions of a second?

To those questions, we answered and answered with vigor in the form of an ECS, which became an ECS v2, which became CTypes.

Now we ask the analogous questions with respect to logic: how can we efficiently, easily, and extensibly define the format and manage the lifetime of the logic that runs on and creates the characteristic behavior of an entity? How can we do so in a way that will support diverse types of entity behaviors, and potentially hundreds-of-thousands of such behaviors in play at once, and potentially tens-of-thousands of these behaviors stopping, starting, or being replaced with a different behavior at any given moment?

I've touched on partial answers to this question in previous logs. Part of the answer is: exploit the sparsity of game logic to do 'as little work as possible' at every moment of time (note the symmetry with how we store 'as little data as possible' in CTypes by using native memory, pools, static typing, etc.) This means, for example, that an AI player who is travelling to a location does not enter a state where the script constantly checks whether the player has arrived at the destination. This is wasteful and fails to exploit the sparsity of the situation (the probability of arrival is exceptionally low on any given frame). Instead, the script informs the engine of the specific condition(s) upon which it should be resumed: "Dear engine, I am writing to inform you that I no longer require CPU cycles until my client, Sir NotEdisonTrent III, aka 'Trent', arrives within 100 metres of the asteroid @ memory address 0xA513701D. Alternatively, if my client should be the victim of an attack, I would like to be notified in spite of not having arrived at said asteroid. Thank you for your time, see you in a few billion cycles. Love, script/ai/TravelTo.lua." If the underlying logic engine is clever with respect to how it services such a request, then the difference here is the difference between being able to run a few hundred AI units vs. being able to run a few hundred thousand (not that we need to run a few hundred thousand -- but keep in mind that if we consider all the other logic going on in the game, then indeed, we are looking at on the order of a few hundred thousand bits of logic). It's an orders-of-magnitude subject, like many things in computing.

We already have a job system for easily running logic on large batches of entities that is very cache-friendly as well as easily-parallelizable. At the moment, I am working to fill the final piece of this puzzle: enabling said system to accommodate highly-sparse logic like the example above. The good news is that I have already designed (on paper) the architecture for the system. I'm working on the implementation as we speak (well, I mean, by the time you read this I'll be back to working on it)! Interestingly, implementing such a system is, in many ways, symmetric to designing a robust OS kernel, as many of the concepts have parallels in OS design: a piece of logic is a thread, sparse logic is a thread that waits on condition variables (a fundamental synchronization primitive used in low-level kernel / driver implementations), thread groups are woken by condition variable signals / broadcasts, a job is a process, each instance of a job is a thread, job state is thread-local storage (TLS), ...and so-on (of course it's much easier to do for a game than for an OS, don't be alarmed by my comparison!) Lucky for me, I did manage to pass the operating systems course at uni before dropping out ;) (Side-note: our professor was the guy who made one of the first scripting languages, Tcl. He was the best prof I ever had.)

An infrastructure for handling game logic is the last remaining element that I see between where we are now and a time where we can iterate wildly on shippable gameplay code. It's all very exciting :D



Success! Would you look at that? I finished just in the nick of time (ok, maybe by the time I proof-read and actually post, I will have gone a few grains over :roll:). I realize that you all probably would like more details on some of the other elements, but I'm clearly unable to give 'brief' summaries of things when I'm excited about them. And I'm excited about a lot of those bullet points. At least this way I'm able to write logs in a reasonable amount of time while still providing juicy details + a high-level picture of progress.

I really would have liked to explain a bit more about my work with the proof-of-concept editor, but alas, it would require another turn of the hourglass. For now, I just want to mention that it is being 'mostly shelved,' as the goal was to determine whether the cost-of-implementation of such a tool would be less than or greater than the time saved developing. From my exploration, it seems the cost-of-implementation would not justify the LT-dev time savings, so we will most likely not see continued work on the editor. I can imagine some pragmatists out there cheering and some dreamers out there crying :roll: :ghost: Well, come on, we have to save some things for LT 2, right? :lol: That being said, it still has some useful features: we can use it to easily get output from LuaJIT tracing & assembly dumping on Windows, where it's more difficult to do so than on Linux.

Now then. I'm going to hold myself to posting the next devlog next Friday. Period. Next Friday I will do the same thing that I did today: turn the hourglass, and write. Talvieno is right, as always: consistency is the key. And if I just wrote a devlog more often, I wouldn't have to squish so darn much into one anyway!

Oh dear. It appears next friday is the 13th. Ha. Oh well, numbers are just numbers, and, after all, 13 is both a prime AND a member of a twin prime pair, so I don't see how it could possibly be unlucky. See you all on the 13th... :ghost: :wave:



I took a few screenshots to accompany the wall of text: Limit Theory Devlog Gallery ~ October 4, 2017

Wireframe view of an asteroid field, where you can see the LOD mesh optimizations (notice the wireframe density remains approximately-constant in screen-space, despite asteroid size or distance. Constant screens-space detail is 'ideal' LOD.)

Image

I wrote a simple mod to shoot asteroids out of my ship. I used it to test the new collision detection & resolution job. It's fun blowing apart a big sphere of asteroids with a spray of little ones :nerd: (It's kind of hard to see without an animated GIF, but I'm not nearly as good as Adam at making those..)

Image

Using Plume (the concept LT editor) to get LuaJIT assembly dumps on Windows:

Image




Link to original: viewtopic.php?f=30&t=6315#p155467
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
Post

Re: The Limit Theory Email/RSS News Thread

#36
Talvieno's note: I accidentally missed a devlog, so I'm going to put that one, and the most recent one, into the same post. Please be aware they're two separate posts, spaced about a week apart. I apologize for this and will do my best to ensure it doesn't happen in the future.




Friday, October 13, 2017

In the last log I promised that today, I would turn over the hourglass and write a devlog. So that's what I'm going to do :geek: Better get going, those grains are relentless.



This week, I...

  • Implemented a spatial hash grid structure similar but far superior in performance to what LT C++ used to use for physics
  • Added xxHash to the engine; replaced several internal hash function usages with xxHash to reap the performance benefits; did some performance benchmarking to confirm how awesome it is (thanks to Adam for bringing its existence to my attention)
  • Used the aforementioned spatial hgrid to implement broadphase collision detection; despite being in Lua and still being quite non-clever (other than hgrid usage), it performs remarkably well
  • Implemented a generic C-style hash map with open addressing and lossy hashing for ultra-fast lookups (profiled @ roughly 5x the GCC 4.8 libstdc++ unordered_map impl; not that it matters); will be used in the near future for gameplay-related code
  • Unified parent/child relationships in CType components
  • Introduced and experimented with a new pattern of 'components as functions' -- since the CTypes system has 'type descriptors,' we can add fields and methods via normal functions that operate on type descriptors (a very dynamic-language-esque thing to do); doing so may simplify component logic -- we'll see!
  • Implemented powerful new mechanisms in the CTypes core to support automatic constructors, destructors, initializers (with arbitrary arguments), and automatic reference-counting
  • Used said mechanisms to simplify a lot of code
  • Used said mechanisms to help implement the Item and Blueprint classes in CTypes (I'm itching to get rolling on gameplay, so I've started working on adding some of the gameplay-related datatypes and such!)
  • Implemented two (very) different types of script manager to explore options for writing gameplay logic
  • Wrote a simple AI test script in each script format to explore usability, performance, concision
  • Took some time, while I was at it, to implement a new AI maneuvering algorithm that uses the PID (proportional-integral-derivative) controller algorithm to robustly achieve a desired position and orientation with minimal stupidity; results are really encouraging...the AI is better than ever before at piloting! (That is, in terms of the basic stuff like adjusting course and thrust. We still need to pull over Sean's work to get real formations and collision avoidance and so forth!)

Last week, I talked a lot about building a system to efficiently support gameplay logic. I could continue that subject and talk at length about how I went about doing so this week...but I want to talk about new things. Plus, I'm sure that the scripting engine is something you'll hear about frequently in the coming logs, so let's not beat a dead horse before it even dies ._.

Instead, I'll just talk about the AI logic that I wrote today using one of the two script architectures that I built this week.



AI Maneuvering Using Proportional-Integral-Derivative (PID) Control

A friend at work who is doing some real-life AI piloting coding (for a quadcopter drone, nonetheless) mentioned this 'PID' algorithm to me. I had never heard of it before, but it sounded interesting. Like the vast majority of my memory, it slipped away below the visible horizon of my consciousness until today, when, while trying to write some representative gameplay code to test out the script scheduler, I was messing with AI piloting (read: watching really dumb AI ships thrust themselves into oblivion, or, if lucky, get caught in a headache-inducing micro-orbit, doing donuts in space at 10Hz). That's when it surfaced, by chance, and I went to ask him more about AI piloting algorithms. A few hours and a bit of math later, AI ships were forming up on me in record time, with an all-time-low of overshoots and micro-oscillations. Even the small, drone-sized ships that currently have an absurd angular-thrust-to-inertia ratio (yes, these were the ones doing donuts...) and can't even be piloted by yours truly are stable and well-behaved. The AI now rocks at piloting mechanics. The magic was all in the PID controller algorithm.

PID is a really elegant way to approach 'black box' control. By that, I mean that one does not have to know all details of what is being controlled. In the case of AI, we do, of course, know what actually happens when a thruster is fired: an impulse is applied to the ship at a specific point, resulting in a net torque and net force on the ship, resulting in a change in linear and angular velocity, resulting in the ship changing position or orientation or both over time, etc. We know that there is some drag, both linear and angular (because Josh doesn't think space jousting is fun; if you do then comment out the two lines of code that add drag to the game instead of yelling at me on twitter :ghost:), we know that ships have some mass and maybe an anisotropic inertial tensor that will affect what happens when we thrust.

Yeah, we have a lot of knowledge. But to take all of this garbage and make it into an algorithm that gets a simple ole' mining ship from A to B? Surely it isn't so hard. Surely we don't need ten chalkboards-worth of archaic greek symbols just to make sure that Mr. Node the Nodely of Nodooine can point his ship at a rock and mine some Dense Nodespar without igniting three of his four engines before slamming into a rock and dying a violent space death due to some 'minor' thrust miscalculations.

I mean, what do you -- presumably a human (if not, hi Taiya, glad you finally achieved sentience \o/) -- do when you go to control something in a new game? Figure out what physics equations the game uses, then use them to solve for the button/duration-of-press required to attain a certain position/orientation/steam achievement? Of course not! You're both much less clever and yet much more clever than that at the same time! You bang your hands and face against the keyboard until that little box pops up with "Achievement Unlocked: You have walked 5cm!" (Obviously a joke, no game would congratulate you for walking a certain distance... :ghost: :ghost:) How did you pull this off?? Well, maybe you figured out that if you press this button, you go forward. Maybe you pressed it too hard or for too long and now you're going too fast or have overshot your target, so you back off a bit, and repeat this process with the rest of the controls until you're 720-noscoping noobs through walls with an AWP while moonwalking and bunny-hopping at the same time. Ah, learning. :ugeek:

PID attempts to replicate the coremost features of that control process: identify a desired change (a proportional error), vary a control output with that desire, keep track of how said variation affects the rate of change (derivative) in error, and keep track of total error accumulated (integral). Using these factors alone in a simple equation is sufficient to achieve high-quality control over many processes. If you add a second derivative term, i.e. keep track of the rate of change of the error derivative, you can achieve even finer control over (non-linear) processes. This is exactly what I used to implement the new AI maneuvering algorithm, and it works beautifully -- all without having to touch specific knowledge like thruster output or drag constants.

I'm interested to see if this algorithm can be applied to higher-level bits of the AI, like project management. I'm also interested to get Sean's AI code pulled over so that we can have obstacle avoidance and formations. The good news is that, from these tests, it looks like this gameplay logic solution is viable. That'd be fantastic, because it's turning out to be super-easy to write logic with the candidate systems I built this week. More performance testing is necessary, though, since I only just got some real logic going on today.



Yayyy for sustainable devlog writing! Let's keep at it!

See you next week :)

PS ~ I know I hardly mentioned Adam at all in this log, but he's hard at work as well. I figure, since he's now officially posting devlogs of his own volition, and I'm trying to keep my logs more frequent and less overwhelming, I'll leave his work for his own logs. I will say, however, that he has done some really good stuff this week both with working toward various pieces of gameplay (ship hardpoints, turret control, WIP mining) as well as hammering at low-level things (we have a memory usage graph now! And it's already proven its worth by showing us where to optimize things that needed it badly :ghost:)



Yet another screenshot gallery is in order for this devlog! Just some shots of physics and AI from this week.

Shooting new asteroid fields into existence with the asteroid gun and broadphase collision detection (5K+ asteroids all colliding with one another at >60FPS!)

Image

A thousand AI friendsss using PID to follow me in formation with really nice piloting. I should really cap an animated gif of the AI in action so you guys can see the piloting in motion. Will try to get one soon.

Image

Ready to dominate the universe :X

Image




Link to original: viewtopic.php?f=30&t=6326



Monday, October 23, 2017
Hey guys! Since I've got a bit more than usual to say today, I'm just going to hop right in :squirrel:



Brief Recap of the Week:

  • Added physics 'sleeping' for dynamic entities -- dynamic entities at rest now take the same CPU overhead as non-movable entities; huge performance gains in a highly-dynamic scene with isolated pockets of kinematic activity
  • Added attachable component for external modules like turrets, thrusters, etc.
  • Implemented batched attachable updates for efficiently maintaining constraints between attachables and their parent
  • Moved existing placeholder turret & thruster logic into the real framework of our batched updates
  • Major improvements to the Job system (which handles the batched, per-frame logic)
  • (Experimental...) Removed frame interpolation; switched (back) to variable-dt updates in favor of cleaner/simpler architecture
  • Major cleanup of lots of game code due to the above!
  • Ported alpha pass rendering; ported beam shader (working turrets, hurrah!)
  • Extensive continued perf-testing of game logic overhead with an emphasis on scalability in high-entity + high-logic workload situations
  • Implemented and used 'simulated lag' and 'simulated stutter' to test performance scaling for low-end systems (the simulated stutter is a little too good...it'll make you queasy if you crank it up all the way :ghost: )



Sleepy Physics Awakens New Gameplay Opportunities!

Physics sleeping is a major and notable improvement on all of the previous incarnations of physics in LT, and guarantees that we'll be able to handle way more motion than before. I'll spare all the technical details concerning implementation, but the concept is very simple: don't spend time doing physics on things that are at rest (not moving). A decent implementation of this can actually eliminate essentially all cost of movable objects that aren't moving, meaning you can have lots more movable objects and/or lots more existing types of objects be movable. I say essentially only because movability requires extra information, extra information requires extra per-object memory, extra per-object memory induces extra cache misses when performing operations on those entities, so there's still a tiny up-front cost for being movable. But it's tiny.

One of the great results of this feature is that (subject to performance impact remaining minimal)...asteroids can be dynamic objects! I remember back in the early days when quite a few people were disappointed that asteroids would be unmovable. It seemed like a small issue to me at the time, but now that I've actually got them...it's quite nice :D There are several major gameplay wins at stake, including the 'big ship meets little rock' scenario (something that has caused me and, I imagine, a good many others here quite some pain and financial loss in other games... :ghost:) Previously, a large ship moving through an asteroid field was not just dangerous; it was a nearly-suicidal proposition. A carrier-sized ship hitting a static object 1000x smaller than it is an inherently-problematic scenario. Since an unmovable object has effectively-infinite mass, it can apply an effectively-infinite impulse to colliding objects. If one calculates a proper collision impulse and the resultant damage based on momenta and relative mass (as one should), then all of the kinetic energy of a hulking ship creeping through space can end up being reflected right back on it as kinetic damage at the hands of the tiniest fleck of unmovable space trash! Tragedy ensues as the fate of a faction goes up in flames, all thanks to some careless pirate tossing an empty beer can out the cockpit :ghost:

One can, as usual, come up with various ways to work around the issue, the most immediate of which would seem to be: destroy the debris. Not bad. But it's still unclear what happens when a ship meets something, say, 10x smaller. Or 5x. Or 2x. At what threshold of relative mass do we make the collidee disappear? Isn't it going to be weird having huge rocks vanish in a cloud of dust? Can AI players troll one another by building massive-yet-cheap hollow spheres of metal and launching them in the direction of a competing faction's primary source of raw materials, soon thereafter causing the whole field to disappear in a few glorious moments of poorly-resolved collisions? It actually sounds rather cool :ghost: Still, the resolution of the same situation with dynamic objects in play is even cooler: now the massive trollololosphere would drive a cylindrical hole through the field, dispersing asteroids outwardly and infuriating that one old miner who's been cutting power to his engines to get a tad more juice to the transfer beams. Returning to the more realistic game scenario, a battleship pilot need no longer worry about voyaging through an asteroid/ice/debris/cardamine field. So long as they avoid the largest obstructions, the impulses from clearing smaller obstacles will be easily handled by the shields (but please, have some respect for the poor folks who had to paint your capship -- do pack a shield generator).

It's still unclear if this will also enable mobility of large stations (the larger the scale, the larger the physics performance impact during motion; some back-of-the-prefrontal-cortex calculations tell me it's roughly-proportional to the surface area of the object, so we're dealing with superlinear slowdowns here). Frankly I don't see a whole lot of gameplay value in it. As far as I'm concerned, if you're a capital pilot and you're zipping toward Big Jiub's Quantum Armament Emporium at warp 9, you deserve whatever fate the physics engine deals out :roll: Since the issue is purely one of scale, having weapon platforms or even small outposts be movable -- and at the mercy of behemoth-class vessels -- would seem to be doable.

I've already done some really fun stress-tests with the system by spawning 1000 AI followers (running the PID algorithm discussed last time) and zipping through large (~10K) fields (at 60+ FPS!) Though the pilots aren't yet using Sean's avoidance algorithms, it's actually really neat to watch the formation as a whole either bump into (and subsequently compensate for) large obstacles, or clear smaller ones out of the way. Having everything fully-movable and collidable at the same time turns what would otherwise be a boring scenario into an entertaining one! Compare this with LTC++, where I had to (in the later days) turn off collision on NPC ships and force immobility of asteroids to maintain a reasonable framerate :( Suffice it to say, I remain very excited at all of the gameplay doors that are opening up thanks to so many hours spent fretting over CPU cycles :monkey: :D



Lindsey Joins the Dev Team!
Promises "At Least 300% Reduction" in Ship Boxiness!

That's right, there's more! I know you all are already over-the-top with joy after reading the above, since, after all, physics was the #1 most-anticipated feature of Limit Theory :roll: :ghost: But if you'll allow me just a few more paragraphs, I've yet another log to throw on the hype fire!

The third desk here at the office has remained unsettlingly-empty since Sean's departure. We can never hope to replace Sean's zest for flow-field-AI mechanics, nor would we dare try. But I'm very pleased and excited to announce that our third desk has found a new occupant: Lindsey Reid, technical artist extraordinaire.

Lindsey comes from an interesting mix of background experiences. Like Adam, she's worked here at the tech park in the past and been part of the game development community here in Baton Rouge, which is how we met in the first place. Some of you have already met her on IRC or the forums -- we've been dating for quite a long time now :) She's had the experience of working as a game programmer at a major AAA studio (making her...yes...the only one of us three to have worked in 'the big leagues'). When she approached me with a desire to work on Limit Theory, it was her knowledge of the juncture of art and code that sold me on it.

Lindsey's foremost area of interest is in improving the procedural generation algorithms, particularly for stations, ships, and artificial entities in general. Rejoice, for the time has come to retire Josh's "Ode to Box" generators and make way for a new generation of algorithms! As someone with genuine artistic talent alongside an ability to code, Lindsey can do something that I can't: she can build space stations and spaceships on paper, generating them manually with sketchpad-and-pen while simultaneously analyzing her own creative process and translating it into code. With assets in which both functional design and aesthetic form are key, this approach is more effective than my own process of rapid guess-and-check. Iterating on equations works well for asteroids and chunks of ice (and has even exceeded expectations in the case of nebulae), but doing so, as you guys have no doubt noticed, gets much harder when one turns attention to ships, stations, and the like without being able to actually draw/reason about them on paper. As I have always asserted of procedural algorithms, though, the failing is not with proceduralism itself, but rather the one doing it. We have but one person to blame for the legendary LT boxships... :oops:

The fruits of this work are already flowing. As of last week, we have a new, joint-based mechanism for procedural algorithms to leverage. Lindsey is working on creating a library of procedural building blocks along with the architectural elements to arrange those building blocks in sensible and aesthetically-pleasing ways. I was particularly pleased when she came to the joint-based solution in her second week of work, since that was an idea that I had been contemplating for the next iteration of better ships and stations. Now I need not contemplate, as the code is happening while I focus my efforts elsewhere. Lindsey will be working over the coming weeks to scale up our procedural geometry toolkit and squeeze as much quality as possible out of proceduralism in LT, especially where it remains lacking. Unlike physics, proceduralism actually is one of the most-advertised and (I like to think?) most-anticipated features :)

I'll leave the fun / detailed bits for her to dive into in her own logs (!!), as I'm sure she can shed more light on converting the artistic process into code.

Note: For personal reasons, Lindsey wishes to go by pseudonym (i.e., Lindsey). Yes, strange, I know...*briefly opens Facebook, sees 40 unread messages & 100+ friend requests from strangers, promptly closes browser and burns laptop* :ghost:



No screenshot gallery for this log, as I don't have much that's visually new to show for the week. In terms of current work, you all now know what Lindsey is up to, but on the other side of the room, Adam is scaling our developer UI up into a full, player-facing UI, since it's currently looking as though LuaJIT may just pull through for us such that UI can remain in Lua (keep your fingers crossed...)! As for me, my focus remains on gameplay logic mechanisms and ensuring performance in the presence of real, representative gameplay logic at large scales (much of which is concerned with computationally-expensive AI that needs to happen frequently but not every frame).

With gameplay logic, UI, and procedural algorithms all getting serious brain-time simultaneously, the months ahead hold quite some promise.

See you next (/this?) week! :wave:




Link to original: viewtopic.php?f=30&t=6340
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
Post

Re: The Limit Theory Email/RSS News Thread

#37
Friday, October 27, 2017

Hi I'm Lindsey

Hello everybody!!! o/// For those of you who don't know, I'm Lindsey Reid, the newest edition to the LT team! Josh announced & explained me joining the team in his last devlog, so read about it there if you haven't already :) TL;DR: Josh and I met (& subsequently started dating) through the Baton Rouge indie dev scene, then I moved to California to work on a game for """the man""", then I moved back here to work with Josh on LT :) I have skills as both an artist and a programmer, so I'm focusing on writing procedural space ships and space stations. I'm starting with stations.

I've been here working on this for a few weeks already, so there's a lot to talk about. I'm also new to writing dev logs for the public eye, so do give me feedback on what you're interested in seeing (other than pictures, I know I know :V) and if I drawl on too long about code and algorithms. OR not long enough!

Soooo... I'm gonna follow Josh's example and first write a brief recap, and then delve into details on some of the fun stuff.

Also, the Imgur gallery is here. Some of it won't make sense without reading the post :V



Brief Recap

  • Researched & drew very, very basic concept art for stations
  • Wrote foundations of Shape library for basic building blocks of procedural geometry, including:
    • Prism (with arbitrary # of sides to the 'base face' & radius, height)
    • Irregular Prism (same as above, but with unequal side length & radius for each side of the 'base face')
    • Ellipsoid (like spheres, but with arbitrary w, l, h)
    • Torus (with arbitrary functions for the inner and outer rings & radius of each)
    • Box (arbitrary w, l, h)
    • Scaffolding (arbitrary w, l, h)
  • Began foundations of Warp library for making those Shapes interesting
  • Wrote foundations of Joint system for combining Shapes along Joints
  • Wrote Piece object for storing combined Shapes
  • Experienced awe and terror at the nebulous challenge ahead of me
  • Pulled art theory out of my butt and made cool shapes with it I guess
(note that I wrote "foundations of" in almost all of these bullet points; while these libraries are pretty solid for now, I fully expect them to expand later as I start building stations with them)



Research & Concept

It is quite the challenge to build a station generator from scratch. Most other games and generators I've seen that leverage procedural content can be broken down into handmade art assets at some level, but we're doing this completely from scratch. Handmade content means you have control over defining an aesthetic down to the details, even if the way you combine them is partially random. But we don't want to hand-make one or two or fives or 100 styles for one or two or five or 100 unique alien ships, we want to be able to generate an aesthetic for an infinite number of alien civilizations. This comes with a handful of challenges that I'll be chronicling my progress in solving with these devlogs. And these problems aren't ones I can sit down and work through in a week- they're questions I've been asking since I started this ambitious project, and I'll be continuing to solve them as I build the codebase and until the day we call this 'done'. I won't really know if I'm solving these problems until I start generating stations and seeing if they look good or not. When they don't look good, I'll re-write and tweak and concept and try again. And when they do look good... we'll have an infinite number of them.

Of course, this is still a game, which means I can't just throw any old random shapes together. Players need consistency and clarity. Most games, even those that use PCG, re-use models, textures, and color schemes for the sake of clarity of their game elements. For example, in any game with treasure chests, even if you're halfway across the game world traversing completely different civilizations, treasure chests all pretty much look the same. But in LT, we want to use PCG to stretch the limits of maintaining recognizability between game elements while encouraging visual diversity between different alien cultures. With stations, for example, it'll be pretty important to be able to look at a station and immediately identify if it's likely to destroy you, say, by the number of visible weapons it has. We can't go crazy and generate just any shape we want and call it a 'gun'- that would be impossible to read. With either clever UI or similarity in shape, texture, or color scheme of weaponry, players need to be able to immediately look at a station armed with 10 huge guns and be like, ah, maybe I shouldn't mess with this guy right now. (or 'ah', let me get out my 10 even BIGGER guns and blow him to smithereens. :V) Then again, it'll be boring (and confusing in its own way) if the high-tech, opulent, flamboyant alien station has guns that look exactly like the guns on the small, scrappy station across the system. Again, we want each alien culture to have their own unique aesthetics. It's a tough balance to maintain. And a challenge I'll enjoy tackling.

My main goal with concepting stations, therefore, much unlike traditional concept art, is not to define exactly what stations and ships will look like, but to identify patterns in how stations are built and why they look good. Over the past few weeks, I've been collecting reference art & photos, both in books and on a Pinterest board, and doodling stations myself. I use these references to identify patterns for how stations are visualized: basic shapes used, clusters and patterns those shapes are arranged in, and overall silhouettes. Those patterns form the basis for how the generation algorithms will work and what shapes, warps, and patterns they'll use to create awesome, diverse stations.

Spoiler:      SHOW
Pictured below: concept doodles for whole stations & for shapes & patterns to build those stations out of. Also, random ideas scribbled on sticky notes, and at the very bottom, art books indexed with sticky tabs for quickly finding my favorite space station concepts.
Image


Shapes, Pieces, & Joints

As an artist, when I look at the world and want to translate it into a drawing, I look for two major things: 1, the composition- the lines of action, negative space, silhouettes, and large shapes holding everything together; and 2, the individual shapes that, combined together, create the composition. I've spent the last few weeks analyzing my own art and other's drawings to figure out the major patterns that can be used to describe these nebulous ideas, and what particular patterns of composition and shapes are used to specifically create space stations. And since I won't have any pieces to generate interesting compositions without the shapes that make them up, I decided to start this whole process by creating a library of shapes.

I chose the shapes to fill the library with by analyzing space station concepts and picking pieces that could conceivably be the foundational shapes that they're all made of. All of these shapes will be warped, combined, rotated, and otherwise modified to create the functional and purely aesthetic modules that make up stations. These are mostly just an estimation of what I need right now, so I'll expand the library as needed.
  • Prism
    • Spoiler:      SHOW
      Image
    • Just imagine the big shape without all of the little rectangles around it, and that's a prism :p it allows for arbitrary side #, radius, and height. The rectangles are actually a demonstration of the Joint system working!!! (which is explained below.)
  • Irregular Prism
    • Spoiler:      SHOW
      Image
    • The code to generate this is actually quite similar to the prism, but the radius and angle for each vertex on the base face is random.
  • Ellipsoid
    • Spoiler:      SHOW
      Image
    • Ellipsoid is just a fancy word for a sphere with arbitrary width, length, and height.
  • Torus
    • Spoiler:      SHOW
      Image
    • I included tori as a primary shape because 1) they're hecking cool and 2) SO many space stations have rings. also, it was fun to code. As you can see in the full Imgur gallery, both the inner and outer ring of the torus use parametric functions to make customizable shapes.
  • Box
    • A box is a box. You know what boxes look like. also i forgot to take a screenshot
  • Scaffolding
    • Spoiler:      SHOW
      Image
    • Scaffolding is kind of an odd one out because it defines a very specific shape. It's one of the first clusters of shapes that I made, and it's really here more just to demonstration that my mesh rotation & translation was working than anything else. I do expect to use it, since scaffolding is such a common shape on many space stations, but it may or may not be one of the 'primary' shapes.

In addition, I started cooking up the library of Warps and Joints. Warps have the ability to apply all kinds of functions to every vertex on a mesh in order to rotate, stretch, skew, and wiggle it into a particular shape or pattern. I don't have fun screenshots or examples just yet because I didn't have time to experiment with this. Instead, Joints were my highest priority this past week. A 'Joint' represents a point on a mesh with a position and a direction. Connecting two meshes on joints, you rotate and translate the first mesh so that the direction on the 1st mesh's joint aligns with the direction on the 2nd mesh's joint. (In math words, you transform the 1st mesh into the basis defined by the 2nd joint.) This is imperative for enabling the station generation algorithms to connect shapes in an interesting way, so it's exciting that we made this happen so quickly. And not only did we get joint connection working, but I also wrote a function that can create joints for meshes that would be difficult to hard-code. The function places a joint at the center of every quad and tri in the mesh, with the joint pointing in the direction of the surface normal. I'm hoping this will turn out to be a useful pattern for the procedural shape-combining algorithms. If it's not, I'll find a way to tweak it. And finally, I created an object for storing combined shapes- that's kinda technical in a boring way, so I won't bother to go into detail. TL;DR: We can now define connection points for shapes and GENERATE connection points for complicated and randomized shapes :)



Fin

Whew!!! I just finished my first ever public devlog!!!!! Whoaa!!! Let me know what y'all think :) And don't forget to look at the Imgur gallery for the full visual goodness of this week!!!




Link to original: viewtopic.php?f=30&t=6343
"You’ve got to work on something dangerous. You have to work on something that makes you uncertain. Something that makes you doubt yourself... because it stimulates you to do things you haven’t done before. The whole thing is if you know where you’re going, you’ve gone, as the poet says. And that’s death."
- Stephen Sondheim
Post

Re: The Limit Theory Email/RSS News Thread

#38
Monday, November 6, 2017

Welcome back, bienvenue, aloha, etc etc. You know how this works by now :D Onward!



Recap since Last Time:

  • Moved all heavy physics code (kinematics, collision, attachments/parenting, sleeping) to C (HUGE gains)
  • Added lots of new instrumentation & utilities for profiling; we have a much better handle on CPU time now
  • Major improvements to HDR tonemapping to prevent color desaturation, especially in bright regions
  • Added time acceleration (for testing purposes...)
  • Ported dust-flecks from LTC++; dust clouds to follow shortly
  • Had a lot of fun with Lindsey's new extrusion & stellation functions in our PCG library
  • Added AI aiming / turret control to existing gameplay logic test
  • Stress-tested gameplay script scheduler with great results
  • Loads and loads and loads of fixes, optimizations, cleanups
  • Scripting scripting scripting gameplay logic gameplay logic gameplay logic and such

(I feel compelled to emphasize that seeing these changes in bullet-point format doesn't even come close to conveying the amount of progress made since the last log :) )



Gameplay Logic with Coroutines and Automatic Scheduling

While much of my time over the past few weeks has been spent spelunking in the various caverns of our existing systems in search of milliseconds, it has all been in service to the bigger-picture goal of supporting 'a lot' of gameplay going on at a large scale. Ultimately, although the journey likely won't be complete for yet another week or two, I'm happy to report that things thus far are going very well. In this log, I'll give a more complete picture of what that actually means!

Now, I know already that I'm going to indulge myself here in my technical explanations, particularly since I'm excited about the system at hand. As such, I'll offer a TL;DR for those who don't find my forrays into techno-gibberish-land to be interesting.

TL;DR: It's now very easy to write gameplay logic that can run efficiently at huge scales in the midst of lots of other logic. Our systems take care of figuring out how, when, and which logic to run each frame to achieve maximal accuracy alongside maximal performance. While the system can support a tremendous amount of logic going on at once without hurting performance, it can also automatically and gracefully degrade the accuracy of simulation logic in order to maintain FPS when 'tremendous amount' turns into 'put out the fire on my CPU!!'

Those who enjoy gibberish, proceed, those who don't, skip to next horizontal separator :)

I've given a few code examples in the past of mod format, CType usage, Lua in LT, and so forth. What I haven't given is a look into how 'real' gameplay logic will look under our work-in-progress gameplay logic system (which is, for the moment, rather unimaginatively named the 'Script Scheduler'). Time to change that! How is it that we will develop that which will go on to bring life to LT? Let's have a look!

Meet my 'simple AI test' script:

Image

Clearly I've left out a lot -- I've folded the initialization, which just sets up some state for the AI unit and is largely uninteresting. I've omitted definitions for the two functions that seem to be doing the work here: aiManeuvering and aiTurretControl. All I've left is the outer body, but that's because the interesting thing about this code is not so much the content, but the structure. If you're code-savvy, you may have already picked up on the bizarre elegance of what's going on here: the script itself -- which is, at this point in testing, the only thing giving AI units any behavior at all -- is merely a function! Not only is it just a function, it's a rather bizarre one at that. It initializes some data and then goes into a seemingly-infinite loop (the 'while' block). There's precious little to indicate that it's part of a larger system. In fact, from the standpoint of the one writing this code, it almost looks like this is the only logic in existence! The code seems to completely deny being part of an ecosystem of gameplay logic. It is not worrying about keeping track of some 'state machine;' it's not defining a bunch of callback functions to interact with the outside world; aside from local variables there's virtually no persistent state of any kind! Overally, it looks nothing like what one would expect from a gameplay script.

Remember a while back when I said something about the needs of gameplay logic being very similar to the needs of a process in an operating system? :) That is, in fact, almost exactly what's going on here! Our current implementation of the script manager functions similarly to an operating system scheduler, and gameplay logic scripts function similarly to a process. Indulge me in my desire to explain how and why this is beautiful for both the one writing code and the system running it :nerd:

I've expressed over and over since my performance failures in LTC++ that exploiting sparsity is key to being able to support a thousand AI ships running low-level manuevering logic, while simultaneously having research modules all over the universe working toward technology breakthroughs, while simultaneously letting AI players in high places run cost-benefit analyses of hundreds of potential projects, while simultaneously simulating the black-box economies of colonies everywhere, while...you get the idea! Recognizing that these calculations each have their own unique demand patterns for CPU cycles, then being able to predict and accordingly exploit those patterns...well, this is the difference between LT and solitaire :ghost: But doing so while also making it tractable to write all that logic? I had a name for the problem: FPLT, the fundamental problem of LT. At the time, someone rightly pointed out that it was not so much the fundamental problem of Limit Theory so much as it was the fundamental problem of simulating any large, complex system in real-time -- a problem encountered by many others at many points in the past.

Operating systems get closer to solving that problem than any other piece of software known to me. Some solve it better than others... :ghost: An operating system must cope with: an unknown amount of work needing to be performed by an unknown number of workers at unknown time intervals, interacting with one another in unknown ways, and having an unknown number of CPU cycles to spend each unit of time to accomplish all of this. By unknown, I mean unknown beforehand -- unknown at the time of building the operating system. Hence, a good OS must be prepared to efficiently support an arbitrarily-complex web of processes using a finite amount of computational power. It must also do so in such a way that the burden is not overwhelming on the programmer when he/she needs to write a program that performs such work. If you really think about it, this is quite an amazing feat! Say what you will, but modern OSs are indeed pretty good at this central job of theirs. How?? Chiefly, via two mechanisms: intelligent scheduling and virtualization of resources. The former attempts to maximize resource utilization such that the most work-per-unit time is achieved (over both the short- and long- term). The latter makes it appear to programs that they have exclusive control over and unlimited usage of each resource, in order to ease the burden of programming. The most important resource in question is, of course, CPU time.

Enough of that, let's get back to our AI script and understand what's really going on. Clearly, something's up with that 'sleep' function. Indeed, it is central to how our script mechanism works. Under the hood, we're using Lua coroutines, a somewhat advanced programming concept to fully understand, but one with simple enough implications: scripts can run as though they were their own 'processes,' living within a function (or a multitude thereof), while still being able to give control back to the rest of the game so as not to forever stall the universe while one silly pilot tries to calculate a successful womp-rat-bullseye. Now we see what's actually happening here: those calls to sleep are 'giving up' (in coroutine parlance, 'yielding') control of the CPU to whatever thing gave it in the first place (here, the script manager). In the world of coroutines, one thinks less of code like this:

Code: Select all

  other code other code other code
    other other other
    ...
    <--  I am here :'[  -->
    ...
  ...

And more like this:

Code: Select all

  my code my code my code
  ...
  <-- yield; everyone ELSE is here >:] -->
  ...
  mine mine mine mine mine

This so-called inversion does wonders for being able to reason about game logic! We no longer have to consider the context in which our code is running -- much like a process in an OS need not worry about what else is actually going on with memory or the CPU (multithreading aside...). We simply write the entirety of what we want our game logic to do, from start to finish, and, wherever it makes the most sense (usually inside a looping construct), we will insert these 'sleep' calls to allow the rest of the world to do what it wants.

But that's not quite the end of the story, because this code is not really leveraging our system to the fullest -- it's being intentionally greedy in order to stress-test the script manager. As you might have guessed, that number inside the sleep call is a number that indicates how long the script wants to sleep. Sleep(0) essentially means "look, I'm going to let everybody else have some time to do their thing, but WAKE ME UP AS SOON AS YOU CAN." There is no 'sparsity' being exploited here. Let's consider a different script, one for a manufacturing module's logic, which is almost too good of an example for exploiting sparsity:

Image

This one may have mistakes since I just cooked it up, but it's still very much representative of what writing LT logic will look like. Manufacturing here is dead-simple. Suppose we have a 'job' description (it will come from the blueprint of whatever's being manufactured) that includes some inputs (materials consumed), outputs (materials produced), and a time indicating how long a single run of the given job takes. All we do in our script is check that our inventory contains all of the required materials to run the job, then remove the prescribed amount of input material from inventory, call sleep with the given job time, and, upon waking, feed the outputs into inventory. We can easily put this inside a loop so as to allow instructing manufacturing units to continue manufacturing until some prerequisite number of runs have been completed (or materials run out).

Here we have a true example of something that will benefit immensely from sparsity. The vast majority of the time, manufacturing logic consists of precisely nothing. If you were to write this in 'onUpdate' style, you would have a state variable, a switch or if/elseif/else sequence checking the state, and, within one of those branches, you would simply be checking if elapsedTime >= job.batchTime (which would be false for 99.9% of the time, until the final frame where the job completes). Now, run a million manufacturing jobs accross the universe at the same time, and what will happen? The other style will bog down. Maybe it will take more than a million. But it will come to a grinding halt at some point. Our version with sleep? You will see no noticeable difference in performance until the number of jobs gets so large that it starts forcing other game memory out of RAM and into swap space. It will be long, long after the other system has come to a screeching halt. This is because, with a good scheduler, we need exactly no time at all to process sleeping jobs (this is a tiny lie, but only tiny -- asymptotically, I am not lying, because sleeping jobs require O(1) time for the whole lot of them).

I have shown two examples at the extremes -- low-level AI maneuvering isn't a great example of sparsity, while manufacturing is a perfect one. But consider that LT logic spans the whole spectrum in-between, and that, by my estimation, that spectrum is heavily-biased toward the sparse side. What we have here is indeed the game-changer that we were looking for with respect to managing logic of all granularities. Huzzah! \o/ :squirrel:

Closing Notes:
  • Logic is never scheduled before requested, but may be scheduled after. For this reason, sleep returns 'delta time' -- the actual amount of time elapsed since the script yielded. AI uses this, for example, in the PID algorithm, which requires knowing elapsed time. The manufacturing script has no need for the true dt, though it does operate under the assumption that at least job.batchTime time will have elapsed when it is awoken (our scheduler, like most, guarantees an 'at least' relationship with respect to sleeping)
  • The entirety of script execution can be bounded such that total script time is never allowed to take, for example, more than 10ms per frame. Doing so allows maintaining performance -- but our scheduling algorithm also ensures that each script will still be called with an importance that is proportional to how long it's been since the script should have awoke
  • Due to the above 'fairness' policy, dt becomes 'larger than expected' for each script at roughly the same rate,
    meaning that each script's temporal accuracy degrades at roughly the same rate, so the simulation accuracy degrades smoothly and fairly in the face of too much work
  • Since the scheduler is deeply concerned with time, it is very easy to profile the total CPU time of each script! This allows much more insightful information than just 'Ship.onUpdate' or 'AI.onUpdate,' as it was in LTC++ profilers. Now we will actually see accurate time consumption for each piece of logic, aggregated over all entities on which that logic is running!
  • Speaking of which, even in its infancy, scheduler has been profiled with tens-of-thousands of scripts running at the same time with no degradation
  • The future of gameplay logic is BRIGHT! :D



Whew. Took two-and-a-half turns of the hourglass that time...but I regret nothing! It has been such a long road to this system. As I mentioned above, it'll still be a week or two before the whole thing is finalized. But things are shaping up really nicely, both from the performance standpoint and from the ease-of-use standpoint.

It's a shame that I don't have more time to talk about the tremendously-exciting developments on Friday, what with Lindsey's implementation of extrusion and stellation for station-building, plus the addition of the icosahedron primitive. I did, at least, take an excessive number of screenshots while playing with the additions. So...gallery! Note that any lack of diversity here is not the fault of Lindsey, as our primitives library is now quite large, but is to blame on my obsession with icosahedra and their beautiful symmetry...which made it hard for me to focus on any other shape :squirrel: There are also a few AI unit shots and nebula shots in there, just because :)

I must now turn attention to the KS update! See you next time ;)



Limit Theory Devlog Gallery ~ November 6, 2017

A few of my favorites from the gallery, showcasing some very basic results using almost nothing but icosahedra, stellation, extrusion, and warping (the occassional torus does show up). Thanks, Lindsey!

Image

Image

Image

Image




Link to original: viewtopic.php?f=30&t=6356
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford

Online Now

Users browsing this forum: No registered users and 6 guests

cron