The Limit Theory Email/RSS News Thread

Discuss technical features of Limit Theory and the LT Engine

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Wed Mar 01, 2017 9:45 am

Note from Talvieno:
We had two major updates in rapid succession, so I'll be posting them both here.




Tuesday, February 28, 2017

A rather solid but unremarkable week; I feel this week could aptly be described as a lingering shadow of last week. For both better and worse, last week's "I started on" could basically be replaced with "I finished" or "I almost finished," and the log would be nearly complete. Thus, as has been falsely stated many times now, we may actually see a reasonably-sized devlog tonight!? (EDIT: at this point, it's a tradition...:ghost: )

---

PlateMeshes: 90+% complete in the LT core. As discussed last week, I emphasized building a release-quality implementation since they became an invaluable tool in LTC++. I now have a high-quality, fast, C library implementation of all of the old PM functionality with the exception of rotation of individual plates. I will need to spend a bit more time in order to finish that bit efficiently, but the rest of the work was by far the bulk. Frankly it required more time than I would have liked, but all of that time was spent adding supporting functionality to the core in order to efficiently support a good PM implementation. The good thing about this kind of time is that it's "permanent progress," as is all progress of the LT core (which is, as a reminder, designed to be the 'etched in stone' part of the LT engine).

Mesh 'Warps': 20% complete (I figured out how to do them). These are the things that allowed me to distort models in cool ways that added curvature and interesting global features to ships and stations. Unfortunately, their implementation in C++ using all that fancy (and incredibly-nasty) template stuff I used to do made it fairly easy. I had some trouble figuring out how to do the same thing effectively in a clean, C API (i.e., making it easy to build warping functions from LuaJIT). I've designed the implementation, about which I'm pleased. I haven't coded it yet. Interestingly, solving this design problem brought me insights on how to make reference-counting work cleanly across API boundaries (for example, to enable both the core and LuaJIT to hold pointers to the same object and still have all memory management work correctly). I've not had this insight before, but when the time invariably comes that I need to do this (beyond warps, which will require it from the get-go), I now know how to do it painlessly.

Networking & 'Control Panel': 100%/90% complete (server/client), and completely new to any of the LT engines! Yes, this is where about 2.5 days of my time went this week, but it's a technology investment that I'm betting will pay dividends. While working with planets, it occurred to me that I would really like a UI for changing the procedural algorithm parameters in real-time to see exactly how they're affecting the final result (with complex procedural algorithms it can be very difficult to reason about the effect of individual parameters). I didn't want to spend a huge amount of time building something just for this task, though, so, inspired by a friend at LTP who does web development, I decided to use a tool that already exists and is very fast and easy for creating quick UIs: the modern web browser. This necessitated networking functions in the LT core, which I had promised I would provide for modders who want to attempt 'multiplayer LT' mods anyway. So, I implemented a multi-platform TCP & UDP socket API, then used it from LuaJIT to spin up a very simple HTTP server run by in the background by the engine that serves up a simple control panel webpage when connected to via a browser. It can change internal engine variables automatically in response to controls like sliders or buttons being changed in the panel.

Though I'm hoping the ability to connect to and communicate with the engine via networking will pay substantial dividends over the remaining development time, having the control panel available is already really helpful: I have a really easy way to expose real-time-tweakable parameters, and, as a bonus, since it's based on sockets & HTTP, I can even connect remotely so that I can use one machine (like my tablet) exclusively as my 'controller' while I develop on the other. I'm also thinking this will be great for monitoring and managing long-running universe simulations (when I get to that point, ideally sooner rather than later)! I'd even like to explore using this for asynchronous resource generation on the GPU, which is traditionally difficult to do in a single process (due to threaded GL contexts being messy). There are a whole lot of ways I can envision using even this very basic networking tech :geek: Also, of course, the tools are now there (in theory) for anyone who wants to give an LT Multiplayer mod a shot at release (in practice, good luck, I recommend putting together a team :ghost:).

---

In the LTASM world, I did exactly what I had mentioned and hoped for last week: basic x87 floating point operations. Quite happy to have that stuff working. I've got enough to, for example, do most if not all of the basic FP math that you might do in the LT logic, although I don't necessarily have all of the opcodes for doing things in the most efficient way. Frankly, though, my first goal in this world of Plan Z is to have a script -> machine code pipeline working at all. That will be amazing enough for me, then we can worry about micro-optimization. My tests this week with FP instructions in simple loops are already blazingly-fast. I am getting a stronger and stronger feeling that this work will play a role in LT even if LuaJIT continues to hold. In fact, I can see it being a beautiful addition to LJ! LJ is basically a dynamic asm generator, why not have a static asm generator for perf-critical code? Man does it make a difference getting down to that raw level (I mean compared to something like Lua, not C/++).

Speaking of which, as usual, I spent some quality time with the Intel manuals this week, covering chapters 10 & 11 (SSE, SSE2), and skimming 12, 14, & 15, (SSE3/4, AVX/AVX2, AVX512). So basically, I learned a whole lot about vector / 'SIMD' instruction sets this week. Fun as always :geek: (Can you believe AVX512 has 2KB of register space for FP math?? Insane...)

---

It's worth mentioning that I spent a good bit of time this week on other, non-coding work. This week saw an abundance of emails, real-life events, and a bit more time spent with the forum than usual. Speaking of which, I assume that most of you have seen the announcement of Talvieno as Community Manager by now, but if not, check it out :) Although it took a bit of time this week to write that post and get everything squared away, his new role as CM will ensure that the amount of non-coding time spent during future weeks is kept to a minimum! I'm really happy to have Talvieno (officially) on board.

---

No, I'm not terribly impressed with this week (maybe I'm just tired right now...looking at the log, it seems fairly substantial?), next week should see less clerical work as well as less of having to finish up things, and more of getting to start new things. Seriously, next week I hope to see ships and stations in their full glory: PlateMeshes at 100% (rotations are definitely needed for stations), mesh warps implemented, procedural algorithms 100% ported, mesh ambient occlusion calculations 100% complete. I'd like to see asteroids as well, meaning scalar field extraction in the C core. If I have all of those things in a week, I'll be happy! I'll also have a good-looking, content-filled system, presumably. On the asm side, perhaps basic SSE functionality; let's say scalar operations at first, so that I can do the equivalent operations that the x87 FPU would do, but using the more-modern SSE (interestingly, the SSE instruction set is not just for vector processing, it has scalar (single-element) instructions as well).

Until some time in the next 7±2 days...! :wave:



Link to original: viewtopic.php?f=30&t=5825




Greetings LT forumers!

Those of you who have spent any substantial amount of time here have no doubt encountered Talvieno at some point or another. It's hard not to notice his outstanding and inexhaustible dedication to making the Limit Theory Forums a better place. In fact, it's no secret that a lot of you even wish for him to have more power ('Talv for president!') Well, you've already read the title, so you can guess what's coming ;) Before we get to that part, however, I'd like to take a moment to recognize the myriad ways in which Talvieno has made the (LT)world a better place -- some of you may not know about all of them; indeed, even I had to have an IRC chat with him just to make sure I had everything down right...it's awfully hard to keep up with all the things this guy does! Things like...


    ...building Taiya, our favorite resident Limit Theory forum & IRC RSS feed bot. For those who don't know Taiya, she's not only a sweetheart of an IRC chat bot with a boatload of neat functions, she's also the toughest forum policewoman in existence. We get hit by, on average, 2-3 new bot registrations a day (which is down a bit from previous months, where it peaked around 15/day). A lot of these bots post spam topics. Taiya shuts them down almost instantly, banning those poor bots of lesser intellect than her and moving the spam to a hidden 'trash' sub-forum where mods and I can look at it later. There's no telling how many bans Taiya has issued, nor how many spam threads she's killed. She's made my life so much easier! Thanks Taiya!

    ...running the Limit Theory Fan Competition, an awesome idea started way back in the day by the legendary Katorone and re-ignited by none other than Talvieno. The current LTFC ends this Saturday, so check it out if you want to participate (there are prizes!)

    ...conceiving of and running REKT, an incredibly-impressive forum-based RPG whose main thread ran for 181 pages and over 2700 posts. If you've ever wondered just how intense, creative, and amazing roleplaying on a forum can get, you need only take a peek at the REKT wiki, which is both a triumph of creativity and an inspiration to me. Talvieno informed me that most of the wiki is actually attributed to the hard work of Dinosawer, so hats off to him!

    ...starting a novel based on the Limit Theory Universe, entitled Small Choices. The existing chapters are, in my opinion, very impressive, even though Talvieno has told me that he feels the same way about Small Choices that I feel about code I wrote a long time ago :shifty: Sadly, he didn't have enough time to devote to the project, so Small Choices has been discontinued in favor of redirecting more effort toward the other three million projects he's got going on :ghost:

    ...creating an LT IRC-Based RPG, which is currently under development and whose progress I am keenly interested in following! Talv may just end up creating LT before I do... :wtf: Seriously, check this one out. It's awesome. Incidentally, the game is played through (and GM'd by) the ever-talented Taiya!

    ...heading up Hawkspears, an 'LT Community DF Succession Game.' I'll admit that, since I've (regrettably) never sunk time into Dwarf Fortress, I don't really understand what's going on here...but the screenshots are pretty and the whole thing is just one more example of Talvieno starting a creative project that engages and excites the LT community!

    ...creating and maintaining the Limit Theory Email/RSS News Thread, an aggregation of my 'important' posts since the time when they started to become...scarce. This helpful thread has allowed people to easily track a sometimes-difficult-to-track developer :oops:
---

All of that is on top of his normal moderator duties. Suffice it to say, Talvieno has worked very hard for and has no doubt earned the title of Limit Theory Community Manager. As of a few weeks ago, I offered him the position and he accepted, but we've been keeping it under wraps until I had the chance to make this post. Finally, it's time!

As CM, Talvieno will continue his normal duties as moderator, but will also act as a point-of-contact between me and the LT community & backers, helping me reduce time spent on activities that aren't directly-related to getting Limit Theory out of the door. For example, I was tempted to ask him to write this post and announce his own CMship, as a demonstration of the time that he would be saving me :ghost: I decided, however, that the least I can do is spend a bit of time on this post considering how much he's already helped. Talvieno has also offered to help with maintaining the website, writing up the aggregation of weekly dev logs for Kickstarter updates, helping handle social media, etc. All of this will result in more time spent on me building LT. It's been said many times before by you guys that I need to get some other hands on-board to help bring Limit Theory to fruition. While I've discussed the difficulties of doing so in the coding department, arguments against hiring an official community manager to help handle the social facets of LT were essentially nonexistent, especially in light of already having the perfect candidate for the job. Well, here you have it!

I wouldn't feel right about this post if I didn't also take a moment to recognize the other LT forum moderators, all of whom have made my life tremendously easier over the course of development. Sincerely, thank you to Bele, DWMagus, Gazz, Grumblesaur, and Just_Ice_Au, our fantastic moderator team. In promoting Talvieno to CM, I absolutely do not want to overlook or belittle the enormous contributions that our other moderators have made to the community! Thank you guys :)

---

So, without further hesitation, please give it up for Talvieno, now the official Community Manager for Limit Theory! :clap:



Link to original: viewtopic.php?f=11&t=5824
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4237
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Wed Mar 08, 2017 8:08 pm

Wednesday, March 8, 2017

My oh my, what a week. It started out unimpressively, but after a few days I got my act together and turned out some very serious work. Today it culminated in the completion of my final goals from last week, and a fun little photo shoot :geek: Since I finally am able to come bearing shinies, I will do a bit less talking this time and a bit more showing!

---

I said last week that I'd be a happy camper if I had my primary assets 100% back, and I'm pleased to say that I am indeed a happy camper!

BoxMeshes: 110%. (PlateMeshes were renamed to the slightly-less-flashy but more-appropriate 'BoxMesh,' since that's really what they are). Rotation was implemented. An extra 10% is credited to the additional functionality of per-axis bevel control, rather than a single bevel value for each box, which makes the LT Core's BoxMeshes even more expressive than those of LTC++. The elements of a BoxMesh can now take on cylindrical and other anisotropic shapes that weren't possible previously, lending more options to the ship & station algorithms.

Scalar Fields / Distance Fields / Surface Extraction: 125%. In addition to a higher-quality implementation than existed previously, the underlying algorithm has changed from marching cubes to surface nets, which is both more elegant and yields nicer mesh quality. Not only were fields implemented in their own right, but 3D textures were implemented at the LT core level with more features than previously, including facilities to easily generate 3D textures from shader output as well as to convert said textures into scalar fields in a single handy function. I had quite some fun playing with fields this week thanks to this functionality. Generating asteroid geometry on the GPU requires perhaps 20 lines of Lua and about 15 lines of GLSL (most of which is just #include and uniform declarations).

Mesh Occlusion & Warps: 100%. All kinks in the occlusion computation are fixed. Mesh warps (spatial distortions) are now implemented and can be used to (efficiently) apply a wide variety of distortions to meshes with a single line of Lua. I also implemented parallel warps (i.e., applied at the same time rather than taking into account previous warps; not to mean parallel in the thread or SIMD sense) in a compact and efficient way, much nicer than the old. Warps are the first construct in the LT core to make use of the epiphany I had last week concerning how to make reference counting work easily in a C library, even accross the API boundary (i.e., behaving correctly with the LuaJIT GC). It's stupidly-simple and I should have thought of it ages ago.

Porting of PCG Algorithms for Asteroids, Ships, and Stations: 100%. Seeing the ship and station algorithms again reminds me that I do need to put some more time into them, but it's refreshing nonetheless to have them back!

Now for some show-not-tell! Note that there's a tiny bit of cheating going on in these screenshots: I rendered them at 4x supersampling since I didn't port SMAA yet and wanted high-quality, non-jaggy shots. The FPS is rather low even on my new laptop, but that's only due to supersampling (I'm drawing 16x more pixels than needed). At normal resolution (1080p) the framerate is, of course, excessively smooth :) Second note: object surfaces (metal, rock) do not look quite the same / quite as good as in the old shots due to the fact that the deferred lighting pipeline is not back (not a huge deal, just requires me to spend some time with shaders). Graphics details weren't the priority here; getting assets back in play was!

I took a lot of screenshots. It was fun and I know you guys have been hungry for some, so you can view the full album here. I hope I'm not giving away too much candy, you still have to enjoy future shots (and visual fidelity will not improve a whole lot from here, other than there being more going on in the shots)! :squirrel:

Favorite ship shots:

Image
Image
Image

Favorite station shots:

Image
Image
Image

Some planets:

Image
Image
Image

Some asteroids:

Image

---

That's it for this week. No LTasm work; the asset work was too rewarding. This week I'll be attempting to sprint on gameplay features in the LT Demo (can't really call it PAX demo anymore), since I'll be showing it again on Saturday at LSU's TEDxLSU event. I'm not a speaker, but the tech park was offered some presentation space there, so I'll be on-campus with a few other devs from LTP, doing something similar to what I was doing at PAX. I'd love to bump up the fun factor of the demo and take the opportunity to crunch some good gameplay work at the same time!

And honestly, I think it's worth stressing: REJOICE! This may look like the LT you knew and loved (hopefully), but it's so much more tractable under the hood. It's free of technical debt and anxiety...erm...yes, a codebase can have anxiety. If I gave you guys the LT core and the Lua code I've got right now, you'd be able to make a great space game with it (without having to hire an artist). I've never been able to say that with confidence before. Of course, I'm going to take a shot at making that great space game first before I let you guys in on it, sorry :ghost:

Time to get sprinting :geek:

PS ~ Apparently 8-day weeks are more natural for me. Keep expecting unaligned devlogs :roll:


Link to original: viewtopic.php?f=30&t=5853
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4237
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Fri Mar 17, 2017 12:17 pm

Friday, March 17, 2017

YES. That is all.

But seriously, about 12 hours ago I was not feeling nearly as happy about the state of LT as I am now. Warning: rambling about performance ahead.

(EDIT: I'm so sorry. This one really went on too long. That's the bad and the good news of writing a log on the tail end of an exciting night of work :ghost: )

---

So as for the beginning of the week, I failed. I just did. I didn't get exciting gameplay implemented in time for TEDxLSU; it was all I could do to bring all of last week's work together and get it all into a system. Granted, with real assets and a pseudo-real star system, this iteration of the demo looked about 1000x as good as the last. But I honestly didn't even put a dent in what I wanted to do gameplay-wise for this iteration, as various problems drained my hours away until there weren't any left. But I'm not going to spend much text on this failure, because I've been grinding hard at trying to make gameplay progress ever since, and have made a lot of progress that's leading up to gameplay in a big way. I don't know where to start really, because I have so many thoughts bouncing around in my head now that it's hard to concentrate, so excuse me if this is just one big braindump :oops:

Even if the hours hadn't been stolen from me by heaps of minor annoyances, I would have still run into the problems that I came up against this week in trying to get some real gameplay going. Those problems, as you've probably already guessed, are actually just one problem: FPLT! Flatfingers rightfully pointed out in the comments to my last log that FPLT can't really be considered solved (or side-stepped) until we see a substantial amount of universe-wide action going on. Unfortunately, I underestimated how badly LJ has been hurting me. It became more evident once I had my assets back in. Achieving that classic LT silky-smooth 60FPS was quickly becoming an uphill battle. I'm sad to say that the LJ profiler hasn't been completely honest with me, and I'm not sure I'll be using it anymore. On Windows, I couldn't even achieve a satisfactory amount of smoothness, and I'm still baffled by how significant the LJ performance difference is between Linux and Windows (yes, I did enough of my own profiling to be quite sure that it's LJ, not the graphics drivers (and certainly not my C code :) )).

This has, honestly, had me pretty down for most of the week. I've been swimming in gameplay ideas and writing down lots of ideas and notes on paper, but I found myself scared to put them into code. Gameplay hinges almost entirely on a robust gameobject / entity system. Problem is, if I can't build a solid such system in LJ, then I'm wasting my time -- it would eventually require a re-write (of which we're oh-so-fond around here... :hmph:), and I'm just not up for another week-long refactoring

Practical Josh was called in to consult on this problem. His analysis: we have good technologies all-around. LJ is very fast for what it is, and writing gameplay code in it would and will be a joy. Drawing the line between the LT Core and Lua is the challenge. Using the tech that we already have in the right places is the trick. This is where I think I've been making some really bad mistakes in my reasoning. Luckily, I'm getting better at seeing them. After a lot of thought, the obvious dawned on me: moving some entity logic into C would not at all be the end of the world. I've been operating under the false assumption that doing so would sacrifice moddability. Using the sensible part of my brain, I realized that there is absolutely no reason to be computing, for example, the Newtonian physics time step (i.e., computing changes to position, velocity, orientation, angular velocity from forces & torques; updating spatial data accordingly) in Lua. No one will ever need to mod it. In fact, this led me to more thinking, and more realizations: bottlenecks are, of course, in the computations that have to happen every frame (especially if they have to happen on a large number of objects). Motion calculations are an obvious culprit. Physics too. Other than that? ...? AI? No way, AI calculations are cheap and absolutely don't need to happen every frame -- not even close. Other logic like weapons firing, trade orders being processed, ore being extracted from an asteroid...? Some may need higher-frequency updates than others, but none are even close to the 60Hz mark at which motion calculations occur. This suggest a hybrid entity system: let the 'core' of the entity (the most basic data) be handled in C. Let LJ 'wrap' that core entity with additional data and functionality based on the type of object. But forget about 'onUpdate.' Seriously, let's just forget about it. 'onEvent' is much, much better -- we'll wake objects and let them know when something requires their attention (the player pressed a button to fire a weapon, a ship collided with something, an order was placed at a market, etc.) That's the beauty of an event system: we waste no computation time; we compute only what must be computed. Previously I didn't have the skill or foresight to see how I could do this effectively. I do now :)

Back to why I'm so exited right now: I worked all night implementing the entity system in the LT Core, pulling out all my optimization cleverness to make that Newton step blazingly-fast. Rendering, too -- this is something that should be handled by the C entity. Lua tells the entity what it looks like, but doesn't perform the draw loop. That's really the key theme of how we're going to defeat FPLT: we're going to pull tight loops out of Lua, because high-level gameplay doesn't happen in them. High-level gameplay happens in response to events or, at worst, in response to a low-frequency timer saying 'yo, update the economy dude.' Essentially, the C core should be responsible for 'steady-state' computations, while Lua should just be notifying the core of differentials in that state -- a force was applied to an object, a new object was spawn, a station module went boom, a trader decided he wants to navigate here rather than there. How big of a difference does this approach make? It was scary running the perf test for the first time.

Right now I only have enough logic in the core to handle simple objects (I haven't yet implemented any kind of event system, but trust me, it's coming). So I chose our lovely, shapely asteroid friends...always a performance problem, even back in LTC++. Updating and drawing 1000 asteroids in LJ: ~10ms. Updating and drawing 1000 asteroids with Lua leveraging the C entity system: ~2ms (in other words, 0ms, because 2ms is approaching the minimal frametime it takes just to do the other stuff). To make the point more clear: 10000 asteroids in LJ: death. 10000 asteroids in LJ/C entities: ~2ms. You didn't read that incorrectly, although I thought I had when I first saw it...! Indeed, folks, we have found our milliseconds! They were hiding in those inner loops, just as I had suspected. 50000 asteroids in LJ: plzstopburningmycpu. 50000 asteroids in LTC++: plzstopburningmycpu. 50000 asteroids in LJ/C: ~30ms (profiled; almost all time spent in GPU rendering; the update and draw loops still contribute almost nothing). And, just so we're clear, that's way more asteroids than you've ever seen in LT. It's just stupid how fast it is :geek: Oh, and LTC++ was even scared to let the asteroids move, because that killed performance. 50000 asteroids with slight rotations and drifts in LJ/C: ~30ms (performing the basis rotation & re-orthogonalizing 50000 objects is, apparently, trivial for modern CPUs and well-optimized C!) I nearly cried when I saw that. It was unreal. In an instant, I went from feeling that I was 'too close to the edge of the cliff' with respect to LTLJ's perf, to feeling 'dear God, this is going to blow LTC++ out of the water ._.'

Sure, we still need physics (as in, collision checking & resolution). But guess what? That, too, will be very much doable with this approach. Thanks to the simple, minimal structure of the C entities, I'll be able to tightly integrate the physics engine. It'll be faster than LTC++ for sure. Speaking of which, I did a lot of thinking about collision checking. I have some very exciting ideas about which I'm stoked. Lots of cool math is involved, but the end result, I believe, will be really, really fast. I've already gotten technical enough, but capsule trees and a progressive, adaptive delaunay triangulation of space using entities as vertices...I think it will be close to optimal in terms of minimizing CPU cycles spent on Physics :)

This is really the kick I needed. I felt that FPLT could be overcome with the right balancing of LJ and C. I think I've proven it to myself at last. I need to implement a scheduling / event / notification / trigger system to really see how beautiful I can make this. I have a feeling that the beauty of gameplay code is going to profit from these developments almost as much as the performance!

---

Sorry, I know it was a ramble. I wish I could just do a Vulcan mind-meld with you guys so you could see all the possibilities I'm seeing right now...I've butchered it with my ever-erroding ability to use words. I'm just...really excited. This success has shown me that I was more worried than I knew about LJ being able to deal with entities robustly. Feels good to have that weight gone :D As usual, we must still be cautious. There are more than a few challenges left to solve. How big of a price will we have to pay when it comes to wrapping the C entities in LJ to add high-level-gameplay-related information? Frankly I don't think it will matter provided I can muster a robust event system in LT Core. But we'll still have to see. Once I have all entities, including those that require more complex rendering (a ship, for example, must render itself as well as thruster effects at a minimum, which requires much more information than an asteroid), then I'll feel even better about it. And once physics is in and working robustly....well..that'll just be the end of it. I'll be one happy gameplay camper.

---

I know this has gone on long enough, but I do want to mention that I was serious about doing some deep gameplay thinking this week. I'm working on some potentially-major changes to the economy in LT in an effort to make things better, simpler, more fun, and more AI-friendly, all at the same time. Don't worry, I'm not talking about cutting depth. I'm talking about re-framing the way I thought of some things before. I think it's fair to say that I fell into the trap of over-complicating things, ironically, in my quest to unify them. I hope to speak more about my ideas soon once I get some more time to flesh them out and, hopefully, implement some of them. FPLT sure has been a downer, but, as with every major struggle, I think it's going to pay off. It's given me the opportunity to come back to my old gameplay concepts with a 'fresh set of eyes'...eyes that are quite a bit more practical and concentrated on the 'big picture,' if I do say so myself! :geek:

---

The coming week will be all about getting the LT Core and LJ to fall in love and have gameplay babies. Yeah, that was a weird way of saying it. I didn't sleep last night, let's blame it on that :roll: But really, I'm so excited to see how this union comes together!

PS ~ I meant to post some screenshots, just so you guys can see what the demo looks like with some 'real' assets. Unfortunately I ran off without my dev machine since I was excited to write this log. I'll try to take some shots and upload them later (in a few hours, hopefully) when I head back to the office.

PPS ~ I finally learned how to use 'restrict' in C, which means I'm no longer so scared of using functions that take potentially-aliasable pointers. Yay for not being scared!

PPPS ~ I worked a bit on a grammar engine (as in, context-free grammars, etc; not grammar as in English grammar) this week. Originally the purpose was for auto-generating my tedious vector math C libraries. However, one thing led to the next and it became quite a powerful tool, one that I believe might play a role somewhere in the plethora of tech being used for LT. It's certainly more powerful than a context-free grammar engine. In particular, I believe it could enable very easy crafting of domain-specific pseudo-languages without having to actually do all that technical stuff of building a compiler. Practically-speaking, I'm envisioning it as a way to write high-level gameplay code that can be mapped to Lua (or C, if necessary) and be resistant to changes in the underlying API. Kind of like how a lot of games use config files to set constants, except this would be half-way between data and code. It could also be used for dialogue generation, and, with adaptation, possibly mesh generation. We'll see. So yeah, quite a few pieces of excitement this week!

PPPPS ~ For the TEDx event I did manage to get LT rendering on my new 4K, 10-bit HDR TV :D It's...pretty breathtaking. I had no idea how much difference the 10-bit HDR makes. It was easy to make LT compatible, since it already renders in HDR. Just had to do proper tonemapping & gamma correction and voila. Looking at the primary star in HDR mode definitely gives you the impression of "yes, that's a bright star." In general the extra contrast & brightness just makes things look incredible. Not to mention 4K making the detail look insane! Despite the underwhelming gameplay, people were generally amazed with the beauty and were very interested to learn about what they'll eventually be able to do inside of that beautiful universe! :D


Link to original: viewtopic.php?f=30&t=5889
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4237
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Sun Mar 26, 2017 8:22 pm

Sunday, March 26, 2017

One of those weeks where loads of things were started, but none were finished. This week was dedicated almost exclusively to working on the entity system in the LT Core. This is, as we've found out from previous experience, where we'll either make or break LT's performance. Sadly, it'll be a rather bland log since I don't have all the solutions yet (getting these particular bits structured correctly is of utmost importance). I'll describe my work-in-progress nonetheless. On a side note, it's been a pretty rough week for me due to unfortunate events in my personal life, so forgive me if I'm not as enthusiastic as usual.

---

Entity Component System in C: In progress. I've been running through lots and lots of ideas on how to structure this bit, since, again, it's part of the overarching entity system that's going to make or break performance. I'm experimenting with several different architectures at the moment. Component-based entities are no doubt the right way to go in games (as opposed to big inheritance chains and/or multiple inheritance), but with them, the devil is in the details. Trade-offs are everywhere. Do we want to be able to add or remove components dynamically? Perhaps not -- but if not, can we still define new types of objects from script efficiently? If not -- can we define a 'generic enough' set of prefab entity types to support all that we want to do from Lua? Should we prefer structure-of-arrays format to save instruction cache misses, or array-of-structures format to save data cache misses (most people people assume the former when speaking of component-based entities; I'm not completely convinced.) Basically, there are a lot of choices to be made here that will have far-reaching consequences for both how gameplay is written in Lua as well as how performant everything is. Only time and experimentation is going to tell which set of choices are optimal :geek:

Notification/Event System: In progress. I spoke in the last log about my excitement for handling gameplay in a fundamentally different way using event-driven programming. I've started implementing the machinery for it in the core and have a few basic event types defined, although I can't test this system effectively until the entity component system is stable. Nonetheless, the more I get into the implementation and the more I shift my mindset to using events rather than 'polling,' the more comfortable I feel about LT's performance. I'm still absolutely convinced that we're going to see incredible results from this paradigm shift! When speaking about FPLT, I've often explained how the performance problems that arose in the old LT were, at their core, due to the sheer number of entities requiring simulation, resulting in a 'death by a thousand papercuts' situation. When the event system is complete, number of entities will no longer be the problem, because any entities that don't require computation on any given frame won't take up a single CPU cycle!

Scripting: I've had some revelations this week about my usage of Lua. In particular, I think I'm using it sub-optimally with respect to scripting the behavior of entities. I have a new idea that I've yet to get around to testing that involves binding an entity's local Lua state very tightly to the C Entity structure. When implemented, this will result in, essentially, me being able to turn off Lua's GC and manage memory explicitly, which will be a huge win. It will also solve the problem of how best to 'wrap' the core entity data with extra scripted data. However, until implemented, I won't know if the memory overhead of this approach will be a deal-breaker or not. I doubt it will be.

Lua...JIT..??: More performance tests this week yielded some startling news: the JIT part of LuaJIT is hardly helping at all. Turning it off entirely results in almost the same performance. This is both good and bad news. The bad news is that I'm obviously using LuaJIT sub-optimally (and it may have to do with the above point). The good news is that if such is the case, then there's more perf waiting to be squeezed out of the LJ parts of the code! I honestly just need to come to a better understanding of Lua and LJ, and perform more measurements to see what's up.

Physics Engine: In progress. Lots and lots of thinking time put into it this week. It's a toughie, but I'm going to nail it sooner than later :) Just wanted to give it a mention since a large amount of brainpower was spent on it.

LT @ NOEW: On Thursday I showed LT off in New Orleans at the 'New Orleans Entrepreneurship Week' event (check it). Since it was less amenable to interaction than the previous events, I wrote a new 'cinematic camera' that flies around a system, smoothly moving and panning while looking for interesting things to look at. Yes, my demo was, essentially, 'Limit Theory Screensaver' :P It was quite lovely and I got very positive feedback on the aesthetics (of course, that's not of particular consequence; we already know LT looks great...the being really fun part is more important now!) One guy even mentioned that he used to love playing Freelancer, and was excited because LT looked like a modern version of it. Naturally this made my day :)

---

In the coming week I'll just be continuing with all of this work and, if the fates align, perhaps even finishing some of it. The day where we leave FPLT in the dust is fast approaching :)

:wave:

PS ~ To those who voiced their concern at the last log: my sleeping patterns were significantly more regular this week :ghost: Don't worry, I did learn my lesson :ghost:






Hey everyone! This is Talvieno, ready to serve up your non-technical summary!

Josh first mentioned that he's been working on how objects (entities) are designed in the code. In code, an "entity" could be a ship, or a station, or an asteroid, for instance, and could all use the same base code. Josh is working on designing this "entity code" so it will run as fast as possible, because with so many "entities" in the game, the quality of this code will really decide what kind of frame rate you see. He's not sure of the best way to do it yet, but he's making steady progress.

Something else Josh worked on: the "notification/event system" lets the program decide where it can safely (temporarily!) forget about things like ships to save even more frame rate (e.g. if a ship is idling, the game doesn't necessarily need to think about that ship all the time). LT runs fast, sure - but that's on his beefy machine. He wants it even faster.

As to LuaJIT (the scripting language that modders can use), Josh discovered that he was using it sub-optimally - kind of like if you were using a paintbrush with the wrong kind of paint. (It's a poor analogy, but hopefully you get the idea.) He's working on fixing that too, again for more performance improvements.

Finally, Josh took an "LT screensaver" to NOEW - the New Orleans Entrepreneurship Week event. This "LT screensaver" had a ship that flew around and looked at things in a system. People loved it, even though they couldn't actually play. Josh was quite pleased with this.


tl;dr: Josh is working on performance improvements so that almost anyone can run LT.


Link to original: viewtopic.php?f=30&t=5925
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4237
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Mon Apr 03, 2017 9:47 pm

Monday, April 3, 2017

Happy April! Boy was it a long week. So much stuff happened it's hard to remember it all, but that's definitely better than the alternative. This week was all about two big pushes toward the completion of two big systems: the entity component system (ECS), and the physics engine. Neither was finished, but major progress was made on both fronts.

---

Entity Component System: Just to quickly recap, this is the part of the engine that powers all objects in the game (hence, the part of the engine most relevant to gameplay). It defines how the data that describes objects like ships, stations, colonies, asteroids, etc. is defined, managed, and altered over time in response to gameplay. From the point-of-view of most would-be LT modders (as well as Gameplay Josh), it's the most important part of the engine, perhaps followed by the UI system.

I spoke last week about the abundance of trade-offs inherent in designing a game engine's ECS. This week I continued to explore that abundance of hard choices by...implementing them. Describing all those choices in their technical detail would take too long and bore you all to tears. I started describing my current implementation in quite some detail in this log, stopped, and deleted it because I'm still too unsure about it :heh: With the lessons I learned from trying several different architectures, I started implementing a different one yesterday. But yesterday is...well, yesterday. Practical Josh, in general, does not feel confident about things being 'the answer' after only having had them around for a day.

But since I don't want to talk implementation details, I'll mentioned a new (to me) development paradigm that I've been using while playing with all of the options. Like many things I do, the idea sounds stupidly-obvious...but it almost felt like a revelation to me when I started doing it. I've been writing a good bit of gameplay pseudocode. Yep. Quelle breakthrough! :P But seriously:

Code: Select all
/* Basic Mining AI Pseudocode. */
start:
  self.clearActions()
  targets = self.getSystem().getEntitiesWithComponent(Component.Harvestable)
  miningTarget = argMax over target in targets of
    target.getInventoryUsed() / self.getTravelTime(target)
  self.pushAction(TravelToObject(miningTarget), align)

align:
  /* Align myself 'tangent'-ish to the target. We could get way fancier here,
     taking into account turret line-of-sight. Quick version is to just assume
     being tangent will yield goood LOS for all. */
  toSurface = miningTarget.getNearestSurfacePoint(self.getPos()) - self.getPos()
  self.pushAction(AlignAxisYTo(toSurface), harvest)

harvest:
  /* TODO: Maybe move around the target a bit so as to be less...boring? */
  /* TODO: Fire each turret at a different (visible) surface point. */

  for turret in self.getTurrets()
    if turret.getType() == TurretType.TransferUnit
      turret.fireAt(miningTarget)

  self.notify(InventoryFull(self), full)
  self.notify(InventoryEmpty(miningTarget), start)
  self.notify(NotInRange(self, miningTarget), reposition)

full:
  self.pushAction(DockAt(dropoff), unload)

reposition:
  /* Calculate a point near the surface of the target such that all transfer
     units are in range but I'm still a safe distance away */
  self.pushAction(TravelToPoint(point), align)

unload:
  self.transferInventory(dropoff, ItemType.RawMaterial)
  self.pushAction(Undock(), start)


It's so simple, incomplete, and really quite boring. Yet amazingly, it helps me in so many ways to think about how I should design the ECS from a practicality standpoint to do the stuff that I'm actually going to want to do. Even writing this silly code, I can get a feel for challenges that aren't obvious when designing from a blind, theoretical perspective. That code isn't real; it's not any real language, just my made-up pseudocode. But what we really want is to design things such that we can write gameplay code in Lua that rivals the simplicity of the code above.

Frankly, this has been equal parts difficult and rewarding. I'm bouncing back-and-forth between the highest-possible levels of gameplay coding (I mean, it's pseudocode, it's higher-level than the high-level gameplay code), and the lowest levels of engine architecture. It's mentally jarring to constantly bounce between those levels. But it's also rewarding to feel that these low-level choices are being made with very concrete goals in mind.

Physics: Great progress and several new structures/algorithms that I've never implemented before in LT, even back in the C++ engine. The LT Core picked up a lot of code. Octrees. AABB trees. K-d trees (unfinished). I have efficient code for building octrees & AABB trees from meshes or arbitrary volumes, as well as for raycasts. Full intersection tests will come soon.

On top of implementing a ton of physics-related data structures and algorithms, I did a lot of testing to determine which ones best solved the previous performance problems with the narrow-phase collision detection on large meshes. Not surprisingly, AABB trees are winning, at least with raycasts. I can do raycasts against a large, ultra-quality station mesh in something like 3 microseconds now. Although broadphase is not really a pressing issue and will come later, it was interesting to see that my tree construction code is fast enough to actually do the dumbest thing possible without killing FPS (that is, to reconstruct the whole acceleration structure every frame, inserting all objects in the system; this is incredibly stupid compared to real broadphases, which exploit temporal coherence to do dramatically less work).

Gratuitous screen from physics work:

Image

And a few more if you want...

Spoiler:      SHOW
Image
Image
Image


Lua & LuaJIT: As per my concerns last week, I did some investigation with respect to how to use LJ most efficiently. I used the profiling framework that I mentioned last month (and have expanded on greatly this week) to guide my investigation. I also started examining LJ bytecode dumps & trace information so that I can better understand how and where I'm losing cycles. The profiling framework has already proven itself to be invaluable!

I now have a better (but still expanding) understanding of writing efficient scripts, especially for gameplay logic. I implemented a system (in the LT Core) for script loading & running that allows better memory usage and performance than my previous monolithic architecture.

---

The coming week should be all about the entity component system. I've made great progress so far with trying a lot of implementations, and I'm getting ever-closer to a design that fits all of the requirements of LT. I'm putting this at max priority and giving myself full permission to do nothing but ECS work for the coming week. I want to start getting 'real' gameplay (not LTDemo gameplay) up; I'm ready for that pseudocode to become reality!

:wave:






Hey everyone! Talvieno here, with another non-technical summary!

Josh has spent most of the past week or so working on the completion of two things - entities and physics. Just to recap, the entities are every object in the game - ships, stations, asteroids, planets, colonies - everything. The physics engine is how these objects interact physically with game space and each other - collisions, acceleration, motion and so on.

You may have noticed that he dumped a ton of "mining code" up there, and may have no idea what he's doing. Josh is basically doing his own version of "Rubber Duck Debugging". Rubber Duck Debugging is an old programmer method to figure out how to fix bugs: you talk to a little rubber duck sitting beside your computer and tell it what the problem is. In talking about the problem, you understand it better, and are often able to solve it much more easily than just from staring at the screen. Josh is writing pseudocode (fake code - "code" that's not actually code) to help him understand his problems, and finding it incredibly helpful.

The physics stuff is much more complicated, but in summary, Josh is working on collision detection. Basically, it splits up the game space into smaller, more manageable areas - splitting them up means the program has less space to check at any one time, which means a faster program. Josh includes a few visualizations of this in the form of shiny pictures.

As to the LuaJIT stuff - he's been experimenting with LuaJIT (the modding/scripting language) to try to find out how to get the most out of it - to squeeze out every last drop of efficiency for maximum framerate.

tl;dr: Josh has worked a lot in several areas the past week, mostly in solving complex problems. He's almost ready to start putting all the pieces of the puzzle together and make something you can actually play as LT.


Link to original: viewtopic.php?f=30&t=5964
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4237
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Thu Apr 13, 2017 8:57 pm

Thursday, April 13, 2017

Rawr. I had almost finished this log when I accidentally closed it before having saved. Why hadn't I been aggressively saving after every few keystrokes like I usually do when I'm in vim? We will never know the answer :ghost: Please forgive me if I'm more terse this second time around...

Last week I gave myself "full permission to do nothing but ECS work." And boy did I take that permission and run with it. Frustratingly, I'm not yet able to come back with a happy 'I GOT IT YAYYY!' like I wanted to.

---

Entity Component System: What can I say? I am finding that building this thing requires taking into account a massive number of factors at all times, making it slow and mentally-taxing. I showed last week how I've been writing gameplay pseudocode to help guide me as I architect this thing. In a continued spirit of practicality, I did more of that this week, but additionally went so far as to write down a detailed list of every entity that I anticipate being in LT 1.0, along with every 'component' / piece of data and functionality that I anticipate said entities to need. At least this moves some of the factors I must consider onto the screen and out of my brain.

Doing so was quite revealling: it reinforced in my mind that very few components require 'special treatment' for performance. Virtually everything that does require special treatment can be grouped under a broad interpretation of 'related to physics.' Thrusters are the odd-man-out, as they require special treatment for rendering purposes. Whether I can gerrymander said special treatment into the realm of Lua via some magic or another (I do have ideas) is still unknown. Basically, I'd like to be able to fence off all such special treatment and shrink it to the minimal viable subset. I'd then like to draw a big line in the sand between C and Lua that makes said special treatment 'not the concern' of the gameplay programmer (aka me + mod writers). This is a Really Hard™ problem that I've tried attacking from a million different angles and will continue to attack from a million more until one angle finally prevails.

The TL;DR of the technical details is: they haven't been solidified. The technique to which I alluded in my last log (as well as the long post I made in that thread, in which some details of that ECS architecture were revealed as the 'first level' of the mod API) has been put aside after further consideration as being over-engineered. If it comes down to it, I'll come back to that one. But my review of the actual needs of LT gameplay showed me that it's probably overkill to use Lua to give metacode-style definitions of ECS-related data to the engine, which then JIT compiles them to native code. I now believe a less 'heavy-handed' approach is feasible by drawing the aforementioned line in the sand carefully enough. I'm currently working on such an approach.

Mod Loader / Hooks / Environment: Believe it or not, one of the easiest ways for me to audition ECS ideas is to treat them like mods. All it really does is force me to write highly-compartmentalized logic and to think in terms of events and event hooks (which I need to be doing). The end result is that I can swap out different ECS ideas (at least on the Lua front) without breaking other things.

To get this all working, I of course had to write a basic mod loader (which is very, very easy in Lua), define some initial engine-level event hooks, and hammer down a format for mods. It was all quite rewarding. Mods, at least those that operate only on code (rather than adding or replacing new non-code assets) already work, because I'm implementing my ECS ideas that way, which makes prototyping very quick!

I developed several neat facilities for the mod system. Sandboxing was a big one (mods can't access arbitrary stuff from the outside world), and Lua again makes this very easy. It's easy to make sure that mods don't, for example, plug into the filesystem and start deleting your C drive. The way it works currently is that mods have to explicitly request access to the APIs that they need. Without an API request, mods have virtually no power (they can't even use basic (unsafe) Lua statements like require!) So if I want to use the function 'Draw.Rect' anywhere in my mod's code, I need to add 'Draw' to the list of APIs that is requests usage of (which is done by adding to a list in an 'info' table that every mod defines which contains name, description, version, dependencies, apis, and more). If I don't, the engine won't inject the API into my mod's environment, so my mod will simply cause a crash when it tries to use a function from the Draw API. This is very cool, because it allows the game tight control over what mods do, and can provide modders and users alike with detailed information. No hidden network access attempts :) It also makes it easy for me to reason about my own code dependencies while I develop. And for one more bonus, it makes debugging easier, because I can see exactly what 'mod' (remember that I'm using them right now to prototype things) caused a crash or even issued a print statement.

Physically-Based Rendering Stuff: Ok, so I admit that, after more than a week of waging ECS war, I needed a break yesterday evening. I gave the evening to graphics Josh :monkey: I implemented the 'blind-you-with-all-the-things' style of HDR boom that seems so characteristic of high-end engines these days. It does look impressive. I'll turn it down a lot though...with it fully-enabled, the lighting in LT is literally blindingly-gorgeous :P

I also ported some of the PBR stuff from LTC++. I don't have the stamina since this is my second go at writing the log to talk details, so I'll just spew words this time. GGX environment map generation, better math this time (so arctangent, very sqrt, much spherical coordinates...wooww), faster as well; lighting just looks better with PBR yayy; combine it with above bloom tech and wowww beautiful reflections, eyes bleeding in all the right places. There you have it.

---

Guess what I'm (still) giving myself full permission to do in the coming week? Yes. ECS. Send your mental clarity vibes, I need them.

:think:




Hello again everyone! Talvieno here with a non-technical summary of this week's devlog.

Since our last update, Josh has been trying to figure out the ECS (Entity Component System) - the thing that makes ships, stations, asteroids, and more. It's a tough problem to work out. He wants modders to be able to access the bits they need (in Lua), while still keeping the slower portions in the faster C code (the LT core). Figuring out which bits go where (and how to make them all work together) is kind of tricky, but he's working on it.

Part of working on it is actually making "mods" to test it out. Josh can now load "mods" into the game, which is helping him figure out how to structure the ECS. As an added bonus, he's fixed the modloading process so that you can't make mods that do things they're not supposed to - like deleting everything on your hard drive or connecting to the Internet and downloading things. This is pretty much a necessity anyway, but he's come up with a clever way of doing it that he's proud of.

Finally - he made the game shinier. The graphics monkey has escaped, ladies and gentlemen - can we ever put it back into its cage now that it's released? :thinking: (It was only an evening's worth of graphic monkey, though, so we should be fine. ;) ) He implemented HDR bloom - a shiny glow you typically see from particularly bright light sources like sunlight. He's also ported some of his PBR code from the old LT version, so LT looks very pretty. (PBR is a lighting model that attempts to make materials model the flow of light the same way as you would see in the real world. This means more realistic-looking textures.)

tl;dr: Josh is working on the whole ship/station/asteroid problem and it's more difficult than he expected. He also added in some pretty graphics additions yesterday.


Link to original: viewtopic.php?f=30&t=5993
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4237
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Sun Apr 23, 2017 9:50 pm

Sunday, April 23, 2017

It's not a perfect resolution of the grand war against entities, but this week has brought out the fiercest practicality that I can possibly muster, and for that I think we'll all be grateful. It is time to move on, dear friends, from this monstrous black hole of time and effort. Brighter prospects lie ahead.

---

Entity Component System: So...we're all tired of it by now, right? Me too. After many more hours drifted by this week with no sign of bringing me the end-all-be-all solution I've been wanting, I made what I consider to be a very practical compromise: I said "screw it." Since such crass language isn't allowed in my treasured notebooks, I instead wrote something marginally more profound: "design for the rule, not the exception." I'm sure someone has said that before. But now I'm saying it again. After enough neural burning on the problem, I decided it's rather absurd to continue this temporal bonfire (yes, LT development sometimes involves lighting time itself on fire) when the edge cases, not the majority ones, are holding back the design. I banged out an ECS that covers the 90% and gave myself permission to worry about thruster trails later. Here's my theatrical rendition of the past two weeks (Boss is, of course, practical Josh in disguise):

Boss : "Build me that ECS."
Josh : "Aye sir."

< 1 week passes. >

Boss : "Where's my ECS."
Josh : "Working on it."
Boss : "Get it done kid."

< 1 week passes. >

Boss : "Where's my ECS."
Josh : "I have a lot of good designs."
Boss : "Neat. So where's my ECS."
Josh : "Well sir, currently none of them can elegantly accommodate the fact that thrusters have a key part of their rendering behavior tied to one little piece of state that should, in earnest, live on the Lua side, as it's really not 'intrinsic' to anything. Sadly, in the case of thrusters, there are enough of them that the data martialling will become problematic and result in framerate loss. On the other hand, it's difficult to see where this state should live on the C side of the system."
Boss : "...that right? So...just...thrusters..."
Josh : "Well sir the general problem is likely to crop up in a few other pla--"
< Boss raises hand so as to silence Josh. >
Boss : "It's all so...incredible. So fascinating. But see here, look Josh, let me ask you just this one thing."
Josh : "...yes sir?"
Boss : "WHERE. IS. MY. <expletives censored> ENTITY COMPONENT SYSTEM?"
Josh : *Quickly hands over imperfect but functional system*

< Boss storms away angrily. Cornelius Evazan enters from stage left. >

Cornelius : "You just watch yourself."
Josh : "I'll be careful."
Cornelius : "You'll be dead!"


And there you have it. Kinematics and a few other pieces of physics-related information, along with graphical information, live in C. The rest (requiring no per-frame updating) lives in Lua. Components are Lua-side. So really, it's just an ES that lives in C...an Entity System without the components, since I decided these are, in the 90% of cases, lightweight enough to be handled by Lua without issue. And so it will be until something starts causing major issues.

RenderQueues: A nice advancement necessitated by this week's ECS work was a core rendering feature missing from all previous version of the LT Engine: the 'RenderQueue.' Truthfully, the RQ type is little more than a 'modern' implementation of something OpenGL has had since the dawn of mankind (GL called them 'Display Lists'). Since modern man has decided that OpenGL's display lists were a bad choice to put in a 3D API, we've deprecated them (along with 90% of the GL API). Hence why I implemented them as a first-class engine feature (driver support for display lists, unlike with many deprecated features, actually is a problem). The feature is little more than exactly what it sounds like: a queue of rendering commands; essentially a data structure that encodes a miniature rendering program. This allows rich flexibility to specify rendering operations in Lua by building a RenderQueue, but allows the operations to be executed at the full, blistering speed of a tight C loop. In fact, I've some notion that, since RQs are basically implemented as mini bytecode interpreters (i.e., a decode loop with a switch statement), they might outperform my previous implementations of certain rendering operations (thanks to the fact that all of the relevant GL code is sitting right inside that hot loop; external dispatches that aren't GL calls are nowhere to be seen). But all that aside, the point is that I needed a flexible and fast way to specify rendering operations from Lua, and I've now got it.

I previously had an engine construct called "Model" that handled this type of thing; it was basically a list of Material/Mesh pairs, telling the engine what meshes to render (e.g., for a ship), and what material to use for each mesh (a material was itself a combination of a Shader and various shader parameters). Upon starting to write my Model class into the LT Core, I saw the light, realizing that all of these various constructs that I used to have (Model, Material, ShaderState (a bunch of shader parameters)) could be unified by a structure that encapsulates exactly what they are, down to the core: just some rendering operations and various shufflings of state. Now I can think of a Material as being a RenderQueue that binds a shader and sets up some shader parameters specific to that material. I can think of a Model as being an RQ that calls a sub-RQ corresponding to a material, then draws a mesh, etc. RQs can be nested within one-another to encapsulate the exact kind of sharing that I used to think of in terms of all these disparate constructs that I no longer need! Yay unification. Rendering life is quite simpler now. To think the GL guys had this insight in 3000 B.C. or whenever it was that GL came about :monkey:

I have some dangerous notions about how we might get cute with dynamically building RQs to achieve various optimizations. I'll try to steer my mind away for now.

Mods, Content, The Works: As you'll recall, my objective at the moment is to write the Lua-side LT code in 'mod' format, so as to make life easy on myself while simultaneously ensuring that I know how everything is going to come together with respect to actual mods. I've made a good many changes and simplifications to the format this week to help myself pump out that core gameplay code. I was honestly getting a little overwhelmed and turned around with my previous system of hooks and so forth. I've simplified the mod format to be much more like a 'data description,' which makes things cleaner and easier to reason about (without any sacrifice in flexibility). This is one thing I really love about Lua. It's a beautiful language for describing data, given that the one and only 'advanced' datatype is a table, and data is, after all, just tables inside tables inside tables. Throw in the ability to bind table fields to actual code/functions, and you've got all the power you need to build a piece of 'data' that tells the engine how to handle whatever it is that the data is defining.

I now just have a 'content' sub-table within my mod description table that contains any number of entries. Those entries represent...well...the content of the mod! Right now, they can be entity type definitions (ship, station, planet, etc.), event listeners (to add functionality in specific places), component definitions (define new components for use with the ECS), and miscellaneous type definitions (I'm still not sure what to call things like 'Item' and 'Corporation' and so forth...they're datatypes of course, but don't need to be handled as 'entities' by the ECS).

Stupid example of what I'm working on right now:
Code: Select all
return {
  id      = '_LTBE_SYSTEM',
  name    = 'Limit Theory Base Entities: System',
  desc    = 'Defines the System Entity',
  author  = 'Josh',
  version = 0.01,

  content = {
    {
      type = 'Entity',
      id = 'System',
      components = {
        'CollisionDetector',
        'EntityContainer',
        'NaturalResources',
        'Zones',
      },
    },

    {
      type = 'Listener',
      id = 'System.onCreate',
      fn = function (Game, e, seed)
        print('Looky, a system!')
        local rng = RNG.Create(seed)
        e.name = 'Omicron ' .. tostring(rng:getInt(10, 20))
        e.starDir = rng:getDirection3()
        e.starColor = rng:getVec3(0.5, 1.0)
        -- Populate the system with...you know...lots of stuff
      end,
    },

    {
      type = 'Listener',
      id = 'Engine.onInit',
      fn = function (Game)
        local seed = 1337
        local system = Game:createEntity('System', seed)
        -- TODO: Implement Limit Theory
      end,
    },
  }
}


Once all content is loaded from mod files (or base files, in this case), the engine pulls together all these bits of content and resolves everything to figure out exactly what every entity consists of, what events should be fired when, etc. Less manual labor than before. FYI the content creation code there is obviously placeholder :P

It has a ways to go to be sure; we can certainly pretty up the syntax later. Right now I'm interested in getting all of this stuff working. It honestly strains my brain a bit learning to think in this new manner of events and handlers rather than the old, linear update logic. But the straightforward 'dumbness' of the old update logic is exactly what killed performance, so it's time I learned to man up and think asynchronously.

---

In the coming week I have another LT showing at LSU on Friday, so my goal will be pretty straightforward: I'd like to see LTDemo running on the 'real' systems: using the C entity code, event-driven rather than monolithic update loops, etc. I'd honestly be happy to have that all working with the same level of functionality as before, because that would mean that I'd be in good shape to rapidly move forward with all systems sitting on a solid foundation. It'd be a major step. At present I'm not entirely sure how many man-hours of work that will entail, but I do intend to find out :)




Hey everyone! Talvieno here, with another week's edition of The Non-Technical LT Devlog.

This week, Josh focused hard on the whole ECS problem and got it solved - at least to the point that it's "good enough". He now has a single unified structure with which he can handle ships, stations, asteroids, planets, and other objects. This will make implementing content much faster. OldJosh is not pleased with this development and would rather he have a complete solution, but PracticalJosh - or NewJosh - has won out here. I think this is what everyone was hoping for. :)

RenderQueues: Exactly what it sounds like. Josh has implemented a unification of rendering commands that'll make it a lot easier to progress in the future. You can think of it like a to-do list for the engine's rendering component - the part that draws everything to the screen. Everything has its own little to-do list now, and can be modified/controlled via Lua - which means it can be modded.

Mods, Content, The Works: Essentially, this is Josh talking about how he's simplified the modding format to make it easier to use. There's not really much else to be said; I think that just about sums it up. At any rate, this next week he's going to try to focus on getting the LT code up and running for another showing at Louisiana State University.

tl;dr: Finished the entity component system, and set up the framework to make it much easier to add content, as he's getting to that point.



Link to original: viewtopic.php?f=30&t=6021
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4237
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Previous

Return to Technical



Who is online

Users browsing this forum: No registered users and 4 guests