Note from Talvieno:
We had two major updates in rapid succession, so I'll be posting them both here.

---

Tuesday, February 28, 2017

A rather solid but unremarkable week; I feel this week could aptly be described as a lingering shadow of last week. For both better and worse, last week's "I started on" could basically be replaced with "I finished" or "I almost finished," and the log would be nearly complete. Thus, as has been falsely stated many times now, we may actually see a reasonably-sized devlog tonight!? (EDIT: at this point, it's a tradition... )

---

PlateMeshes: 90+% complete in the LT core. As discussed last week, I emphasized building a release-quality implementation since they became an invaluable tool in LTC++. I now have a high-quality, fast, C library implementation of all of the old PM functionality with the exception of rotation of individual plates. I will need to spend a bit more time in order to finish that bit efficiently, but the rest of the work was by far the bulk. Frankly it required more time than I would have liked, but all of that time was spent adding supporting functionality to the core in order to efficiently support a good PM implementation. The good thing about this kind of time is that it's "permanent progress," as is all progress of the LT core (which is, as a reminder, designed to be the 'etched in stone' part of the LT engine).

Mesh 'Warps': 20% complete (I figured out how to do them). These are the things that allowed me to distort models in cool ways that added curvature and interesting global features to ships and stations. Unfortunately, their implementation in C++ using all that fancy (and incredibly-nasty) template stuff I used to do made it fairly easy. I had some trouble figuring out how to do the same thing effectively in a clean, C API (i.e., making it easy to build warping functions from LuaJIT). I've designed the implementation, about which I'm pleased. I haven't coded it yet. Interestingly, solving this design problem brought me insights on how to make reference-counting work cleanly across API boundaries (for example, to enable both the core and LuaJIT to hold pointers to the same object and still have all memory management work correctly). I've not had this insight before, but when the time invariably comes that I need to do this (beyond warps, which will require it from the get-go), I now know how to do it painlessly.

Networking & 'Control Panel': 100%/90% complete (server/client), and completely new to any of the LT engines! Yes, this is where about 2.5 days of my time went this week, but it's a technology investment that I'm betting will pay dividends. While working with planets, it occurred to me that I would really like a UI for changing the procedural algorithm parameters in real-time to see exactly how they're affecting the final result (with complex procedural algorithms it can be very difficult to reason about the effect of individual parameters). I didn't want to spend a huge amount of time building something just for this task, though, so, inspired by a friend at LTP who does web development, I decided to use a tool that already exists and is very fast and easy for creating quick UIs: the modern web browser. This necessitated networking functions in the LT core, which I had promised I would provide for modders who want to attempt 'multiplayer LT' mods anyway. So, I implemented a multi-platform TCP & UDP socket API, then used it from LuaJIT to spin up a very simple HTTP server run by in the background by the engine that serves up a simple control panel webpage when connected to via a browser. It can change internal engine variables automatically in response to controls like sliders or buttons being changed in the panel.

Though I'm hoping the ability to connect to and communicate with the engine via networking will pay substantial dividends over the remaining development time, having the control panel available is already really helpful: I have a really easy way to expose real-time-tweakable parameters, and, as a bonus, since it's based on sockets & HTTP, I can even connect remotely so that I can use one machine (like my tablet) exclusively as my 'controller' while I develop on the other. I'm also thinking this will be great for monitoring and managing long-running universe simulations (when I get to that point, ideally sooner rather than later)! I'd even like to explore using this for asynchronous resource generation on the GPU, which is traditionally difficult to do in a single process (due to threaded GL contexts being messy). There are a whole lot of ways I can envision using even this very basic networking tech Also, of course, the tools are now there (in theory) for anyone who wants to give an LT Multiplayer mod a shot at release (in practice, good luck, I recommend putting together a team ).

---

In the LTASM world, I did exactly what I had mentioned and hoped for last week: basic x87 floating point operations. Quite happy to have that stuff working. I've got enough to, for example, do most if not all of the basic FP math that you might do in the LT logic, although I don't necessarily have all of the opcodes for doing things in the most efficient way. Frankly, though, my first goal in this world of Plan Z is to have a script -> machine code pipeline working at all. That will be amazing enough for me, then we can worry about micro-optimization. My tests this week with FP instructions in simple loops are already blazingly-fast. I am getting a stronger and stronger feeling that this work will play a role in LT even if LuaJIT continues to hold. In fact, I can see it being a beautiful addition to LJ! LJ is basically a dynamic asm generator, why not have a static asm generator for perf-critical code? Man does it make a difference getting down to that raw level (I mean compared to something like Lua, not C/++).

Speaking of which, as usual, I spent some quality time with the Intel manuals this week, covering chapters 10 & 11 (SSE, SSE2), and skimming 12, 14, & 15, (SSE3/4, AVX/AVX2, AVX512). So basically, I learned a whole lot about vector / 'SIMD' instruction sets this week. Fun as always (Can you believe AVX512 has 2KB of register space for FP math?? Insane...)

---

It's worth mentioning that I spent a good bit of time this week on other, non-coding work. This week saw an abundance of emails, real-life events, and a bit more time spent with the forum than usual. Speaking of which, I assume that most of you have seen the announcement of Talvieno as Community Manager by now, but if not, check it out Although it took a bit of time this week to write that post and get everything squared away, his new role as CM will ensure that the amount of non-coding time spent during future weeks is kept to a minimum! I'm really happy to have Talvieno (officially) on board.

---

No, I'm not terribly impressed with this week (maybe I'm just tired right now...looking at the log, it seems fairly substantial?), next week should see less clerical work as well as less of having to finish up things, and more of getting to start new things. Seriously, next week I hope to see ships and stations in their full glory: PlateMeshes at 100% (rotations are definitely needed for stations), mesh warps implemented, procedural algorithms 100% ported, mesh ambient occlusion calculations 100% complete. I'd like to see asteroids as well, meaning scalar field extraction in the C core. If I have all of those things in a week, I'll be happy! I'll also have a good-looking, content-filled system, presumably. On the asm side, perhaps basic SSE functionality; let's say scalar operations at first, so that I can do the equivalent operations that the x87 FPU would do, but using the more-modern SSE (interestingly, the SSE instruction set is not just for vector processing, it has scalar (single-element) instructions as well).

Until some time in the next 7±2 days...!



Link to original: http://forums.ltheory.com/viewtopic.php?f=30&t=5825

---

Greetings LT forumers!

Those of you who have spent any substantial amount of time here have no doubt encountered Talvieno at some point or another. It's hard not to notice his outstanding and inexhaustible dedication to making the Limit Theory Forums a better place. In fact, it's no secret that a lot of you even wish for him to have more power ('Talv for president!') Well, you've already read the title, so you can guess what's coming Before we get to that part, however, I'd like to take a moment to recognize the myriad ways in which Talvieno has made the (LT)world a better place -- some of you may not know about all of them; indeed, even I had to have an IRC chat with him just to make sure I had everything down right...it's awfully hard to keep up with all the things this guy does! Things like...

• ...building Taiya, our favorite resident Limit Theory forum & IRC RSS feed bot. For those who don't know Taiya, she's not only a sweetheart of an IRC chat bot with a boatload of neat functions, she's also the toughest forum policewoman in existence. We get hit by, on average, 2-3 new bot registrations a day (which is down a bit from previous months, where it peaked around 15/day). A lot of these bots post spam topics. Taiya shuts them down almost instantly, banning those poor bots of lesser intellect than her and moving the spam to a hidden 'trash' sub-forum where mods and I can look at it later. There's no telling how many bans Taiya has issued, nor how many spam threads she's killed. She's made my life so much easier! Thanks Taiya!
  
  ...running the Limit Theory Fan Competition, an awesome idea started way back in the day by the legendary Katorone and re-ignited by none other than Talvieno. The current LTFC ends this Saturday, so check it out if you want to participate (there are prizes!)
  
  ...conceiving of and running REKT, an incredibly-impressive forum-based RPG whose main thread ran for 181 pages and over 2700 posts. If you've ever wondered just how intense, creative, and amazing roleplaying on a forum can get, you need only take a peek at the REKT wiki, which is both a triumph of creativity and an inspiration to me. Talvieno informed me that most of the wiki is actually attributed to the hard work of Dinosawer, so hats off to him!
  
  ...starting a novel based on the Limit Theory Universe, entitled Small Choices. The existing chapters are, in my opinion, very impressive, even though Talvieno has told me that he feels the same way about Small Choices that I feel about code I wrote a long time ago Sadly, he didn't have enough time to devote to the project, so Small Choices has been discontinued in favor of redirecting more effort toward the other three million projects he's got going on
  
  ...creating an LT IRC-Based RPG, which is currently under development and whose progress I am keenly interested in following! Talv may just end up creating LT before I do... Seriously, check this one out. It's awesome. Incidentally, the game is played through (and GM'd by) the ever-talented Taiya!
  
  ...heading up Hawkspears, an 'LT Community DF Succession Game.' I'll admit that, since I've (regrettably) never sunk time into Dwarf Fortress, I don't really understand what's going on here...but the screenshots are pretty and the whole thing is just one more example of Talvieno starting a creative project that engages and excites the LT community!
  
  ...creating and maintaining the Limit Theory Email/RSS News Thread, an aggregation of my 'important' posts since the time when they started to become...scarce. This helpful thread has allowed people to easily track a sometimes-difficult-to-track developer

---

All of that is on top of his normal moderator duties. Suffice it to say, Talvieno has worked very hard for and has no doubt earned the title of Limit Theory Community Manager. As of a few weeks ago, I offered him the position and he accepted, but we've been keeping it under wraps until I had the chance to make this post. Finally, it's time!

As CM, Talvieno will continue his normal duties as moderator, but will also act as a point-of-contact between me and the LT community & backers, helping me reduce time spent on activities that aren't directly-related to getting Limit Theory out of the door. For example, I was tempted to ask him to write this post and announce his own CMship, as a demonstration of the time that he would be saving me I decided, however, that the least I can do is spend a bit of time on this post considering how much he's already helped. Talvieno has also offered to help with maintaining the website, writing up the aggregation of weekly dev logs for Kickstarter updates, helping handle social media, etc. All of this will result in more time spent on me building LT. It's been said many times before by you guys that I need to get some other hands on-board to help bring Limit Theory to fruition. While I've discussed the difficulties of doing so in the coding department, arguments against hiring an official community manager to help handle the social facets of LT were essentially nonexistent, especially in light of already having the perfect candidate for the job. Well, here you have it!

I wouldn't feel right about this post if I didn't also take a moment to recognize the other LT forum moderators, all of whom have made my life tremendously easier over the course of development. Sincerely, thank you to Bele, DWMagus, Gazz, Grumblesaur, and Just_Ice_Au, our fantastic moderator team. In promoting Talvieno to CM, I absolutely do not want to overlook or belittle the enormous contributions that our other moderators have made to the community! Thank you guys

---

So, without further hesitation, please give it up for Talvieno, now the official Community Manager for Limit Theory!



Link to original: http://forums.ltheory.com/viewtopic.php?f=11&t=5824

Wednesday, March 8, 2017

My oh my, what a week. It started out unimpressively, but after a few days I got my act together and turned out some very serious work. Today it culminated in the completion of my final goals from last week, and a fun little photo shoot Since I finally am able to come bearing shinies, I will do a bit less talking this time and a bit more showing!

---

I said last week that I'd be a happy camper if I had my primary assets 100% back, and I'm pleased to say that I am indeed a happy camper!

BoxMeshes: 110%. (PlateMeshes were renamed to the slightly-less-flashy but more-appropriate 'BoxMesh,' since that's really what they are). Rotation was implemented. An extra 10% is credited to the additional functionality of per-axis bevel control, rather than a single bevel value for each box, which makes the LT Core's BoxMeshes even more expressive than those of LTC++. The elements of a BoxMesh can now take on cylindrical and other anisotropic shapes that weren't possible previously, lending more options to the ship & station algorithms.

Scalar Fields / Distance Fields / Surface Extraction: 125%. In addition to a higher-quality implementation than existed previously, the underlying algorithm has changed from marching cubes to surface nets, which is both more elegant and yields nicer mesh quality. Not only were fields implemented in their own right, but 3D textures were implemented at the LT core level with more features than previously, including facilities to easily generate 3D textures from shader output as well as to convert said textures into scalar fields in a single handy function. I had quite some fun playing with fields this week thanks to this functionality. Generating asteroid geometry on the GPU requires perhaps 20 lines of Lua and about 15 lines of GLSL (most of which is just #include and uniform declarations).

Mesh Occlusion & Warps: 100%. All kinks in the occlusion computation are fixed. Mesh warps (spatial distortions) are now implemented and can be used to (efficiently) apply a wide variety of distortions to meshes with a single line of Lua. I also implemented parallel warps (i.e., applied at the same time rather than taking into account previous warps; not to mean parallel in the thread or SIMD sense) in a compact and efficient way, much nicer than the old. Warps are the first construct in the LT core to make use of the epiphany I had last week concerning how to make reference counting work easily in a C library, even accross the API boundary (i.e., behaving correctly with the LuaJIT GC). It's stupidly-simple and I should have thought of it ages ago.

Porting of PCG Algorithms for Asteroids, Ships, and Stations: 100%. Seeing the ship and station algorithms again reminds me that I do need to put some more time into them, but it's refreshing nonetheless to have them back!

Now for some show-not-tell! Note that there's a tiny bit of cheating going on in these screenshots: I rendered them at 4x supersampling since I didn't port SMAA yet and wanted high-quality, non-jaggy shots. The FPS is rather low even on my new laptop, but that's only due to supersampling (I'm drawing 16x more pixels than needed). At normal resolution (1080p) the framerate is, of course, excessively smooth Second note: object surfaces (metal, rock) do not look quite the same / quite as good as in the old shots due to the fact that the deferred lighting pipeline is not back (not a huge deal, just requires me to spend some time with shaders). Graphics details weren't the priority here; getting assets back in play was!

I took a lot of screenshots. It was fun and I know you guys have been hungry for some, so you can view the full album here. I hope I'm not giving away too much candy, you still have to enjoy future shots (and visual fidelity will not improve a whole lot from here, other than there being more going on in the shots)!

Favorite ship shots:
Favorite station shots:
Some planets:
Some asteroids:
---

That's it for this week. No LTasm work; the asset work was too rewarding. This week I'll be attempting to sprint on gameplay features in the LT Demo (can't really call it PAX demo anymore), since I'll be showing it again on Saturday at LSU's TEDxLSU event. I'm not a speaker, but the tech park was offered some presentation space there, so I'll be on-campus with a few other devs from LTP, doing something similar to what I was doing at PAX. I'd love to bump up the fun factor of the demo and take the opportunity to crunch some good gameplay work at the same time!

And honestly, I think it's worth stressing: REJOICE! This may look like the LT you knew and loved (hopefully), but it's so much more tractable under the hood. It's free of technical debt and anxiety...erm...yes, a codebase can have anxiety. If I gave you guys the LT core and the Lua code I've got right now, you'd be able to make a great space game with it (without having to hire an artist). I've never been able to say that with confidence before. Of course, I'm going to take a shot at making that great space game first before I let you guys in on it, sorry

Time to get sprinting

PS ~ Apparently 8-day weeks are more natural for me. Keep expecting unaligned devlogs


Link to original: http://forums.ltheory.com/viewtopic.php?f=30&t=5853

Friday, March 17, 2017

YES. That is all.

But seriously, about 12 hours ago I was not feeling nearly as happy about the state of LT as I am now. Warning: rambling about performance ahead.

(EDIT: I'm so sorry. This one really went on too long. That's the bad and the good news of writing a log on the tail end of an exciting night of work )

---

So as for the beginning of the week, I failed. I just did. I didn't get exciting gameplay implemented in time for TEDxLSU; it was all I could do to bring all of last week's work together and get it all into a system. Granted, with real assets and a pseudo-real star system, this iteration of the demo looked about 1000x as good as the last. But I honestly didn't even put a dent in what I wanted to do gameplay-wise for this iteration, as various problems drained my hours away until there weren't any left. But I'm not going to spend much text on this failure, because I've been grinding hard at trying to make gameplay progress ever since, and have made a lot of progress that's leading up to gameplay in a big way. I don't know where to start really, because I have so many thoughts bouncing around in my head now that it's hard to concentrate, so excuse me if this is just one big braindump

Even if the hours hadn't been stolen from me by heaps of minor annoyances, I would have still run into the problems that I came up against this week in trying to get some real gameplay going. Those problems, as you've probably already guessed, are actually just one problem: FPLT! Flatfingers rightfully pointed out in the comments to my last log that FPLT can't really be considered solved (or side-stepped) until we see a substantial amount of universe-wide action going on. Unfortunately, I underestimated how badly LJ has been hurting me. It became more evident once I had my assets back in. Achieving that classic LT silky-smooth 60FPS was quickly becoming an uphill battle. I'm sad to say that the LJ profiler hasn't been completely honest with me, and I'm not sure I'll be using it anymore. On Windows, I couldn't even achieve a satisfactory amount of smoothness, and I'm still baffled by how significant the LJ performance difference is between Linux and Windows (yes, I did enough of my own profiling to be quite sure that it's LJ, not the graphics drivers (and certainly not my C code )).

This has, honestly, had me pretty down for most of the week. I've been swimming in gameplay ideas and writing down lots of ideas and notes on paper, but I found myself scared to put them into code. Gameplay hinges almost entirely on a robust gameobject / entity system. Problem is, if I can't build a solid such system in LJ, then I'm wasting my time -- it would eventually require a re-write (of which we're oh-so-fond around here... :hmph:), and I'm just not up for another week-long refactoring

Practical Josh was called in to consult on this problem. His analysis: we have good technologies all-around. LJ is very fast for what it is, and writing gameplay code in it would and will be a joy. Drawing the line between the LT Core and Lua is the challenge. Using the tech that we already have in the right places is the trick. This is where I think I've been making some really bad mistakes in my reasoning. Luckily, I'm getting better at seeing them. After a lot of thought, the obvious dawned on me: moving some entity logic into C would not at all be the end of the world. I've been operating under the false assumption that doing so would sacrifice moddability. Using the sensible part of my brain, I realized that there is absolutely no reason to be computing, for example, the Newtonian physics time step (i.e., computing changes to position, velocity, orientation, angular velocity from forces & torques; updating spatial data accordingly) in Lua. No one will ever need to mod it. In fact, this led me to more thinking, and more realizations: bottlenecks are, of course, in the computations that have to happen every frame (especially if they have to happen on a large number of objects). Motion calculations are an obvious culprit. Physics too. Other than that? ...? AI? No way, AI calculations are cheap and absolutely don't need to happen every frame -- not even close. Other logic like weapons firing, trade orders being processed, ore being extracted from an asteroid...? Some may need higher-frequency updates than others, but none are even close to the 60Hz mark at which motion calculations occur. This suggest a hybrid entity system: let the 'core' of the entity (the most basic data) be handled in C. Let LJ 'wrap' that core entity with additional data and functionality based on the type of object. But forget about 'onUpdate.' Seriously, let's just forget about it. 'onEvent' is much, much better -- we'll wake objects and let them know when something requires their attention (the player pressed a button to fire a weapon, a ship collided with something, an order was placed at a market, etc.) That's the beauty of an event system: we waste no computation time; we compute only what must be computed. Previously I didn't have the skill or foresight to see how I could do this effectively. I do now

Back to why I'm so exited right now: I worked all night implementing the entity system in the LT Core, pulling out all my optimization cleverness to make that Newton step blazingly-fast. Rendering, too -- this is something that should be handled by the C entity. Lua tells the entity what it looks like, but doesn't perform the draw loop. That's really the key theme of how we're going to defeat FPLT: we're going to pull tight loops out of Lua, because high-level gameplay doesn't happen in them. High-level gameplay happens in response to events or, at worst, in response to a low-frequency timer saying 'yo, update the economy dude.' Essentially, the C core should be responsible for 'steady-state' computations, while Lua should just be notifying the core of differentials in that state -- a force was applied to an object, a new object was spawn, a station module went boom, a trader decided he wants to navigate here rather than there. How big of a difference does this approach make? It was scary running the perf test for the first time.

Right now I only have enough logic in the core to handle simple objects (I haven't yet implemented any kind of event system, but trust me, it's coming). So I chose our lovely, shapely asteroid friends...always a performance problem, even back in LTC++. Updating and drawing 1000 asteroids in LJ: ~10ms. Updating and drawing 1000 asteroids with Lua leveraging the C entity system: ~2ms (in other words, 0ms, because 2ms is approaching the minimal frametime it takes just to do the other stuff). To make the point more clear: 10000 asteroids in LJ: death. 10000 asteroids in LJ/C entities: ~2ms. You didn't read that incorrectly, although I thought I had when I first saw it...! Indeed, folks, we have found our milliseconds! They were hiding in those inner loops, just as I had suspected. 50000 asteroids in LJ: plzstopburningmycpu. 50000 asteroids in LTC++: plzstopburningmycpu. 50000 asteroids in LJ/C: ~30ms (profiled; almost all time spent in GPU rendering; the update and draw loops still contribute almost nothing). And, just so we're clear, that's way more asteroids than you've ever seen in LT. It's just stupid how fast it is Oh, and LTC++ was even scared to let the asteroids move, because that killed performance. 50000 asteroids with slight rotations and drifts in LJ/C: ~30ms (performing the basis rotation & re-orthogonalizing 50000 objects is, apparently, trivial for modern CPUs and well-optimized C!) I nearly cried when I saw that. It was unreal. In an instant, I went from feeling that I was 'too close to the edge of the cliff' with respect to LTLJ's perf, to feeling 'dear God, this is going to blow LTC++ out of the water ._.'

Sure, we still need physics (as in, collision checking & resolution). But guess what? That, too, will be very much doable with this approach. Thanks to the simple, minimal structure of the C entities, I'll be able to tightly integrate the physics engine. It'll be faster than LTC++ for sure. Speaking of which, I did a lot of thinking about collision checking. I have some very exciting ideas about which I'm stoked. Lots of cool math is involved, but the end result, I believe, will be really, really fast. I've already gotten technical enough, but capsule trees and a progressive, adaptive delaunay triangulation of space using entities as vertices...I think it will be close to optimal in terms of minimizing CPU cycles spent on Physics

This is really the kick I needed. I felt that FPLT could be overcome with the right balancing of LJ and C. I think I've proven it to myself at last. I need to implement a scheduling / event / notification / trigger system to really see how beautiful I can make this. I have a feeling that the beauty of gameplay code is going to profit from these developments almost as much as the performance!

---

Sorry, I know it was a ramble. I wish I could just do a Vulcan mind-meld with you guys so you could see all the possibilities I'm seeing right now...I've butchered it with my ever-erroding ability to use words. I'm just...really excited. This success has shown me that I was more worried than I knew about LJ being able to deal with entities robustly. Feels good to have that weight gone As usual, we must still be cautious. There are more than a few challenges left to solve. How big of a price will we have to pay when it comes to wrapping the C entities in LJ to add high-level-gameplay-related information? Frankly I don't think it will matter provided I can muster a robust event system in LT Core. But we'll still have to see. Once I have all entities, including those that require more complex rendering (a ship, for example, must render itself as well as thruster effects at a minimum, which requires much more information than an asteroid), then I'll feel even better about it. And once physics is in and working robustly....well..that'll just be the end of it. I'll be one happy gameplay camper.

---

I know this has gone on long enough, but I do want to mention that I was serious about doing some deep gameplay thinking this week. I'm working on some potentially-major changes to the economy in LT in an effort to make things better, simpler, more fun, and more AI-friendly, all at the same time. Don't worry, I'm not talking about cutting depth. I'm talking about re-framing the way I thought of some things before. I think it's fair to say that I fell into the trap of over-complicating things, ironically, in my quest to unify them. I hope to speak more about my ideas soon once I get some more time to flesh them out and, hopefully, implement some of them. FPLT sure has been a downer, but, as with every major struggle, I think it's going to pay off. It's given me the opportunity to come back to my old gameplay concepts with a 'fresh set of eyes'...eyes that are quite a bit more practical and concentrated on the 'big picture,' if I do say so myself!

---

The coming week will be all about getting the LT Core and LJ to fall in love and have gameplay babies. Yeah, that was a weird way of saying it. I didn't sleep last night, let's blame it on that But really, I'm so excited to see how this union comes together!

PS ~ I meant to post some screenshots, just so you guys can see what the demo looks like with some 'real' assets. Unfortunately I ran off without my dev machine since I was excited to write this log. I'll try to take some shots and upload them later (in a few hours, hopefully) when I head back to the office.

PPS ~ I finally learned how to use 'restrict' in C, which means I'm no longer so scared of using functions that take potentially-aliasable pointers. Yay for not being scared!

PPPS ~ I worked a bit on a grammar engine (as in, context-free grammars, etc; not grammar as in English grammar) this week. Originally the purpose was for auto-generating my tedious vector math C libraries. However, one thing led to the next and it became quite a powerful tool, one that I believe might play a role somewhere in the plethora of tech being used for LT. It's certainly more powerful than a context-free grammar engine. In particular, I believe it could enable very easy crafting of domain-specific pseudo-languages without having to actually do all that technical stuff of building a compiler. Practically-speaking, I'm envisioning it as a way to write high-level gameplay code that can be mapped to Lua (or C, if necessary) and be resistant to changes in the underlying API. Kind of like how a lot of games use config files to set constants, except this would be half-way between data and code. It could also be used for dialogue generation, and, with adaptation, possibly mesh generation. We'll see. So yeah, quite a few pieces of excitement this week!

PPPPS ~ For the TEDx event I did manage to get LT rendering on my new 4K, 10-bit HDR TV It's...pretty breathtaking. I had no idea how much difference the 10-bit HDR makes. It was easy to make LT compatible, since it already renders in HDR. Just had to do proper tonemapping & gamma correction and voila. Looking at the primary star in HDR mode definitely gives you the impression of "yes, that's a bright star." In general the extra contrast & brightness just makes things look incredible. Not to mention 4K making the detail look insane! Despite the underwhelming gameplay, people were generally amazed with the beauty and were very interested to learn about what they'll eventually be able to do inside of that beautiful universe!


Link to original: http://forums.ltheory.com/viewtopic.php?f=30&t=5889

Sunday, March 26, 2017

One of those weeks where loads of things were started, but none were finished. This week was dedicated almost exclusively to working on the entity system in the LT Core. This is, as we've found out from previous experience, where we'll either make or break LT's performance. Sadly, it'll be a rather bland log since I don't have all the solutions yet (getting these particular bits structured correctly is of utmost importance). I'll describe my work-in-progress nonetheless. On a side note, it's been a pretty rough week for me due to unfortunate events in my personal life, so forgive me if I'm not as enthusiastic as usual.

---

Entity Component System in C: In progress. I've been running through lots and lots of ideas on how to structure this bit, since, again, it's part of the overarching entity system that's going to make or break performance. I'm experimenting with several different architectures at the moment. Component-based entities are no doubt the right way to go in games (as opposed to big inheritance chains and/or multiple inheritance), but with them, the devil is in the details. Trade-offs are everywhere. Do we want to be able to add or remove components dynamically? Perhaps not -- but if not, can we still define new types of objects from script efficiently? If not -- can we define a 'generic enough' set of prefab entity types to support all that we want to do from Lua? Should we prefer structure-of-arrays format to save instruction cache misses, or array-of-structures format to save data cache misses (most people people assume the former when speaking of component-based entities; I'm not completely convinced.) Basically, there are a lot of choices to be made here that will have far-reaching consequences for both how gameplay is written in Lua as well as how performant everything is. Only time and experimentation is going to tell which set of choices are optimal

Notification/Event System: In progress. I spoke in the last log about my excitement for handling gameplay in a fundamentally different way using event-driven programming. I've started implementing the machinery for it in the core and have a few basic event types defined, although I can't test this system effectively until the entity component system is stable. Nonetheless, the more I get into the implementation and the more I shift my mindset to using events rather than 'polling,' the more comfortable I feel about LT's performance. I'm still absolutely convinced that we're going to see incredible results from this paradigm shift! When speaking about FPLT, I've often explained how the performance problems that arose in the old LT were, at their core, due to the sheer number of entities requiring simulation, resulting in a 'death by a thousand papercuts' situation. When the event system is complete, number of entities will no longer be the problem, because any entities that don't require computation on any given frame won't take up a single CPU cycle!

Scripting: I've had some revelations this week about my usage of Lua. In particular, I think I'm using it sub-optimally with respect to scripting the behavior of entities. I have a new idea that I've yet to get around to testing that involves binding an entity's local Lua state very tightly to the C Entity structure. When implemented, this will result in, essentially, me being able to turn off Lua's GC and manage memory explicitly, which will be a huge win. It will also solve the problem of how best to 'wrap' the core entity data with extra scripted data. However, until implemented, I won't know if the memory overhead of this approach will be a deal-breaker or not. I doubt it will be.

Lua...JIT..??: More performance tests this week yielded some startling news: the JIT part of LuaJIT is hardly helping at all. Turning it off entirely results in almost the same performance. This is both good and bad news. The bad news is that I'm obviously using LuaJIT sub-optimally (and it may have to do with the above point). The good news is that if such is the case, then there's more perf waiting to be squeezed out of the LJ parts of the code! I honestly just need to come to a better understanding of Lua and LJ, and perform more measurements to see what's up.

Physics Engine: In progress. Lots and lots of thinking time put into it this week. It's a toughie, but I'm going to nail it sooner than later Just wanted to give it a mention since a large amount of brainpower was spent on it.

LT @ NOEW: On Thursday I showed LT off in New Orleans at the 'New Orleans Entrepreneurship Week' event (check it). Since it was less amenable to interaction than the previous events, I wrote a new 'cinematic camera' that flies around a system, smoothly moving and panning while looking for interesting things to look at. Yes, my demo was, essentially, 'Limit Theory Screensaver' It was quite lovely and I got very positive feedback on the aesthetics (of course, that's not of particular consequence; we already know LT looks great...the being really fun part is more important now!) One guy even mentioned that he used to love playing Freelancer, and was excited because LT looked like a modern version of it. Naturally this made my day

---

In the coming week I'll just be continuing with all of this work and, if the fates align, perhaps even finishing some of it. The day where we leave FPLT in the dust is fast approaching



PS ~ To those who voiced their concern at the last log: my sleeping patterns were significantly more regular this week Don't worry, I did learn my lesson

---

Hey everyone! This is Talvieno, ready to serve up your non-technical summary!

Josh first mentioned that he's been working on how objects (entities) are designed in the code. In code, an "entity" could be a ship, or a station, or an asteroid, for instance, and could all use the same base code. Josh is working on designing this "entity code" so it will run as fast as possible, because with so many "entities" in the game, the quality of this code will really decide what kind of frame rate you see. He's not sure of the best way to do it yet, but he's making steady progress.

Something else Josh worked on: the "notification/event system" lets the program decide where it can safely (temporarily!) forget about things like ships to save even more frame rate (e.g. if a ship is idling, the game doesn't necessarily need to think about that ship all the time). LT runs fast, sure - but that's on his beefy machine. He wants it even faster.

As to LuaJIT (the scripting language that modders can use), Josh discovered that he was using it sub-optimally - kind of like if you were using a paintbrush with the wrong kind of paint. (It's a poor analogy, but hopefully you get the idea.) He's working on fixing that too, again for more performance improvements.

Finally, Josh took an "LT screensaver" to NOEW - the New Orleans Entrepreneurship Week event. This "LT screensaver" had a ship that flew around and looked at things in a system. People loved it, even though they couldn't actually play. Josh was quite pleased with this.


tl;dr: Josh is working on performance improvements so that almost anyone can run LT.


Link to original: http://forums.ltheory.com/viewtopic.php?f=30&t=5925

Monday, April 3, 2017

Happy April! Boy was it a long week. So much stuff happened it's hard to remember it all, but that's definitely better than the alternative. This week was all about two big pushes toward the completion of two big systems: the entity component system (ECS), and the physics engine. Neither was finished, but major progress was made on both fronts.

---

Entity Component System: Just to quickly recap, this is the part of the engine that powers all objects in the game (hence, the part of the engine most relevant to gameplay). It defines how the data that describes objects like ships, stations, colonies, asteroids, etc. is defined, managed, and altered over time in response to gameplay. From the point-of-view of most would-be LT modders (as well as Gameplay Josh), it's the most important part of the engine, perhaps followed by the UI system.

I spoke last week about the abundance of trade-offs inherent in designing a game engine's ECS. This week I continued to explore that abundance of hard choices by...implementing them. Describing all those choices in their technical detail would take too long and bore you all to tears. I started describing my current implementation in quite some detail in this log, stopped, and deleted it because I'm still too unsure about it With the lessons I learned from trying several different architectures, I started implementing a different one yesterday. But yesterday is...well, yesterday. Practical Josh, in general, does not feel confident about things being 'the answer' after only having had them around for a day.

But since I don't want to talk implementation details, I'll mentioned a new (to me) development paradigm that I've been using while playing with all of the options. Like many things I do, the idea sounds stupidly-obvious...but it almost felt like a revelation to me when I started doing it. I've been writing a good bit of gameplay pseudocode. Yep. Quelle breakthrough! But seriously:
It's so simple, incomplete, and really quite boring. Yet amazingly, it helps me in so many ways to think about how I should design the ECS from a practicality standpoint to do the stuff that I'm actually going to want to do. Even writing this silly code, I can get a feel for challenges that aren't obvious when designing from a blind, theoretical perspective. That code isn't real; it's not any real language, just my made-up pseudocode. But what we really want is to design things such that we can write gameplay code in Lua that rivals the simplicity of the code above.

Frankly, this has been equal parts difficult and rewarding. I'm bouncing back-and-forth between the highest-possible levels of gameplay coding (I mean, it's pseudocode, it's higher-level than the high-level gameplay code), and the lowest levels of engine architecture. It's mentally jarring to constantly bounce between those levels. But it's also rewarding to feel that these low-level choices are being made with very concrete goals in mind.

Physics: Great progress and several new structures/algorithms that I've never implemented before in LT, even back in the C++ engine. The LT Core picked up a lot of code. Octrees. AABB trees. K-d trees (unfinished). I have efficient code for building octrees & AABB trees from meshes or arbitrary volumes, as well as for raycasts. Full intersection tests will come soon.

On top of implementing a ton of physics-related data structures and algorithms, I did a lot of testing to determine which ones best solved the previous performance problems with the narrow-phase collision detection on large meshes. Not surprisingly, AABB trees are winning, at least with raycasts. I can do raycasts against a large, ultra-quality station mesh in something like 3 microseconds now. Although broadphase is not really a pressing issue and will come later, it was interesting to see that my tree construction code is fast enough to actually do the dumbest thing possible without killing FPS (that is, to reconstruct the whole acceleration structure every frame, inserting all objects in the system; this is incredibly stupid compared to real broadphases, which exploit temporal coherence to do dramatically less work).

Gratuitous screen from physics work:
And a few more if you want...
Lua & LuaJIT: As per my concerns last week, I did some investigation with respect to how to use LJ most efficiently. I used the profiling framework that I mentioned last month (and have expanded on greatly this week) to guide my investigation. I also started examining LJ bytecode dumps & trace information so that I can better understand how and where I'm losing cycles. The profiling framework has already proven itself to be invaluable!

I now have a better (but still expanding) understanding of writing efficient scripts, especially for gameplay logic. I implemented a system (in the LT Core) for script loading & running that allows better memory usage and performance than my previous monolithic architecture.

---

The coming week should be all about the entity component system. I've made great progress so far with trying a lot of implementations, and I'm getting ever-closer to a design that fits all of the requirements of LT. I'm putting this at max priority and giving myself full permission to do nothing but ECS work for the coming week. I want to start getting 'real' gameplay (not LTDemo gameplay) up; I'm ready for that pseudocode to become reality!

---

Hey everyone! Talvieno here, with another non-technical summary!

Josh has spent most of the past week or so working on the completion of two things - entities and physics. Just to recap, the entities are every object in the game - ships, stations, asteroids, planets, colonies - everything. The physics engine is how these objects interact physically with game space and each other - collisions, acceleration, motion and so on.

You may have noticed that he dumped a ton of "mining code" up there, and may have no idea what he's doing. Josh is basically doing his own version of "Rubber Duck Debugging". Rubber Duck Debugging is an old programmer method to figure out how to fix bugs: you talk to a little rubber duck sitting beside your computer and tell it what the problem is. In talking about the problem, you understand it better, and are often able to solve it much more easily than just from staring at the screen. Josh is writing pseudocode (fake code - "code" that's not actually code) to help him understand his problems, and finding it incredibly helpful.

The physics stuff is much more complicated, but in summary, Josh is working on collision detection. Basically, it splits up the game space into smaller, more manageable areas - splitting them up means the program has less space to check at any one time, which means a faster program. Josh includes a few visualizations of this in the form of shiny pictures.

As to the LuaJIT stuff - he's been experimenting with LuaJIT (the modding/scripting language) to try to find out how to get the most out of it - to squeeze out every last drop of efficiency for maximum framerate.

tl;dr: Josh has worked a lot in several areas the past week, mostly in solving complex problems. He's almost ready to start putting all the pieces of the puzzle together and make something you can actually play as LT.


Link to original: http://forums.ltheory.com/viewtopic.php?f=30&t=5964

Thursday, April 13, 2017

Rawr. I had almost finished this log when I accidentally closed it before having saved. Why hadn't I been aggressively saving after every few keystrokes like I usually do when I'm in vim? We will never know the answer Please forgive me if I'm more terse this second time around...

Last week I gave myself "full permission to do nothing but ECS work." And boy did I take that permission and run with it. Frustratingly, I'm not yet able to come back with a happy 'I GOT IT YAYYY!' like I wanted to.

---

Entity Component System: What can I say? I am finding that building this thing requires taking into account a massive number of factors at all times, making it slow and mentally-taxing. I showed last week how I've been writing gameplay pseudocode to help guide me as I architect this thing. In a continued spirit of practicality, I did more of that this week, but additionally went so far as to write down a detailed list of every entity that I anticipate being in LT 1.0, along with every 'component' / piece of data and functionality that I anticipate said entities to need. At least this moves some of the factors I must consider onto the screen and out of my brain.

Doing so was quite revealling: it reinforced in my mind that very few components require 'special treatment' for performance. Virtually everything that does require special treatment can be grouped under a broad interpretation of 'related to physics.' Thrusters are the odd-man-out, as they require special treatment for rendering purposes. Whether I can gerrymander said special treatment into the realm of Lua via some magic or another (I do have ideas) is still unknown. Basically, I'd like to be able to fence off all such special treatment and shrink it to the minimal viable subset. I'd then like to draw a big line in the sand between C and Lua that makes said special treatment 'not the concern' of the gameplay programmer (aka me + mod writers). This is a Really Hard™ problem that I've tried attacking from a million different angles and will continue to attack from a million more until one angle finally prevails.

The TL;DR of the technical details is: they haven't been solidified. The technique to which I alluded in my last log (as well as the long post I made in that thread, in which some details of that ECS architecture were revealed as the 'first level' of the mod API) has been put aside after further consideration as being over-engineered. If it comes down to it, I'll come back to that one. But my review of the actual needs of LT gameplay showed me that it's probably overkill to use Lua to give metacode-style definitions of ECS-related data to the engine, which then JIT compiles them to native code. I now believe a less 'heavy-handed' approach is feasible by drawing the aforementioned line in the sand carefully enough. I'm currently working on such an approach.

Mod Loader / Hooks / Environment: Believe it or not, one of the easiest ways for me to audition ECS ideas is to treat them like mods. All it really does is force me to write highly-compartmentalized logic and to think in terms of events and event hooks (which I need to be doing). The end result is that I can swap out different ECS ideas (at least on the Lua front) without breaking other things.

To get this all working, I of course had to write a basic mod loader (which is very, very easy in Lua), define some initial engine-level event hooks, and hammer down a format for mods. It was all quite rewarding. Mods, at least those that operate only on code (rather than adding or replacing new non-code assets) already work, because I'm implementing my ECS ideas that way, which makes prototyping very quick!

I developed several neat facilities for the mod system. Sandboxing was a big one (mods can't access arbitrary stuff from the outside world), and Lua again makes this very easy. It's easy to make sure that mods don't, for example, plug into the filesystem and start deleting your C drive. The way it works currently is that mods have to explicitly request access to the APIs that they need. Without an API request, mods have virtually no power (they can't even use basic (unsafe) Lua statements like require!) So if I want to use the function 'Draw.Rect' anywhere in my mod's code, I need to add 'Draw' to the list of APIs that is requests usage of (which is done by adding to a list in an 'info' table that every mod defines which contains name, description, version, dependencies, apis, and more). If I don't, the engine won't inject the API into my mod's environment, so my mod will simply cause a crash when it tries to use a function from the Draw API. This is very cool, because it allows the game tight control over what mods do, and can provide modders and users alike with detailed information. No hidden network access attempts It also makes it easy for me to reason about my own code dependencies while I develop. And for one more bonus, it makes debugging easier, because I can see exactly what 'mod' (remember that I'm using them right now to prototype things) caused a crash or even issued a print statement.

Physically-Based Rendering Stuff: Ok, so I admit that, after more than a week of waging ECS war, I needed a break yesterday evening. I gave the evening to graphics Josh I implemented the 'blind-you-with-all-the-things' style of HDR boom that seems so characteristic of high-end engines these days. It does look impressive. I'll turn it down a lot though...with it fully-enabled, the lighting in LT is literally blindingly-gorgeous

I also ported some of the PBR stuff from LTC++. I don't have the stamina since this is my second go at writing the log to talk details, so I'll just spew words this time. GGX environment map generation, better math this time (so arctangent, very sqrt, much spherical coordinates...wooww), faster as well; lighting just looks better with PBR yayy; combine it with above bloom tech and wowww beautiful reflections, eyes bleeding in all the right places. There you have it.

---

Guess what I'm (still) giving myself full permission to do in the coming week? Yes. ECS. Send your mental clarity vibes, I need them.

---

Hello again everyone! Talvieno here with a non-technical summary of this week's devlog.

Since our last update, Josh has been trying to figure out the ECS (Entity Component System) - the thing that makes ships, stations, asteroids, and more. It's a tough problem to work out. He wants modders to be able to access the bits they need (in Lua), while still keeping the slower portions in the faster C code (the LT core). Figuring out which bits go where (and how to make them all work together) is kind of tricky, but he's working on it.

Part of working on it is actually making "mods" to test it out. Josh can now load "mods" into the game, which is helping him figure out how to structure the ECS. As an added bonus, he's fixed the modloading process so that you can't make mods that do things they're not supposed to - like deleting everything on your hard drive or connecting to the Internet and downloading things. This is pretty much a necessity anyway, but he's come up with a clever way of doing it that he's proud of.

Finally - he made the game shinier. The graphics monkey has escaped, ladies and gentlemen - can we ever put it back into its cage now that it's released? (It was only an evening's worth of graphic monkey, though, so we should be fine. ) He implemented HDR bloom - a shiny glow you typically see from particularly bright light sources like sunlight. He's also ported some of his PBR code from the old LT version, so LT looks very pretty. (PBR is a lighting model that attempts to make materials model the flow of light the same way as you would see in the real world. This means more realistic-looking textures.)

tl;dr: Josh is working on the whole ship/station/asteroid problem and it's more difficult than he expected. He also added in some pretty graphics additions yesterday.


Link to original: http://forums.ltheory.com/viewtopic.php?f=30&t=5993

Sunday, April 23, 2017

It's not a perfect resolution of the grand war against entities, but this week has brought out the fiercest practicality that I can possibly muster, and for that I think we'll all be grateful. It is time to move on, dear friends, from this monstrous black hole of time and effort. Brighter prospects lie ahead.

---

Entity Component System: So...we're all tired of it by now, right? Me too. After many more hours drifted by this week with no sign of bringing me the end-all-be-all solution I've been wanting, I made what I consider to be a very practical compromise: I said "screw it." Since such crass language isn't allowed in my treasured notebooks, I instead wrote something marginally more profound: "design for the rule, not the exception." I'm sure someone has said that before. But now I'm saying it again. After enough neural burning on the problem, I decided it's rather absurd to continue this temporal bonfire (yes, LT development sometimes involves lighting time itself on fire) when the edge cases, not the majority ones, are holding back the design. I banged out an ECS that covers the 90% and gave myself permission to worry about thruster trails later. Here's my theatrical rendition of the past two weeks (Boss is, of course, practical Josh in disguise):

Boss : "Build me that ECS."
Josh : "Aye sir."

< 1 week passes. >

Boss : "Where's my ECS."
Josh : "Working on it."
Boss : "Get it done kid."

< 1 week passes. >

Boss : "Where's my ECS."
Josh : "I have a lot of good designs."
Boss : "Neat. So where's my ECS."
Josh : "Well sir, currently none of them can elegantly accommodate the fact that thrusters have a key part of their rendering behavior tied to one little piece of state that should, in earnest, live on the Lua side, as it's really not 'intrinsic' to anything. Sadly, in the case of thrusters, there are enough of them that the data martialling will become problematic and result in framerate loss. On the other hand, it's difficult to see where this state should live on the C side of the system."
Boss : "...that right? So...just...thrusters..."
Josh : "Well sir the general problem is likely to crop up in a few other pla--"
< Boss raises hand so as to silence Josh. >
Boss : "It's all so...incredible. So fascinating. But see here, look Josh, let me ask you just this one thing."
Josh : "...yes sir?"
Boss : "WHERE. IS. MY. <expletives censored> ENTITY COMPONENT SYSTEM?"
Josh : *Quickly hands over imperfect but functional system*

< Boss storms away angrily. Cornelius Evazan enters from stage left. >

Cornelius : "You just watch yourself."
Josh : "I'll be careful."
Cornelius : "You'll be dead!"

And there you have it. Kinematics and a few other pieces of physics-related information, along with graphical information, live in C. The rest (requiring no per-frame updating) lives in Lua. Components are Lua-side. So really, it's just an ES that lives in C...an Entity System without the components, since I decided these are, in the 90% of cases, lightweight enough to be handled by Lua without issue. And so it will be until something starts causing major issues.

RenderQueues: A nice advancement necessitated by this week's ECS work was a core rendering feature missing from all previous version of the LT Engine: the 'RenderQueue.' Truthfully, the RQ type is little more than a 'modern' implementation of something OpenGL has had since the dawn of mankind (GL called them 'Display Lists'). Since modern man has decided that OpenGL's display lists were a bad choice to put in a 3D API, we've deprecated them (along with 90% of the GL API). Hence why I implemented them as a first-class engine feature (driver support for display lists, unlike with many deprecated features, actually is a problem). The feature is little more than exactly what it sounds like: a queue of rendering commands; essentially a data structure that encodes a miniature rendering program. This allows rich flexibility to specify rendering operations in Lua by building a RenderQueue, but allows the operations to be executed at the full, blistering speed of a tight C loop. In fact, I've some notion that, since RQs are basically implemented as mini bytecode interpreters (i.e., a decode loop with a switch statement), they might outperform my previous implementations of certain rendering operations (thanks to the fact that all of the relevant GL code is sitting right inside that hot loop; external dispatches that aren't GL calls are nowhere to be seen). But all that aside, the point is that I needed a flexible and fast way to specify rendering operations from Lua, and I've now got it.

I previously had an engine construct called "Model" that handled this type of thing; it was basically a list of Material/Mesh pairs, telling the engine what meshes to render (e.g., for a ship), and what material to use for each mesh (a material was itself a combination of a Shader and various shader parameters). Upon starting to write my Model class into the LT Core, I saw the light, realizing that all of these various constructs that I used to have (Model, Material, ShaderState (a bunch of shader parameters)) could be unified by a structure that encapsulates exactly what they are, down to the core: just some rendering operations and various shufflings of state. Now I can think of a Material as being a RenderQueue that binds a shader and sets up some shader parameters specific to that material. I can think of a Model as being an RQ that calls a sub-RQ corresponding to a material, then draws a mesh, etc. RQs can be nested within one-another to encapsulate the exact kind of sharing that I used to think of in terms of all these disparate constructs that I no longer need! Yay unification. Rendering life is quite simpler now. To think the GL guys had this insight in 3000 B.C. or whenever it was that GL came about

I have some dangerous notions about how we might get cute with dynamically building RQs to achieve various optimizations. I'll try to steer my mind away for now.

Mods, Content, The Works: As you'll recall, my objective at the moment is to write the Lua-side LT code in 'mod' format, so as to make life easy on myself while simultaneously ensuring that I know how everything is going to come together with respect to actual mods. I've made a good many changes and simplifications to the format this week to help myself pump out that core gameplay code. I was honestly getting a little overwhelmed and turned around with my previous system of hooks and so forth. I've simplified the mod format to be much more like a 'data description,' which makes things cleaner and easier to reason about (without any sacrifice in flexibility). This is one thing I really love about Lua. It's a beautiful language for describing data, given that the one and only 'advanced' datatype is a table, and data is, after all, just tables inside tables inside tables. Throw in the ability to bind table fields to actual code/functions, and you've got all the power you need to build a piece of 'data' that tells the engine how to handle whatever it is that the data is defining.

I now just have a 'content' sub-table within my mod description table that contains any number of entries. Those entries represent...well...the content of the mod! Right now, they can be entity type definitions (ship, station, planet, etc.), event listeners (to add functionality in specific places), component definitions (define new components for use with the ECS), and miscellaneous type definitions (I'm still not sure what to call things like 'Item' and 'Corporation' and so forth...they're datatypes of course, but don't need to be handled as 'entities' by the ECS).

Stupid example of what I'm working on right now: Once all content is loaded from mod files (or base files, in this case), the engine pulls together all these bits of content and resolves everything to figure out exactly what every entity consists of, what events should be fired when, etc. Less manual labor than before. FYI the content creation code there is obviously placeholder

It has a ways to go to be sure; we can certainly pretty up the syntax later. Right now I'm interested in getting all of this stuff working. It honestly strains my brain a bit learning to think in this new manner of events and handlers rather than the old, linear update logic. But the straightforward 'dumbness' of the old update logic is exactly what killed performance, so it's time I learned to man up and think asynchronously.

---

In the coming week I have another LT showing at LSU on Friday, so my goal will be pretty straightforward: I'd like to see LTDemo running on the 'real' systems: using the C entity code, event-driven rather than monolithic update loops, etc. I'd honestly be happy to have that all working with the same level of functionality as before, because that would mean that I'd be in good shape to rapidly move forward with all systems sitting on a solid foundation. It'd be a major step. At present I'm not entirely sure how many man-hours of work that will entail, but I do intend to find out

---

Hey everyone! Talvieno here, with another week's edition of The Non-Technical LT Devlog.

This week, Josh focused hard on the whole ECS problem and got it solved - at least to the point that it's "good enough". He now has a single unified structure with which he can handle ships, stations, asteroids, planets, and other objects. This will make implementing content much faster. OldJosh is not pleased with this development and would rather he have a complete solution, but PracticalJosh - or NewJosh - has won out here. I think this is what everyone was hoping for.

RenderQueues: Exactly what it sounds like. Josh has implemented a unification of rendering commands that'll make it a lot easier to progress in the future. You can think of it like a to-do list for the engine's rendering component - the part that draws everything to the screen. Everything has its own little to-do list now, and can be modified/controlled via Lua - which means it can be modded.

Mods, Content, The Works: Essentially, this is Josh talking about how he's simplified the modding format to make it easier to use. There's not really much else to be said; I think that just about sums it up. At any rate, this next week he's going to try to focus on getting the LT code up and running for another showing at Louisiana State University.

tl;dr: Finished the entity component system, and set up the framework to make it much easier to add content, as he's getting to that point.



Link to original: http://forums.ltheory.com/viewtopic.php?f=30&t=6021

Friday, May 5, 2017

Been a longgg week. The leadup to the LTDemo showing at LSU last Friday involved the typical intense sprint, followed by the typical intense binge-sleep. Last Friday feels like only yesterday thanks to all that sleep (oh how the great Bank of Sleep always collects...with interest..)

Although it was, in fact, a great 'week' for progress, it was a total and complete failure with respect to my goals from last week. For that I'm thankful!

---

LTDemo Using 'Real' Systems: Nope. 0%. This was my goal from last week, and I achieved precisely 0% of it. With the showing approaching, I asked myself...would I rather show the same thing with architectural changes? Or would I rather show something better? After all this 'boring' ECS work this month, my self-control simply wasn't there. I worked on things that I wanted to work on. And I'm happy I did...it felt good!


Improvements in Ship Generator + 'Fighter' Specialized Algorithm: Things were getting confusing in the demo with every ship looking (roughly) similar, regardless of scale. To that end, I sketched out a basic 'fighter' generating algorithm, which is fairly rigid at the moment, but does produce things that look more like fighters than the other, 'generic' ship algorithm.

The fighter generator, although simple, is an important step in the generating algorithms: it's the first to use sub-generators for coherence. In particular, the fighter generator leverages a 'wing' generator for obvious reasons. As I move toward more coherent ship generation, I expect more and more of my generating will be factored out into 'component' sub-generators. Cargo bay generator. Engine generator. Cockpit generator. Solar panel array generator (station). This has always been the plan, it's just the first time I'm actually working on it (and the first time I've worked on ship algorithms in...a long time)!


Beam Weapons: I wanted capital ships to be fearsome. So I banged out some beam weapon code and a new shader (assuming, correctly, that I could do better than the old one). It's been quite some time since beam weapons were working in LT (did you...notice that...? Mining lasers, sure...but when's the last time you saw a beam weapon in the update videos? ) It turned out nicely. The initial balancing was...quite poor. It was not a fun time to be a fighter pilot, despite your fancy new fighter ship design (Part of that was that I forgot to put a range cap on the beams...yikes... )


Factions & NPC AI: Factions were loosely-implemented in the LTDemo already, but I wanted a less-loose implementation so that I could expand the gameplay. Factions in the demo now have unique ship designs, their own credit account for spawning new ships, a tech level and a spawn zone. NPC AI now (more) intelligently selects enemy targets based on distance, firepower, etc, and will of course only attack enemy faction assets. Combat AI was split up into distinct maneuvering, target selection, and firing sections, and generally improved.

A note on how this relates to the 'real' LT: you may notice that some of these things, in particular, some of the faction elements, are not in line with how things are supposed to work in Limit Theory. In, shall we say, 'real LT', factions don't have spawn zones. Factions can't just spawn assets by spending credits. There's no such thing as a 'tech level,' only explicit technologies. Although factions do have unique designs, they are supposed to be influenced by factional style traits, which don't yet exist. And, perhaps most importantly, factions in LT aren't first-class things in the way that they are in the demo. They're abstractions created by NPC or human players to manage projects, assets, etc. Behind every faction in LT is someone leading it; the faction is not an autonomous entity. All of these things I have simplified solely for the sake of implementation into LTDemo, but that does not mean that the mechanics for 'real' LT have changed

These additions to factions, by the way, have put me very, very close to having some awesome new gameplay available in the demo, involving growing your own faction by defending successfully against enemy attacks, purchasing new AI ships from your factional credit pool, upgrading weapon technology, etc.


LTDemo -> Real LT, or <void> -> Real LT?: The continued development of the demo begs the question: do I try to build up functionality in the demo and, over time, smoothly transition that functionality to match the gameplay mechanics of Limit Theory? Or am I better off regardling and building Limit Theory separately (by separate I mean...separate Lua scripts; of course they will always share the same engine and utilities), so as to focus on building things as they are intended to be from the beginning? For now I have no great answer, as there are still a great deal of variables in flux...which brings me to my next point...


'Startling' LuaJIT Investigations: For technical reasons, I haven't been able to run LuaJIT in 'verbose' debug output mode until this week. Basically to do so I needed my engine to be built as a shared library (.so / .dll) rather than an executable, which required some build infrastructure tweaks -- more on that in the next segment.

Upon finally having this option available to me, I was able to acquire detailed information about the JIT part of LuaJIT. I won't explain it all, because it's highly-technical. But the short story is: LJ is currently helping me very little compared to what it could be doing. I'm basically killing much of the 'JIT' ability of LJ due to some of the things I do in Lua, meaning that most of my Lua code is being run through a standard interpreter. That's bad. And slow. It also explains my observation a few weeks ago that turning off the JIT engine didn't impact performance very much.

This is actually a welcome development in my ongoing quest to harness LuaJIT. Clearly, it is not nearly as 'flexible' as I had thought. There are certain very basic things that cause LJ to abort potential optimizations. Now that I'm aware, I can do a much better job of avoiding them. I've already significantly pumped up performance of the demo by working to eliminate JIT aborts. But much work remains if I'm to get full utilization of LJ. This is good news. It also means that I will need to re-evaluate the C / Lua split. Drawing that line, as I explained before, is crucial. But if I've been operating under false assumptions about LJ performance (and I have), this might mean that LJ can handle more of the work than I anticipated.

In the extreme case, it could mean I don't need any C ECS or any gameplay-related code living in C (highly unlikely). In the worst case it means everything stays the same Either way, the discovery necessitates that I move 'maximize LJ utilization' in front of 'build and integrate C systems for efficient gameplay code,' as I won't be making well-informed choices concerning which systems to build until I've mastered LJ -- or at least significantly reduced the interpreter fallbacks, which are currently happening way too frequently.


New Build System: Minor news. Born out of frustration with how difficult it is to maintain my builds accross all projects & platforms, I spent a few hours writing a Lua script that works as a build system that's custom-tailored to my needs. No more Makefiles (yes, I wrote Makefiles previously, because ), no more .bat files calling the MSVC command-line tools (yes, again, ). One Lua script that does it all directly. And yes, I know about CMake (LT used to use it), SCons, premake, etc. But it didn't take long at all to whip out something that is optimal for my needs

Other than convenience, the big news here is that I can easily create new sub-libraries for code that doesn't belong in the LT Core, but doesn't belong in LJ either. Gameplay systems match that description perfectly!


Upgraded Website: This one was actually all Talvieno, to whom I gave the task of 'modernizing' the LT homepage a bit by replacing the pictures with more recent ones. He did that and much more. As usual, Nathan has gone above and beyond

Check out the new look for our homepage! (Refresh if you don't see the changes. We're still working on caching issues )
((Addendum by Talvieno: You may need to ctrl+f5 or shift+f5 (depending on the browser) if you don't see the changes/the pages look odd.))

---

Obviously it was quite a scattered week. Big, but scattered. I'd actually like to do it again. By that, I mean I'd like another week of this. I want to improve LTDemo a bit more; write some of that gameplay code that I've been wanting to write in spite of not having it running on the 'real' C systems. I want to continue investigating LJ, because those investigations may reveal that I don't actually need to run as many things on the C systems as I thought. And maybe some new generating algorithms! That was fun work.

In short, my brain needs a break from all that big, architectural work that happened over the past few weeks. It was draining. I had some fun this week with LT. I'd like to have some more fun in the coming week Consider it a mini-vacation. Except that it's equally (if not more, due to unexpected discoveries...) productive as my non-vacation work



PS ~ Oh yeah, and I brought back the color grading post-fx. Forgot to mention that. Everything's a bit prettier now (plus systems have their unique 'mood' since the grading table is generated procedurally for each system).

---

Hello again! Talvieno here, welcoming you to this week's edition of "What the heck did Josh just say".

First off, I'm going to talk about the fact that Josh is mentioning working on LTDemo. Wut? LTDemo? Not LT 1.0? Josh is off the beaten path again, isn't he? ...actually, not at all. See, in programming, putting work into a demo program means that you're also essentially putting that work into everything else that needs it: transferring working code between programs is as easy as copy+paste, and working on just what you're working at the time on lets you prune out complications and distractions. In other words: Josh may be working on the "LT Demo", but the work he puts into it ultimately goes into (and gets us closer to) LT 1.0.

In the same way, Josh is working on the fighter generator first so that he can use this code elsewhere; it's a small "stepping stone" toward making beautiful capital ships. I'd like to point out, though, that Josh hasn't put much work into it so far by his own admission; he's not spending silly amounts of time on graphical elements. In fact, he's been working on gameplay elements as well. LT now has working factions, and the AI has greatly improved.

Going forward, he may now merge everything into the new and improved "LT 1.0"... or he may continue doing what he has been. He's at the point where he could go either way. More on that in future updates.

Regarding LuaJIT: Essentially, what all of this mumbo jumbo means is that Josh isn't using LuaJIT to its true potential, and he plans on rectifying that to squeeze as much performance out of it as possible - which means a more interactive, immersive, living world for us.

tl;dr: Josh has been working on performance improvements, and re-implementing gameplay. We're getting back to where we were before - but at an acceptable frame rate this time.



Link to original: http://forums.ltheory.com/viewtopic.php?f=30&t=6050

---

Although there haven't been that many major visible changes, I figured I'd make a gallery for this week's log anyway. Main new things to notice are the per-system color grading, fighter ships, and beam weapons.

I know the fighters don't look great, but they're clearly fighters and they do have variation. That was a 30-minute algorithm (hey remember the old 7-minute station algorithm? Heh, guess I'm losing my touch.. ). Lots more fun to be had there

Limit Theory Devlog Gallery ~ May 5, 2017

Some favorites:

Tuesday, May 16, 2017

Quite a flurry of an eleven days (for those of you who think you've spotted a trend in the spacing between dev logs, don't get too comfortable ). Were I to remain as mysterious and secretive as my inner Josh is telling me to be, this would be a short and unsatisfying log indeed. So I guess I'll just...not be like that.

I've been stupendously crushed under work this past week-and-a-half. For better and worse, it's not the usual kind of work. Better because the long-term ramifications for LT are awesome. Worse because this dev log is not going to be shiny.

But to tell the tale of this week-and-a-half, I must first set the stage. Prepare to be taken into the life of Josh outside LT (which, as we know, is quite minimal...but does consist of at least one interesting activity).

---

I may have mentioned this once or twice on IRC, but I'm not sure I've ever actually mentioned it on the forums. Over the past year, I've been part of a program at LSU (the local uni) that holds math classes for high school students on Sundays. It goes pretty far beyond what students normally learn in high school; it aims to prepare them for major mathematics competitions like the Harvard-MIT Math Tournament, Berkeley Math Tournament, etc. These are (obviously) kids who are 'gifted' and seeking challenge beyond what high school affords them. The program is called 'MathCircle.'

This year, the director, a graduate mathematics student at LSU, decided he wanted to explore adding a programming component...after all, Math & CS are match made in heaven. As it would turn out (through a series of interesting coincidences, which will be the theme of this whole post), I ended up volunteering to be the teacher for this. Every Sunday during the school year (or most Sundays), I prepared a lesson to teach high school kids basic programming in Python, with the intention of getting them to a sufficient level to compete in the USACO programming competitions, of which there are 4 during the year.

Anyway, I must stop myself from getting carried away with this story. TL;DR: we competed in programming competitions. I also competed so I could judge for myself how hard the different 'divisions' were. I made it to the top division (Platinum) by the hair on my chin. The Gold (second-highest division) questions nearly tore me apart. I expected most students to remain in Bronze (beginner's devision), and perhaps a few of the best to move up to Silver. You can imagine my surprise when one of these high school students with very little prior programming experience made it to, and subsequently did quite well in the same division that had ME breaking a very real sweat. Not that I'm the best competition coder, but I do at least program for a living so, you know. After talking to this student a bit more and coaching him individually in preparation for Gold-level questions, it was clear that his brain was built for programming. He gets it.

Which is why he'll be interning for me over the summer, hopefully starting on Monday.

---

I've had my office at the Louisiana Tech Park for 2.5 years now, so I've gotten to know many of the residents quite well (especially the other game programmers).

It was a very bizarre coincidence indeed when, last week, the most talented game programmer I know here walked into my office, curious if I had ever considered letting someone else work with me on LT. I told him I had considered it but that location and money generally made such considerations very short-lived. As it happens, his location is the same as mine. As it also happens, he, like me, is more interested in hard problems than money.

Which is why he's currently sitting five feet away from me in the office, working on LT engine code.

---

So, what have I been doing over the past 11 days? Very little, with regards to gameplay features and the like. But I must say, Manager Josh has done more in these 11 days than in the rest of the 4.5 years put together.

Hiring Nathan as CM was (obviously) a fantastic choice that continues to pay dividends. I guess I finally got the message: I'm not alone in this world, and, occassionally, it may happen that my path crosses with someone capable of making my life easier (read: getting LT out faster and with less ibuprofen consumption). I've always said before that I couldn't afford to get help. And you guys kept telling me that there's help out there that would come at a relatively cheap monetary price as a consequence of getting to be part of the beautiful project that is Limit Theory.

I get it now

For the first time, I'm excited about the idea of having some help on the code side. I've got an intern with incredible potential who's interested in AI -- and I've got a thousand interesting AI problems. I've got an engine/graphics programmer with very deep knowledge of C & C++, architecture, graphics, and the like -- and I've got a thousand interesting engine/architecture problems.

Now, nothing is certain at this point. I'm 95% sure on the intern front. I'm less sure about the mysterious other developer (btw, these guys will get names when we achieve certainty). He's working with me right now as a sort of 'trial run' while he evaluates offers from other places (that could no doubt give him more than an its-not-about-the-money-right?-sized paycheck). In the mean time, I must ensnare his mind with LT, and show him that there is no other project of such beauty, challenge, reward. Manager Josh has much work to do. Pragmatic Josh is standing back nodding his head and smiling. Graphics Josh is antsy as always, being locked in his cage. Gameplay Josh is hoping he catches a break by being able to delegate away some of the hard, low-level work. Human Josh is quite hungry and is hoping to finish this log right about now so he can eat the warm sandwich sitting beside him. Meta-Analyst Josh is slightly concerned that Josh finds it reasonable to describe himself as a multitude of different persons, and in the third person, no less

I'm sure this log raises loads of questions that I've not answered...but I've got a lot of work to do right now, starting with inhaling this sandwich

The future of LT is bright.




Link to original: http://forums.ltheory.com/viewtopic.php?f=30&t=6081

Thursday, May 25, 2017

It'll go down in history as the first week that Limit Theory had more than one programmer working on it! And I'm coming to learn just what a fine thing that is. It's remarkable how different it feels to have help (I mean, on the programming side), especially help that's sitting a few feet away. So without further delay, let me introduce the new names that will appear on the LT credits under 'programming' (well, no last names yet because I don't want you guys stalking them...not that you would ever do such a thing... *glances at startlingly-high unread facebook message count from unknown people + neverending friend request list* )

---

Adam - In the last log I mentioned the fantastic coincidence of a highly-talented game programmer walking into my office and subtely inquiring about a job. His name is Adam, and you will all come to know him as the other guy on the team who gets excited thinking about spatial partitioning schemes, memory layouts of component systems, and C without the ++. Last week we did a 'trial run.' Monday through Friday of last week, he came in and we tested the waters with respect to whether he would enjoy LT work, whether I would enjoy having someone else in my office / be able to give him things to do, etc. Come Friday it was basically a no-brainer. We both knew how the week had gone. At the end of Friday, I made him an offer and he accepted. Welcome, Adam.

Sean - I also mentioned a potential intern with such an acute talent for learning quickly that it more than offsets his lack of years at the grindstone. He came in and toured the office last Wednesday. We had a chat about some of the things he could work on here -- especially AI-related. I asked him, to get a sense of how much I needed to "sell him" on the internship, what his thoughts were on a scale from 1 to sold. He immediately responded with "sold." Welcome, Sean.

So, last week we were at (unknown%, ~95%), respectively. This week we are at (100%, 100%). They both officially started on Monday. Hard to believe it's only Thursday.

We'll work on getting Adam & Sean on the forums, since I know you guys would enjoy flooding them with enthusiasm and the 'occasional' off-topic meme...

---

Now, let's see how this week is going thus far!

Project Organization - It's worth mentioning, since LT has thus far existed, at various points in time, as: stacks of notebooks brimming with frantically-scribbled ideas, code files spanning dropbox, google drive, 5ish different dev machines (throughout the years, that is), and, at one point, that silly USB-drive chain that I wore around my neck, and of course, countless various patterns of neural firings that have been etched into my brain. Oh, and countless Trello Boards. And sticky notes. And...you get the idea. Not exactly centralized. A team necessitated a change, so Adam and I worked hard on setting up some organization. Version control that isn't Dropbox's version history feature (hey, don't hate, it's an amazing feature), project management tools that aren't just lists of cards. We actually have milestones with dates, task allocation, analytics...you know, all that stuff that fancy folk use. Now, boring stuff aside....

Entity Component System - HA!!! And you thought you had heard the last of it. Come now, you should know better But seriously, we had to come back to this. Once Adam got LT building and running on his machine, I was rather horrified at the performance. Granted, I haven't been overly-focused on optimization. Granted, he's not running the most powerful GPU ever (760M IIRC). Still, it was pretty clear that we needed to rein in the Lua and bring more over to C, as I said would happen if perf declined.

To that end, we've been collaborating on an industrial-strength ECS for which we've set a pretty harsh architectural-lock deadline. The great news is that, having someone else with the skill to design such a system, the task is asymptotically faster. I'm starting to realize just how much time was lost to my own self-doubt and falling into the same loop of bouncing between A, B, C, and back again. I suppose that's how it goes when you have one brain anchored to nothing. But with Adam and I bouncing the design and implementation back and forth, we've already got what we both feel is a really good architecture. As of tomorrow we should have enough done to begin verifying that it's as good as it looks on the whiteboard. It should provide very significant gains, and negligible (or no) loss in moddability.

Universe Creation, Planet & Colony Dynamics, Research, High-Level AI: Yeah, ok, that's kind of unfair. I feel like you guys, from here on out, are probably going to be mostly interested in Sean's work. Maybe I'll make him write the logs Since Sean is still comparatively fresh with Lua, C (and programming), it seemed best to allocate him more towards using that brain for pure thought. Of course, he's not just thinking about gameplay, he's also learning Lua at the same time and getting a feel for what the high-level will look like in Lua.

I've taken several hours over the course of the week to explain some of the high-level mechanics to Sean as he has encountered the need to understand them. At this point, we've been over a lot of territory: how the high-level AI works and reasons, projects, details of the economy and the market/mission board, AI traits, colonies and their spawning of AI + culture vectors, universe generation, research mechanics, and so on. At this point I'd say he's got a pretty good handle on the big picture of LT, and his task is basically: explore it, detail it, make Lua out of it, and generally continue filling in the framework that I laid out for the LT large-scale dynamics. Ultimately I hope to have him doing large-scale, long-time-scale sims on generated universes and doing data-driven balancing and tuning to ensure that we have a dynamic-yet-stable (as in, doesn't blow up after 10 hours of game time) simulation going on in the game that affords maximal fun for players.

As the so-called 'root'/anchor of the game simulation, colonies were a natural place to start, and that's exactly where Sean started his exploration. This exploration naturally led him to the exploration of...basically everything, so it's probably easier to flat-out list the things he's been working on:

• The nature of the colony culture vector and its influence beyond just the spawning of AI
• Planet types and the way in which different types of planets modify the economic processes executed by the colony
• Star types and the way in which different types of stars influence planet generation
• Distributions and evolutions of stellar masses in a galaxy and the way in which physical formulae dictate the evolution of a galaxy / the distribution of stars (it brought me joy to see him exploring some of these things with real-time graphs he created )
• Bringing it all back to mass distributions, and going forward from there to temperature, color, planet distributions, etc.
• Colony-to-Colony interaction mechanics (something that's never been explored before!), including cultural diffusion
• The exact formulas and mechanics that dictate the creation, growth, and death of colonies
• AI spawning mechanics, especially with respect to how the birth and lifetime of an AI player affects the colony on which it was born and vice-versa
• The influence of colonies on planets (! not the other way around! I have never considered this before)
• Probably a lot of other things....

Yeah. As you can see, he has quickly learned just how much fun stuff there is to consider when it comes to the high-level gameplay. Much of these mechanics are for the purpose of generating the universe, as they're obviously extremely-high-level and some of them would have relatively low impact over a short span of gametime. But together, their consideration forms the basis for a coherent procedural universe, a concept that I've stressed to him on several occassions. Limit Theory is not just an endless universe of random stuff. And judging by the way Sean is handling it, the high-level, long-time-span sim is likely to turn out even more beautiful and interesting than I imagined Of course, make no mistake, at every turn we'll make sure that the work he's doing results in a fun universe, another point that I've stressed -- realism is not necessarily fun...as demonstrated by...well...it's been demonstrated... :V

Physics: As I mentioned, Adam and I are essentially trading off the baton on ECS work so as to get the best of both of our brains. His task, while I have the baton, is currently the implementation of the fastest-possible structures for physics. I've talked about my work on various spatial hierarchies for physics pretty recently, and I'm effectively giving him the task of seeing that work through to completion. Interesting side-note: Adam was studying physics in grad school when he decided that game programming was more interesting

Asset Compression: I'd be remiss if I didn't mention the work that Adam pulled off last week in just two days. I tasked him with implementing compression in the engine for binary data, to be used primarily for asset compression (notably, to make sure that procedural stuff like nebulae, planet surfaces, ship/station meshes, etc. don't eat up your whole hard drive when I cache them to disk). I was very impressed when, instead of just integrating zlib like I suggested in passing, he took the time to run actual performance benchmarks on several algorithms, which, in the end, landed us at LZ4. I was totally unaware of LZ4 and how incredibly fast it is. Speed is of the essence here, since we're going to be compressing and decompressing at run-time as we trade certain things off to a disk cache when the player leaves the system. LZ4 blew zlib out of the water with respect to speed, and still maintained quite an acceptable compression ratio. It's now integrated in the engine. As I told Adam, "you've just saved countless man-hours" -- had I done it myself, I would have dropped zlib in and never thought twice about it. And the engine would be a good bit slower at loading / saving. This man respects your time as a player! Only one of several impressive feature completions that made it easy for me to see the value in bringing him on!

Everything: As for me, other than continued ECS work as noted above, the week has consisted of not much less than everything. With two others working on dedicated tasks, I've been somewhat freed up to nail a lot of residuals that have lingered in the to-do list for far too long. Another list seems in order:

• Added 'OpenGL debug mode' to the engine that automatically checks for and reports non-critical GL errors after every call. These can significantly hurt perf on older drivers or less-powerful cards; we saw a nice boost on Adam's machine when I eliminated all of them
• Significant cleaning of engine API so that it's easy and fast for us all to use (not just me), and...
• Automatic LuaJIT bindings to the engine generated via a script (so that the bindings are never out-of-date, which has caused several headaches thus far). Combined with the previous item, these two ensure clean and complete access to the engine from Lua
• Stack-based render state
• GPU hinting for machines with both integrated and dedicated cards. Adam's machine was running the engine with integrated graphics by default; turns out applications can do certain things to 'hint' to NV / AMD drivers that they need high-perf. The engine now does so, and automatically runs on the dedicated card for machines that can choose.
• Minimizing the build process accross Linux & Windows. Since Adam is running Windows and I usually work in Linux, the result is that we get to constantly test LT on both. It's also a headache to keep the build process simple, especially when we want to add test applications. I'm close to having all such headaches totally wiped out (in a simpler way than CMake / Scons / name-your-overly-complex-build-system)
• Planning tasks and generally learning how to make us the most effective team we can be + getting everybody ramped-up logistically
• Everything else

---

Well, I'm exhausted just from typing that; I'm sure you all must be exhausted from reading! (Talvieno: I'm so sorry. The TL;DR will no doubt be a pain... ) But I wouldn't want to miss the chance to convey to you all just how fast we're ramping up the development.

I've been saying it for 4.5 years and I'll keep saying it for whatever (comparatively) little time we have left: LT is coming. You can't stop the train.

See you all soon!



Link to original: http://forums.ltheory.com/viewtopic.php?f=30&t=6108

Saturday, June 3, 2017

Really exciting week; the whole team was on-point this week if I do say so myself! Quite some major developments With any luck I won't spent so much time on the log this week, since there's valuable work to be done and I'm under a lot of time pressure here with this whole concept of 'getting certain things done by a certain time' -- a strange and foreign concept that Adam has convinced me has substantial value to it :V

---

Josh: ECSv2

In the last log, I talked about how we had to go back to the drawing board with our entity component system, since, unfortunately, things were starting to break down in LuaJIT, necessitating pulling work over to C. Most of my time has been spent this week nailing this problem. On Thursday, a turning point came when I had finished enough of the system to start running initial tests in LJ. For me, it was one of the most exciting moments in months when I saw the results. Basically, our ECS design works really, really well AND interfaces with LuaJIT REALLY well. The C itself runs faster than I expected. But the LJ <-> C interface? Multiple orders of magnitude faster than I thought possible. It blew my expectations clear out of the water. So, let me lay down some fun facts about where we are with respect to the system that will support all of the entity data in LT. First, the C-side:

• Logic that runs on the C-side is stupidly fast...
• ..it is also easy to thread. I'm already threading the C-side logic. Yes, LT is actually going to use your cores
• In C we can perform non-trivial, per-frame logic on >100K entities @ 60 FPS with a good deal of headroom on a modern CPU (using threading).
• Depending on how heavy the logic is, we can perform it on the order of 10ns to 100ns per entity. Kinematics updates, for example, are currently clocking in at ~30ns (3ms for 100K entities).
• All of this is based on using non-hard-coded entity types. That means new entity types added by mods enjoy the exact same performance characteristics.
• New component and entity types can be modified and added dynamically. The ECS was designed from the ground-up to ensure that we were not hard-coding anything that we didn't absolutely have to.
• Creating and destroying entities is virtually instantaneous due to the memory pool design I built for entity data storage. 10K missiles created in one frame is boring work for ECSv2.

Now, the part that has always been more problematic. How about LuaJIT?

• Logic that runs in LJ is 'on the order of' C speed (!!!!).
• LJ logic can update tens of thousands of objects per frame without issue. This will improve when LJ gains threading.
• ..oh yeah, since all data is stored and managed C-side, we can thread the Lua as well
• Lua code accesses and modifies entity data natively, with zero overhead, exactly like C code would. In other words, myShip.Integrity.health = myShip.Integrity.health - damage. We can write that in Lua, and the instructions emitted by LJ are 'basically' the same as what Visual Studio / GCC / Clang would emit.
• Speaking of emitting, all Lua code is currently running as JITed assembly in my tests. The new ECS <-> Lua interface was designed to ensure that LJ is able to JIT everything, and never have to fall back to the interpreter (which is why we can achieve C-like speed.)

I realize that's a lot of technical stuff to digest. Let me put it simply: FPLT is solved. Not worked-around. Solved. Those test figures I gave above? They represent substantially more work that will typically be needed in LT. Remember when I talked a long time ago about having performance 'headroom'? Well we've now got plenty of it. All it took was...a better understanding of LuaJIT, a better ECS design, an automatic ECS->LJ binding mechanism, multithreading, and clever, cache-aware memory pools. Yep. Turns out that's 'all there was' to solving FPLT :V

A huge weight has been lifted from my shoulders. I can now say with great confidence that C + LJ is our solution. I can now write gameplay code with confidence that it won't be thrown away when I figure out that it performs poorly. Really, this is just a tremendous leap forward Adam and I are absolutely stoked that our design hit the nail on the head.

Adam: BSP Trees

Adam has spent the week becoming the resident master of BSP trees and narrow-phase physics. He's been really, really thorough about implementing them in such a way as to avoid common pitfalls (BSPs are well-known for their robustness issues -- it's easy to get them to 'mostly work,' another thing entirely to get them to 'really work'). At this point, we've got fast and accurate raycasts on complex geometry thanks to his BSPs. But it's not over yet: with robustness locked-in and hammered-out, Adam's going to start working on optimizing the BSP construction algorithm to ensure that we get the best performance possible for collision checks against annoyingly-complex objects like stations.

I'm really happy that someone on the team is taking the time to understand the nuances of BSPs (and, in general, robustness in bounding volume hierarchies). I'm certain it's going to pay dividends. After all, with the ECS now consuming so little of our frame time, the next-biggest performance hog is likely to be physics, and Adam's on track to make sure we cut it down to the same blazing efficiency that we managed to pull out of the ECS.

It's also worth mentioning that Adam is really good at building small testbeds that visualize results and show us when things are off, even if only by just a hair. This is exactly the kind of thing that I wouldn't do if I thought my implementation was 'mostly working,' and exactly the reason I was impressed with his rigor from week 1. Thanks to these visualizations he's caught a lot of edge-cases and robustness issues that would likely have gone unnoticed otherwise.

Sean: Colony Dynamics and Working Market Simulations

Sean continues to delve into all of the high-level simulation workings of LT, still focusing primarily on colonies. This week, however, I sat down with him for quite some time to explain the way the 'real' markets work in LT (bids/asks), the way it relates to AI, how colonies play into that, etc. I implemented a few basic objects for him in Lua to help with his simulation, the most important of which was a quick implementation of a dynamic market that keeps track of bids / asks, resolves them in the correct order, etc.

From there, Sean has been able to set up his first working economy simulations! At the moment, he's looking at a single, on-colony marketplace and how the colony's supply & demand drives the local market, as well as how external factors influence it. He's also experimenting with the pricing/valuation algorithms and how the AI's ability to changes its own idea of how much any given item is worth affects the global simulation. It may be a small-scale simulation, but to me this is a major step: from here, all he needs to do is continue adding elements to the simulation, studying how they affect it, and adjusting his course appropriately. I'm eager to see this work take off!

---

As you can see, the LT train is showing no signs of slowing down anytime soon! I'm pretty amazed at how much we've burned through in only a few weeks. But the burning must not let up! In the coming week, I'll be attempting to close out ECSv2, with a focus on continued performance studies in LJ. With luck I'll be starting on a new milestone near the end of the week. Adam is looking to have sphere and bounding-box intersection tests working on our BSPs sometime in the coming week, and will hopefully also begin work on optimizing BSP construction. As for Sean, I'm looking to see the market simulation blossom as more and more of the 'real LT' mechanics are factored in and added to the equation!

See you all next week



Link to original: http://forums.ltheory.com/viewtopic.php?f=30&t=6163

Monday, June 12, 2017

I'm afraid that I'm going to be keeping this log briefer than any of us would like (no, I mean *actually* brief this time ). If you're reading this, you no doubt have noticed the new forums. Talvieno and I worked over the past week to upgrade the forum software, which turned out to be hundreds of times more painful than expected. In the end, the greater part of my time was dedicated to forum-related things rather than LT code, so there's kind of a double-whammy situation going on here: on the one hand, I don't have much to talk about, on the other (same?) hand, I really don't want to spend more time not working on LT code

---

Josh: Forums Fires and So Forth

Already said it, pretty much. phpBB3.0 -> 3.2. Pain. Database migration. Pain. Old style did not support the new version (or vice-versa). Had to find new style that looked decent and close enough to our old one that it wouldn't shock and terrify everyone too badly (trust me, prosilver, the default theme, would shock and terrify you all; no one would touch the LT forums ever again ). Pain, time. Time. Lots of time.

LT-side, minimal stuff going on. ECS wrap-up delayed, but still feel good about it. Worked quite a bit at getting a very simple graphical API for Sean to use for simulations. Not much fun watching console output all day for simulations! Especially if you want to get into the low-level AI like dogfighting...and he does.

Adam: BSP Perfection

Adam has officially perfected the robustness of our BSP trees. Like, to an insane degree. Floating-point precision has been told to shove off. Millions of rays being cast against complex shapes, perfect intersections being returned and never any false positives or negatives regardless of the input, etc. It's awesome. He's moving on to optimization now. This is the part where our narrow-phase physics tests become blindingly-fast!! I'm excited. If and when he's successful, we'll be looking at having more milliseconds than we know what to do with

Sean: Continued Colony Work + Low-Level AI

Sean and I spent some time this week talking about how the 'low-level' AI systems work (i.e., the AI that executes specific actions that are directed by the high-level AI). He's very eager to get working on some of this stuff like combat maneuvering, obstacle avoidance, pathfinding, etc. Unfortunately, with things rather in-flux on the Lua side due to the migration to the new ECS, we don't have a great way to do 'simple' visualization tools that aren't in C (like the ones Adam is using for BSPs). That's not at all ideal, since Sean is working in Lua and we want to keep it that way. Thankfully, as of a few hours prior to writing this log, I've finished building a very simple visualization tool that will let him write visual simulations. I'm very excited to see what we can dig into come tomorrow I anticipate the coming week being an interesting one for AI!!

---

Apologies for the brevity, but time is in remarkably short supply at the moment while work is in remarkably high supply. Send us all your positive vibes for this week -- with luck we'll be nailing low-level gameplay systems, physics, and AI all at the same time!



Link to original: viewtopic.php?f=30&t=6176

Tuesday, June 27, 2017

Hey guys!! Sincerest apologies for the long wait on this log -- I hope no one was getting any misguided thoughts about 'Josh is kill' this time Between me being hammered with a cold for the past week and the entire team being hammered by 'minor' annoyances standing between us and our current goal of a 'minimal working LT,' it's been rough trying to stay on top of things. Especially. With. Dayquilbrain. Please excuse me if I'm not as coherent as usual (HA!!!!!)

---

Big-picture-wise, we're pushing hard to get a 'mini' LT -- or, as Adam ingeniously dubbed it -- "Lil'T" built and running. Lil'T is kind of what I was gunning for a while back when I thought I had a good ECS, and wanted to implement everything in terms of the 'real systems' for the most recent public showing of LT. Our goal is to take all the foundational work we've done up to this point -- Adam with physics, me with ECS impl, Sean with AI -- and build a small LT out of it. The critical idea here being that we're doing everything the way we would do it for release, which means, theoretically, no throw-away code (yes, I'm well aware that LT has a long and rich history of non-throw-away code being thrown away....ahem...but...we are more practical now!!)

So yes, we're pretty busy! As for the past two weeks, however, they honestly haven't been my favorite. Adam and I have had to spend a lot more time than we wanted to getting our Windows build system to play nice as we continue to grow the codebase and factor out pieces into different libraries and such. I won't even get started down the path of this rant....I could rant for years and years about how incredibly broken, braindead, absolutely neuron-singingely stupid modern build processes are. It's like we're still cavemen when it comes to turning code into machine code :V But I can't let myself digress...the good news is that we're through it. Things work. And if they ever stop working because of one more flippedy-flopping bad decision that someone made a decade ago, I'm going to burn down the entirety of the known universe. So that's that

On the other hand, Sean over there has been having what I would say is his most productive week(s) yet! After I was able to build a little visualization sandbox for him, in which he could hook up his Lua sim code to drawing functions, he immediately started to explore some territory that he's been itching to get to: low-level AI maneuvering. We talked about 'flow fields' (aka gradient descent on a scalar potential field) and how they can be used to create very natural motion in the face of many conflicting goals (avoid obstacles, avoid hostiles that are stronger than you, stay in formation with fleet if applicable, move towards destination, align to firing trajectory, etc.)

A week later he's got a fleet of ships following a leader, in formation, through a dense, heterogeneous asteroid field, and it's looking pretty awesome as they break formation to avoid collisions, then form back up when the field allows it. Very cool stuff to see. Once more, I envy Sean for the work he's getting to do! I can't wait to see this maneuvering code in 3D with real ships (right now it's quite graphically-limited by the sandbox).

Other than that, honestly, loads and loads and loads of technical gibberish. Surprisingly, I'm not going to subject you guys to it I think it's probably because I am, frankly, not excited about it. Yes, technical gibberish is what makes games work and is what we have to push through to get to Lil'T (don't you just love the name??), but I haven't seen a procedural system in a week or so now and it's starting to hurt. Hopefully by the end of the day we'll have stuff running, because I'm really starting to miss the spacy vibes

---

In summary: All hail Lil'T!! \o/ Send your assorted pagan offerings now, because Lil'T is comin at you FAST! (Yes, please do keep in mind that I'm still a bit...wonky...from the cold medicine.. )

See you in less than two weeks! ()



Link to original: viewtopic.php?f=30&t=6206

Saturday, July 24th, 2017

>> FINALLY!! <<

I'm so sorry about the delay this time around folks; as Nathan has informed you guys, I've been completely deluged with my participation as a teacher for a local coding summer camp. I should have posted a heads-up before the thing started, but I didn't realize it would vacuum up all 9001% of my minutes :V

FYI: Please consider this update to also serve as the dev log for the time period in which I didn't write one. Writing one in retrospect at this point really wouldn't yield much more than you guys already know, and I anticipate a pretty big week now that we're back to full-steam-ahead, so I think it makes more sense to just write one on Friday and resume the usual schedule.

Very happy to be back in the fold...teaching was rewarding, but way too stressful. Also not as shiny. I miss the shinies

Off to the Lil'T I go!



Link to original: viewtopic.php?f=11&t=6245

Saturday, July 29, 2017

Hello hello everyone! Glad to be back to the devlogs. Let's hop right in. It's going to be a long one.

It was a tremendous week. With Sean and I back in the office (and me particularly energized and ready to get back to LT after the stress of teaching), we were already in a position to have a good week. But several major happenings upgraded it the past five days from good to awesome.

---

Adam: (King of) Surface Extraction

This week's award for MVP goes to Adam, about this there can be no question.

Before my three-week teaching leave, I set Adam to the task of studying distance field surface extraction techniques and finding out if there were any good, open-source implementations out there that would allow us to upgrade our distance-field-based mesh pipeline. If you've been following the logs for a long time, you've definitely heard me talk at lengths about distance fields (SDFs) and my various adventures with them. The LT Prototype used SDFs as the exclusive means of generating meshes. Today, they're used for non-artificial constructs like asteroids, while BoxMeshes (the new PlateMesh) are used for ships, stations, and components thereof. Generating meshes from SDFs is a very difficult problem. I've implemented two algorithms for doing so -- marching cubes and surface nets, and currently use the latter to generate asteroids. Both of these are 'naive' algorithms in that they're just about as simple of a solution as you can possibly find. Despite that, they're still quite difficult to understand and implement (SN more so than MC).

I've often pined about not having a great solution to this problem. I'm sure I could find at least a dozen mentions in past devlogs of 'what ifs' with respect to having a really good surface extraction algorithm, and how everything would be golden if we did. Thanks to Adam's work, it's no longer a 'what if' situation

Adam not only succeeded in finding an open-source library that has the functionality we need (and more), he also went through the struggle of figuring out how to build, test, use, and evaluate the quality of the code therein. This week, he was able to show concrete timings and outputs that proved the library capable of performing adaptive, high-quality surface extraction with great speed.

The implications are vast. I've only just started working with the library on Friday when Adam handed it off to me, so I can't say with 100% certainty that we won't run into snags. But already I have been able to do enough that I'm extremely optimistic. With luck, it'll solve/enable the following for us:

• Unified procedural modeling pipeline with everything constructed from distance fields
• No loss of 'sharpness' for objects that should be sharp (like ships, stations, etc.)
• Support for far more detail and complexity in our PCG mesh algorithms, translating directly to more creative variation and higher quality for our models
• Substantial decrease in cost of rendering meshes due to adaptive extraction
• Decrease in cost of computing ambient occlusion for meshes
• Increase in ambient occlusion quality, especially for complex meshes
• Potential solution to the problem of efficient texturing of our meshes thanks to manifoldness (BoxMeshes did not produce manifold meshes)
• Free LOD meshes with optimal fidelity, leading to even more rendering performance at no dev cost to us

At the end of the day, all of that basically boils down to both a lot of dev time saved, a substantial performance gain, and way more flexibility for our PCG. Yes, I'm very happy about it

Sean: AI Combat, Teams, Hierarchical Fleets

This week, Sean dove deep into AI combat. He's set up a combat sandbox that can run really fast combat simulations and provide us with information about the performance of different combat AIs. At the most basic level, we can run head-to-head dogfight simulations between two AI ships using two different algorithms and determine which is superior (in particular, we can determine the win rate of both ships under randomized starting conditions).

After he had gotten the basics down, I tasked Sean with making the sandbox more general, so as to support 'teams' of arbitrary size and count. After only a short while, we had an even more powerful sandbox that can rapidly simulate entire 'faction vs. faction' battles and give us insights about how fleet composition, starting location, and AI reasoning all add up to prowess on the battlefield. At one point, Sean simulated a 100-faction battle with thousands of ships. It was entertaining but terribly slow since his sandbox isn't using our optimized engine as a backend. Once I finish my scaffolding of Lil'T, we'll move his code into the real engine so that he'll be able to use the ECS to simulate such battles in milliseconds

Sean is now working to implement arbitrary fleet configurations and develop combat algorithms that will behave intelligently within the context of a potentially-massive fleet structure (unit X is part of fighter squadron A, which is flying in v-formation escorting cruiser Y, which is part of offensive line B, which is moving in wall formation 10km ahead of battleship C, etc). This is the pinnacle of the general AI combat problem, and a very difficult one indeed. I don't anticipate a solution to this to come about as quickly as he solved the other problems, but I'm sure he'll still end up surprising us

It continues to be really nice that we've got someone working on the hard gameplay problems as Adam and I blow away all remaining technical ones

Josh: Lil'T & LuaJIT

As for me, I've been at the helm of Lil'T development this week, trying to get it all to a point where everyone can start working in the same place. Practically-speaking, it means that I have to have 'the way we do things' locked-down in all departments. I've had some really great success in doing so this week!

So, you're all aware that we've struggled with FPLT in the past, but that LuaJIT + high-quality C systems have proven capable of yielding beautiful performance. As I've expressed before, the only caveat is that one must tread lightly with LJ in order to reap that performance. A single mistake in a tight loop will cause LJ to abort the compilation of something that may run far too slowly in the interpreter. I had previously discovered that my code for LTDemo was doing exactly that, and getting almost none of the benefits of JITing (ouch). This week, while building the Lua base of Lil'T, I had to methodically tackle each of my previous mistakes and figure out how to do the same thing without making LJ angry.

After countless hours poured into reading LJ source, performing experiments, watching trace outputs, reading through LJ's generated assembly dumps, and trying various techniques to determine what does and doesn't work, I can say with confidence that I know exactly how to build Lua code that maintains all of the benefit of Lua's flexibility and readability, while still keeping LJ happy and compiling down to pure assembly. One breakthrough, in particular, was the discovery of a pattern that permits me to do almost all of the 'convenient' stuff that I was doing in the LTDemo code while still getting LJ to give us assembly! This means that much of what I thought would have to be 'thrown away' in favor of carefully-tuned calls to the C engine will, in fact, only require minor modifications! (For those of you familiar with Lua, it means that I can associate metatables with raw engine C types without incurring any overhead and still getting optimal assembly output -- meaning I can create wrappers that make our engine super easy to work with, and all at zero runtime cost. While it may look trivial to accomplish via the ffi.metatype LJ builtin, in practice there are loads of ways to botch it and end up paying a huge price for the convenience!)

In a final tour-de-force of this knowledge, I wrote some code that computes distance fields in Lua and uses our own wrapper of the aforementioned library to extract meshes. Even with JITing to assembly, I expected performance to be rather poor, as the code was just about as naive as possible. Much to my surprise, it ran blazingly-fast. Out of curiosity, I profiled it against optimized C (as in, gcc -O3 et al.) that does the same thing but uses smarter techniques to avoid overhead. Even with all those advantages, I found the C version ran only ~25% faster. When one considers how incredibly simple the Lua code looks in comparison, and considers that the C 'cheats' by using a faster method for populating the SDF grid, the performance difference is simply unbelievable. I'm officially and shamelessly promoting myself to the title of 'LuaJIT Jedi Master' I had entirely too much fun playing with extracted SDF meshes yesterday

---

All-in-all, this week was one of the best weeks in recent memory for LT development. Even as I woke up today, I felt more at ease than I've felt in quite a while. We've come so far with solving so many hard problems that lie at the very heart of Limit Theory's feasibility, and finally, after what feels like an eternity, the road ahead is clear.

The coming weeks should be packed with excitement

---

Although there's not much 'shiny' about this week, you guys continually tell me to post shots even if they're not pretty so...here you are...

Mandatory shot of me being a l33t hacker with all my LuaJIT assembly dumps. Notice LJ is taking almost no CPU and minimal memory. Possibly because there's very little happening...still impressive, though.



Shots of simple test CSG meshes, demonstrating how well the new algorithm works (notice the really nice adaptivity! Triangle size varies hugely depending on how many triangles are required to capture the details):




Sean made a few animated gifs from his AI work. Sadly, he didn't include the runs where he had huge numbers of ships I'll get him to make some for next week




The gallery is here (although you've basically seen them all now).





Link to original: viewtopic.php?f=30&p=153144#p153127