The Limit Theory Email/RSS News Thread

The S&M crowd is getting naughty with the game files.

The Limit Theory Email/RSS News Thread

Postby Talvieno » Wed Mar 09, 2016 4:54 pm

You can subscribe to this thread for future updates. This topic will remain locked but stay updated.
Link to Announcements RSS feed
You can subscribe to email notifications via a link at the bottom left of this page. This requires being logged in.

The posts in this thread are admin-created clones of Josh's original posts, and not posted by Josh himself.
The intention of this thread is to help people stay on top of news for the game even if they don't closely follow the forums. I'll add a date to each post as well as a link to the original. You can continue the conversation for the latest posts in their respective threads.
User avatar
Talvieno
Community Manager
 
Posts: 8214
Joined: Fri Apr 11, 2014 6:50 pm
Location: North GA, USA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Wed Mar 09, 2016 4:59 pm

November 16th, 2015

Kimny wrote:I don't know how many other different ways I can say I believe in him, I trust he is telling the truth, I trust his character, I like him, it's not a personal critique. Really, it's almost like it's necessary to spend paragraphs apologizing before making a criticism. And even I repeating all that in all posts, still one replies that I think Josh is lying, another replies saying I shouldn't distrust his character, another says I should shift to my personal life, you now reply as if I were trying to offend Josh.

I explained extensively in the other posts that on the contrary, I don't want to mix personal and professional. I really like Josh, trust him, etc, etc (second time in a post). I just think he has been in error in what regard not giving one piece of information during 1 whole year. Of course that is worth criticizing. And it's normal to criticize it. He is a professional, LT is a funded game. Sometimes it seems that taking 1 screenshot in an year, or even less, talking a couple of lines about the concrete game in an year, is to put super pressure or to ask Josh something very unusual. Indie developers certainly do not update on their games all the time as Josh did. Also as I said many times now, that's fine and I think he was very right in his change towards focusing in the game (I also have to repeat that all posts...). But it is not true that is normal and acceptable that indie developers go a full year without anything remotely concrete and behave as if it that was a plain fine decision instead of something to apologize for. Much less is usual that people are ok and even repeat in the developer's place that "progress is being made", that we should only thank Josh for everything nice he has (indeed) done so far or for not unplugging our critiques from a forum at the official page of the game!

It's very simple: he's been doing choices and following paths that deserve critics just as we would do with any developer. THAT is just what I meant by it's time to criticize Josh's approach to it. JUST that. But anyway, from now one I will reply with a signature that repeats automatically:I trust Josh's word, character and passion, I like him and admire him personally. Yet critique some of his decisions as I would do to other professionals (and that's a sign of respect). Btw, that's the third time self-justifing all that in the same post....


Hey Kimny, welcome :wave:

I like your mixture of respect and critique. I've no problem with it at all, and agree with you that it's been a rough and very-much criticism-worthy ride since last year. That's just the hard truth of it -- next time I make a game I'll be sure not to put myself into that position. But this time is difficult, I've made mistakes, had to go through mental issues, and had to spend a lot of time recovering from them and unraveling some of the poor decisions that I've made within the LT codebase during the time that I was losing my good judgement.

So the truth is, no, I don't have anything to show at the moment, at least not graphics wise. But there is a tremendously-good reason for that ;)

Spoilers!

Since people are getting angsty again, let me give some more detail on what I've been doing with LT. It's quite technical but very, very important to actually being able to finish the game. So remember how I used to have this concept of testing 'individual features' in sandboxes using LTSL? Right, well, that was all well and good. But at the end of the day, it actually wasn't all well and good, because, LTSL used the entirety of the game engine (liblt.so , liblt.dll, liblt.dylib depending on platform). It had no choice, because the game engine was one big blob. That means that, while each sandbox may only have called on particular features of the game engine, the entire engine was still there, in the background. Way too stuff was affecting the sandboxes simply because the engine wasn't separated, it was 'monolithic.' Changes to improve the functionality of one sandbox would end up cascading through the entire codebase and, far too often, breaking other functionality in other sandboxes. It was remarkably frustrating and time-consuming.

I've spent the past few months, in particular, completely changing this problem. LT is no longer a 'monolothic' engine but rather a compartmentalized ('micro') engine, comprised of many interlinked object files, each of which are independent from most others. Blah blah blah, technicalities. Why should you care? Why am I not able to show screenshots of this?

Here's the beauty: there are no screenshots of the game simulation, of the AI, of the core game logic, etc in isolation -- and yet, I can develop them in isolation because one does not need graphics to develop any of these critical pieces of the game!! 'Sandboxes' are now simple console programs that give me readouts of market data, faction stats, etc etc. They have no concept of graphics. OTOH, the asset generation algorithms, which are a heavy WIP at the moment (I hope to be able to show you all some output as soon as I finalize the structure of the algorithms), need graphics but no other concepts from the game. They are simple graphical sandboxes to view single models, just like Maya or something of the sort.

So. The game? Here's the deal: you won't actually see it until the component pieces are finished. At that point, we will wrap the player interaction layer over all the components, and that will be the LT alpha.

That's a high-level look at what's going on right now. Frankly, I'm really excited about it. On a day-to-day basis, it makes me a lot more comfortable to be able to work towards a specific goal without touching code that needn't be touched. In general, it makes me quite a bit more productive. It might sound like another one of my little 'perfectionist' tendencies, but, in fact, if one considers the complexity of a monolithic engine vs. a micro engine, roughly-speaking, it's the difference between quadratic O(n^2) time to develop and linear O(n) time -- that's because the separation of dependencies ensures that touching one piece of a micro engine doesn't cause a cascade effect that breaks all the other pieces. This is the epitome of the "wall of separation" concept that every good CS student should learn, but which I really failed to do well during my first attempt at structuring the LT engine.

"Ok ok ok very technical, much wow, BUT I STILL WANT TO SEE THE GAME AND STUFF." I know. Once I have asset generators in good shape, I'm going to stop being so stingy. I will be showing example assets as they come out of the pipeline when that pipeline is of sufficient quality. But, once again, as I have hinted at before, the next time you see a shot of the whole game (not just graphically, but including interaction and gameplay) will essentially be very, very close to BETA.

Did that make any sense?? I hope so! :geek:




I just want to add one more thought. For those of you who remember the original "Limit Theory" and where the name came from (the Theory of Nonexistence of Extrinsic Limitation -- scroll down to "The Meaning of Limit Theory"), it is actually critical that I implement the philosophy of Limit Theory in my work on Limit Theory. This game cannot be brute-force made by one person. It is only by the coalescing of a million tiny 'insights', the breaking down of a million tiny mental barriers, that a game like this can be feasible for a small team. Naturally, most of those insights are not graphical, but technical (or artistic-technical). We're all tired of hearing about the technical, but, at the end of this project, when Limit Theory is out and in your hands and you're making your way through a beautiful universe doing whatever you please, you'll have one and only thing to thank for that: those insights. Not a hundred artists, not a massive development team with a massive budget, but an engine and game built off of 3+ years of recursively searching for elegance in each piece of the game. When you are playing LT, you should thank the Limit Theory :)

The dev style is the embodiment of Limit Theory, and the release of LT is intended to be a 'proof' of sorts. The name was chosen as such from the beginning to reflect this concept :) Again, for those who haven't been with us since the beginning, go back and read the post I linked. It should tell you all you need to know about what's happening right now. I am searching for the final insights that I need :geek:



Link to original: viewtopic.php?f=11&t=4640&p=114850
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4261
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Wed Mar 09, 2016 5:03 pm

December 5th, 2015

tl;dr - I know what I'm doing.

Zanteogo wrote:It's just from this side of the computer screen I remember the years of LT development where every month massive improvements and changes were occurring and it seemed like you were on the highway to completion. It seemed like everything was coming together. I know that in almost all software, it's the way of things, the first 80% is the fastest, the last 20% takes 80% of the time. However, it seemed you had mostly side stepped this trend with your one man team approach, (for what ever reason). I recall the day when your showed us LT, and the AI was doing it's own thing and reacting in it's own way to changes, and I recall in awe thinking, "it's happening".

I know I am arm chair quarterbacking here, I am merely seeking to understand and to remove my ignorance to the whole thing.

However, to me, it was just before the start of LTSL that everything seemed to.. stop. Everything seemed to be put on hold. For the things that didn't, it's when the circular development seemed to really happen. Again, this is all from the prospective from this side of the computer screen of course.

From what you have opened up to us on your dark days, you admitted to getting stuck on circular cycle of not really moving forward. You also informed us that something you had done to LT had to be undone.

During the half of year of LTSL development it was sold as the thing that would cause a massive content explosion. I know your keeping your cards close, however, so I don't know what you have done since, but I am guessing there has been no explosion. If I am mistaken I and truly sorry.


Yes, those were back in the days when I was simply pumping out C++ as fast as I could. And there's nothing wrong with that. I could resume that approach today and have a much lesser form (and non-moddable) of Limit Theory out in a very reasonable timespan (and that's only because I've re-architectured the engine and now know how to prevent monoliths). But that's not going to happen, because, as my conception of Limit Theory matured, modding became something that I simply had to have. I want you all to be a part of Limit Theory's development. I want to see what people can do with this technology over which I have slaved for years. I want to play insane variations of my game that blow my mind. And it's all very much possible. I'm fine with dropping other content (especially content that has crept in since the original design doc) -- it can be appended later via modding. But I'm no longer fine with a non-moddable LT, and this is where the real challenge lies.

About LTSL -- the need for LTSL originally arose from a feeling of uncleanliness that I was getting from the LT codebase. It was taking me longer and longer to make trivial changes and recompile / view the outcome, it was getting harder and harder to figure out what was affecting what, and debugging was becoming more and more challenging. All of these problems were scaling linearly with the size of the codebase, meaning the difficulty of development was, effectively, scaling quadratically with time. This is, perhaps, responsible for how 'suddenly' things seemed to change. Hitting the wall was not a linear-time process, unfortunately. LTSL was nowhere near a mistake, and it did cause a fairly significant explosion in my eyes. A lot of it was UI, but that's because UI is one of the most annoying things in existence to hard-code. Remember the latest market interface? The assets interface with holographic views of each asset? There was also a working ship builder UI that was to be revealed in RTB 3.0, which never happened. Warp rails and their graphics effects, HUD, radar, target information UI, scanner UI, custom AI maneuvers, new station algorithms, new ship algorithms -- all LTSL. LTSL did the job. But it was a temporary 'escape' from the real problem: that the engine had grown too difficult to deal with, hence the need to 'escape' into a lightweight scripting engine.

But, as nothing is perfect, LTSL began to show its own problem -- performance. Towards the end of LTE (the Limit Theory Engine), much of the functionality was offloaded to LTSL in an attempt to fix the 'monolithic' problem. But LTSL struck back with performance limitations. It couldn't handle the intensity of true engine work. We were stuck. I could keep pushing and pushing and pushing in an engine that was becoming increasingly difficult to maintain, I could suck it up and accept the (fairly drastic) negative performance implications of engine work in LTSL, or I could use everything I had learned from both experiences and do something better. Anyone who knows me knows immediately which option I chose.



Here is what I would like everyone to fundamentally understand about Limit Theory:

It has ALWAYS been about working smart -- it has ALWAYS been about creating a beautiful technology in order to enable the creation of a beautiful game.

In the beginning, plenty of people said "no way, not gonna happen with a one-man dev team." And then guess what happened? I developed technology that allowed me to show the progress of a game at quite an impressive rate.

Despite the paradigm to which so many have become accustomed -- "shiny new content!" -- Limit Theory development has always been fundamentally about THINKING and finding the smart way to do things. That's the only way it can happen.

And that's exactly what's happening now. Yes, it has taken time, but I now have an elegant technical solution to both the monolithic engine problem AND the scripting performance problem. That much I will divulge. When this solution has been built up to the point of supporting all of LT's existing content, the content algorithms will simply be transferred over -- no work will have been in vain (not even LTE or LTSL, because they were important milestones for my coming to understand the right way). This time, I can guarantee that there will be no circularity, simply because there is nothing more that I could possibly do, technically, to energize LT's development than what I am doing now (to explain what I mean would be both too divulging and too technical). There will be plenty of back-and-forth in finessing the game, but not technology. One can either choose to believe or not believe me in this regard...it doesn't really matter, as LT will come in time and all of this will be over when the release drops.

^ This is the last time I will provide a justification of my current direction, I'll be keeping this link handy for the future.



And about that 'finding an entry point' thing that someone recently expressed (imagine my joy in being reminded of this dev log) -- it was never actually an issue. It was never about Limit Theory. What I was expressing in that log was a projection of my mental state onto my current task. The reality is that I was degenerating mentally to the point of being unable to see a big picture, to find a coherent strand of thinking to follow, to see LT as the beautiful concatenation of interesting subsystems that don't have an entry point. It is entirely possible to develop each independently, and to perform incremental testing by joining them as they are ready to be joined. I no longer have the problem that was being expressed in this log, because it was not a problem with Limit Theory -- it was a problem with me, and, as you all know, I've been working to fix 'me' since the end of the dark days.



PS ~ I would like to apologize for my tone. I realize it is unusual for you all to see anything but a dapper Josh. Current life circumstances + a change in medical circumstances + the tone of this thread have all summed up to a not-so-cheerful forum demeanor. For that I apologize.[/quote]

[quote="JoshParnell"]Thank you, Flat, for sticking with us and continuing to provide valuable input at every juncture. I hope you know that your words don't go unread by me, and, despite the points on which we may disagree, even in these late hours you continue to provide thoughtful-yet-non-destructive input on the dev process.

I'm sure many people think that their words fall upon deaf ears with respect to me (especially given how little time I have to actually respond to the plethora of discourse on here). Such is not the case (not since the Dark Days, at least). Criticism isn't going to cause me to tear down my whole development approach, but it certainly doesn't pass through my mind without leaving something behind.


Link to original: viewtopic.php?f=2&t=4859&start=180#p115867
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4261
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Wed Mar 09, 2016 5:05 pm

January 22nd, 2016

http://theadvocate.com/news/14649715-75/louisiana-technology-park-adding-video-gamers-iron-27-procedural-reality-as-tenants

Had to give a presentation to the 'board' today at work to be 'officially' accepted into the Tech Park (despite the fact that I've had an office and been working there for...a long time now...would have been pretty awkward had they say 'nah, get out' :ghost: ).

They were very pleased with what they saw :D Apparently several wanted to be on my advisory team, so I've now got some serious businessfolk who will be helping me structure the timeline / work / everything else as we get closer to release (I can almost feel the communal sigh of relief at the thought of Josh having some form of 'project management' team!) :)

Woo :thumbup:

PS ~ This article is actually how I learned that they voted 'yes' on Procedural Reality / LT. Hehe. Although I guess someone would have broken down my door and told me to get out had they voted no :monkey:


Link to original: viewtopic.php?p=118714#p118714
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4261
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Wed Mar 09, 2016 5:06 pm

February 24th, 2016

OK, seeing a bit of angst so minor update time again:

Last week brought several major breakthroughs that led to a huge leap in the completion of the mysterious platform upon which LT now lies. It is actually quite different than I expected, but everything that I've done, all the engineering, re-engineering, and various weird tech ideas that I've played with since the beginning seem to have somehow come together in a flurry and led me to the final result, and I'm really pleased and excited about it, both from a technical standpoint (yes, there's some very cool tech underlying that which makes the engine and game both performant and manageable), but also from a content authoring & gameplay standpoint. Fundamentally, it's EASY. Both engine and game work are easier even than writing LTSL was, but orders of magnitude more performant. Finally the tech investment pays off!

I think, when I reveal this final 'new LT' and the architecture underneath it, I will get some :wtf: :? reactions followed by :shock: and then :think: :o :o and, finally, ending in :clap: :thumbup: :D :squirrel: :squirrel: :squirrel: .

Best part is, I am now at the point where I can concurrently develop any pieces that I wish, be it engine, game, or the platform on which it all rests. So yes, game code is in the works concurrent to the finalization of the platform. It's pretty elegant how they can be developed concurrently, but in time you'll all get the details on that! I wouldn't have been capable of writing this solution years ago -- perhaps not even half a year ago, so, as strongly as I regret the fact that LT still isn't on you guys' hard drive, I think the timing happened to work very strongly to LT's advantage.

Good news is good news :) Still being stealthy, but man am I seeing the light lately.


Link to original: viewtopic.php?f=2&t=4859&p=120853#p120850
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4261
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Wed Mar 09, 2016 5:08 pm

March 9th, 2016

So yesterday I had written a rather lengthy response / update to this thread trying to clarify some of the questions popping up about the old vs. new underpinnings of LT...then I had to move locations, and when I hit submit my internet was gone and, naturally, phpBB doesn't save the form state so when I hit back my response was also gone (so I gave up). Gonna try again today, using the clipboard more frequently :ghost:

Here's a basic rundown of what was and what is, why we're in much better shape than before, etc :)

Old Setup:
  • Engine and most gameplay components written in C++ (~100+ source files since I like to keep things separated). Used some heavy-handed C++ constructs, adding to compile-time.
  • Despite being separated in terms of files, engine components were highly-interconnected, as I wasn't good at modular structuring when I started this journey.
  • LTSL implemented via compilation to expression tree (like an executable AST), scripts executed via this tree -- a fairly slow but clean approach
  • High build times due to interconnectivity and 'monolithic' structure (LT engine implemented in a single DLL / dylib / .so); even making changes to an isolated feature required an annoying amount of link time
  • Above problem led to higher use of LTSL for more and more features, eventually leading to exposing the performance limitations of LTSL...this ultimately backed me into a dark corner

New Setup:
  • Minimal 'platform' written in very light C++ (basically C; no usage of the 'heavy' C++ features, no usage of C++ standard library); platform contains core graphics / audio / system functions (i.e. the microcomponents of the engine)
  • Everything else, including higher-level pieces of the engine as well as gameplay, written in a specific subset of an existing high-level language (i.e., an industrial-strength one :P ). The platform configures, injects functionality into, and then starts up the language's environment. LT is then started in the HLL.
  • Wait! Engine functionality written in a high-level language??? Sounds like another performance disaster / LTSL situation waiting to happen! -- probably what you should be thinking right about now. This is where 'magic' happens. The platform on which everything rests allows for something rather awesome -- it allows for pieces of the HLL code to be transformed into native machine code that executes as fast as anything that was pre-compiled into the platform. The awesome part is that the code can remain written in the high-level language, it just needs to adhere to a few things, and mark itself as wanting to execute 'natively,' and the platform automatically takes care of the rest.
  • Ultimately, this means we have both performance-critical and non-performance-critical code written in the same language but executed in different ways. So the beauty of it all is that I don't need to guess whether or not some AI routine or PCG algo is going to be slow...if I run the game and it's slow, all I have to do is mark it (this is done in the language itself, via a mechanism I added), and next time it'll be converted to pure, optimized machine code that runs natively, just like all of the old LT used to run. There are a few drawbacks for code that wants to do this (namely, it can't use the full featureset of the HLL, otherwise it wouldn't be possible to convert it to low-level). But overall it's an awesome setup that makes development so much more tractable AND makes sure that we're ready to handle performance issues if and when they arise.
  • Due to this structuring, engine and game code can be developed independently. Entities needn't be rendered in 3D just to see basic information about what they're doing, what's in their inventory, etc. The old testbed approach is just magnified 100x here, because we can play with even more minimal pieces of the game thanks to not having a monolithic engine.

Please excuse my poor articulation of that all....I'm so caught up in the code right now that I think my english is degrading :P At any rate, the point is, this restructured foundation for LT solves both the performance issue and the development complexity issue (good lord, I can't even tell you how many lines of code are saved writing entity logic in the HLL!)

Yes, details are intentionally withheld until all is ready to be revealed :) For now, though, I hope some of you can take peace in knowing that the platform is finished enough to be able to work on both game and higher-level engine code. All three of these things can be worked on concurrently. And gameplay code is being written - far more quickly than used to be possible :squirrel: It is currently first priority, as I want to know as soon as possible which pieces of gameplay code will require the platform's 'magic,' and then test to verify that it does indeed remove performance bottlenecks as I expect (naturally this testing has already been done (successfully) on typical performance-benchmarking code, just not with LT gameplay code yet). Having written a good bit it already, I can tell you guys that, all this technical stuff aside, the single best part of this new approach is that building LT actually feels like building LT, not like thinking about the best way to implement each tiny piece of it. In that respect, a HUGE burden has been lifted from both myself as well as future modders!

I told you all I was going to play it stealthy until I'm ready to show everything, but there's a little peak at the structure so that you all don't have to guess as much (although now I'm sure I've left new things to guess at...) :ghost:


Link to original: viewtopic.php?f=2&t=4859&p=121865#p121861
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4261
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Tue Apr 05, 2016 8:54 am

April 5th, 2016

Hyperion wrote:It's been itching at me for some time now, as it probably has for plenty of others. I'll put it simply.

I want proof that development is taking place, that the game is closer to being finished than it was last year.

Not "Things are exciting" or "I can work so much faster now" or "I know what I'm doing" Not words and rumors about release date leaks, but cold hard proof. An image or video of something new, something we haven't seen before. It doesn't have to be much, but unless I missed something, it has been over 14 months since there was a single shred of proof that the game has developed beyond where it was in January of 2015.

Josh, I'm not calling you a liar, and I understand that you're holding your cards tight which is fine, but absolutely nothing since you declared the beginning of Golden Days almost 8 months ago? I'm sorry, but I'm not big on faith, and this total silence is testing those limits. Words and promises won't keep that alive, only evidence would carry me months and miles longer, and I have a feeling I'm not alone.

Fair enough, I can understand all viewpoints expressed in the thread so far (with the exception of TGS's 'nil' expectations for this game -- come on now!)

No game or engine screenies -- I'm still not graphically back to where we were previously (meaning I haven't brought all of the rendering and shader tech over to the new platform yet)...have actually been focusing a lot more on the gameplay side of things, and can I just say (I think I've already said this)...it's an absolute joy writing gameplay code in augmented Python (yes, this was going to be revealed in the 'proof' anyway) as opposed to C++.

I use dropbox for my version control (wait, what?). Yeah, seriously, consider it: I have like 4 different machines at the moment, and I need code to be current across all of them. At the same time I need the ability to rollback. Dropbox makes both easy (you pay an extra $15 or something like that per year to get an entire year's worth of version history regardless of how many versions there are or how big the file is....that's kind of amazing). The moment I save a file it's as though I've committed a new version, which is immediately available on other machines. Yes, this is awesome for a solo developer. No, don't try this on a team or many people will be angry.

So for your 'proof' I just printed out two simple version histories, one for ltheory.py (the entrypoint that basically launches the game without doing much else) and one for pyengine.cpp (please DON'T start using the name 'PyEngine' to describe what LT runs on because that's a temporary name!!) which is the entrypoint for the 'compiled' part, which, as I've explained before, basically configures and sets up what you now know is a Python interpreter, and injects facilities for graphics programming as well as JIT magic for pulling any of the performance-critical python down to native. Those histories, or rather, a few pages of them -- dropbox does a scrolling load thing -- are attached. Incidentally I did edit them both earlier today, despite the fact that these two files don't change much in comparison to the directory structures (hence the gaps). Most of what I do now is adding components (all of which are part of a module), data structures that are part of gameplay, shaders, etc. So most of my change is actually adding to the directory structures (no, sorry, not going to give a dir tree! Too many secrets :) ). In some places I actually have code that automatically scans a certain directory to load assets (this makes both dev and modding easier! For example, to create a new component that can be added to an in-game object and give it some kind of behavior, all you have to do is write a new code file (yes, .py) in the component directory and, if you did things correctly, you'll be able to call object.add(MyNewComponent(blah, blah)) without changing code anywhere else!

I know a few of you will now undoubtedly have reactions to the fact that I've just revealed the game language to be Python, (yes, I'm sorry Flatfingers...I am...), but please keep in mind
  • Huge, huge, huge amounts of thought went into choosing it over all the other options I had on the table, and after a lot of going back and forth, pro's and con'sing, running different tests, I was able to make this choice with full confidence.
  • Performance-critical code will not run in Python, so restrain the knee-jerk "oh noes it's going to be slowwwww" reactions (I mean, I would have that reaction if a dev said he was using python for a serious game...until he mentioned native JITing :) )
  • The 'magic' mechanism is of my own coding, this is not pypy or psycho or Cython, this is Josh + CPython. I chose this route because I believe it represents the widest spectrum of performance-devtime trading capability. None of the above, with the exception of psycho (which is now abandoned), could routinely perform as well as native code (Cython comes closest but it doesn't have as much runtime information to work with as I do :geek:). On the other hand, since it's CPython, it's a full and complete implementation of Python 2.7 (yes, 2.7, not 3.*, another calculated decision), meaning no restrictions or quirks. So in your python functions, you've got as much 'expressiveness' as you can possibly have. In your python functions that are marked for native conversion, you've got as much speed as you can possibly have (although you do lose a bit of expressiveness, naturally).

Yes, I was waiting to drop the P-bomb because I knew I would need to provide some comforting words with it :P So, those of you who plan on modding LT can run off and start learning Python (2.7!) now :)

And those of you who want proof that I work on LT, check the pdfs. If those aren't enough (because it's definitely worth my time to forge version histories?? :ghost: ), hang tight. I will try to get graphical 'proof' of the underlying progress as soon as I can :thumbup: Once again, I really appreciate you guys being patient with me. It's been a longer journey than I ever imagined, especially in terms of my own state, but it is converging :)

ltheory version history.pdf
pyengine version history.pdf



Link to original: viewtopic.php?f=2&t=5060&start=15#p123136
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4261
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

The Limit Theory Email/RSS News Thread

Postby JoshParnell » Thu Apr 28, 2016 10:11 am

April 27th, 2016

Hyperion wrote:Module instancing is actually proving to be a slightly difficult problem to hash out. Getting a good way to track 10,000 eventual variations on a single blueprint for every single blueprint without making the system cry is probably doable, but is beyond my realm of expertise. Just as a brief outline of my (probably dumb) thoughts, because this isnt really the place for it:

Blueprints are folders, but also instructions coded with the particular Tech stats. Inside the folder is a registry of every unique instance yet made and its stats, each tagged with the owner and where it's used. The owner and usage tags change freely via trade & plunder.

The currently simulated blueprints are known scripts, executed as necessary, changing the stats of the given instance.

Players & NPCs can also execute their own scripts (aftermarket modifications, improved/shoddy construction techniques & materials, overclocking, etc)on a particular instance which also changes the instance stats, but are wholly different from blueprint scripts. Such modifications don't affect the registry serial, but can affect the Human-readable name.

Destroyed instances (or intact but attached to destroyed ships/stations) continue to exist in the registry for potential salvage and reactivation until cleanup.

Temporary instances can also be attached with semi-random stats to permanent debris fields, but disappear as soon as a player is out of range.

Is this efficient? No idea :lol:


I think you've got most of the general idea, but I'll explain the actual implementation anyway (and why it is efficient). A blueprint is a pointer to another blueprint (its 'parent' or parent directory in your model), plus a list of attribute operations (you can think of that as a very, very limited script, but really it's better thought of as a 'diff' between the parent blueprint and itself -- an object-attribute-space differential, so-to-speak). This is highly memory efficient, as we require very little memory to make lots and lots of slight variations of an object. If the variation is slight (for example, a delta in just a single attribute), then the memory required for the BP is very, very small (quick mental calculation says ~20 bytes on 32-bit machine, ~32 bytes on 64-bit). "Absolute Original" blueprints (i.e., those that haven't been derived through researching a stem of an existing blueprint or implicitly derived by modifying an object (note that virtual blueprints are not visible to the player, but rather an abstract BP that exists solely to help the AI understand and quantify the nature of an object that's been modified -- by keeping an up-to-date virtual blueprint for the object, we allow the AI to apply the same reasoning to all blueprints when it comes to any type of analysis)) are simply blueprints whose parent field is 'None' and whose attribute operations consist of setting all attributes to their correct values (in other words, they're a diff with a 'null object', so to speak). These are significantly bigger than derived BPs, especially when they are for objects that have a 3D representation (because it is the base BP that actually holds the model). This paradigm of using chains-of-diffs pops up frequently in LT. It's a really, really good way to save memory when we're in procedural land where there's a huge amount of generated content, but where that content often has significant underlying similarities that can be captured in this original + diff + diff + ... manner. Kind of like prime factorization :geek:

So the key here is to think of a blueprint as a self-recursive object, much like a tree (except that it's really just a backwards-linked-list). As far as who's got them and when we need to delete them, this is of no concern: when a blueprint is lost -- let's say a ship carrying a blueprint that has no other copies in existence1 blows up and RNGesus decides the BP didn't survive, hence is nowhere to be found in the debris -- the refcount goes to zero and it's gone from memory. Reference counting really does work beautifully. Same applies to any temporary instances that might be generated -- as soon as no object is holding a reference to it (because, for example, the debris container whose inventory held the reference to this blueprint died of old-age and vanished from memory), it vanishes. Note that this hypothetical situation of finding a blueprint in a debris field may not even happen, since we try to always have an explanation for things existing rather than just 'generating them.' Then again, fields are natural sources of value within the game as they have a rate of mineral replenishment, so I could explore the concept of extending and generalizing that to different fields having their value source exist in different forms. :think: For example, a debris field might have a certain rate of 'value replenishment,' meaning once enough time has passed and the field is not at max value capacity, it may spawn an item of value <= its wallet, so to speak, and deduct that value from its wallet. In this way, although we spawn things, we still retain total analytic control over the injection of value into the universe. Doesn't seem too off-track from our motto of real gameplay dynamics, since we already do this with minerals. I might, however, apply a nonlinear valuation function to 'artificial' objects in this case, such that it's generally easier for a field to spawn a natural object or a very common (cheap) artificial object, and disproportionately hard to spawn really valuable artificial objects (I'll probably end up making it a power function, and throw the power into the universal constants so that the user can tweak it for a different gameplay experience -- that's what I do with most things related to valuation and value transfer, since they're somewhat 'arbitrary' compared to many of the functions and formulae in the game). An actual original blueprint is both artificial and typically of very high value, so I wouldn't expect to stumble over them often, even if you're a seasoned explorer who's seen a lot of fields.

Dear lord, I think I've just unified field resource regeneration mechanics with project manager capital expenditure mechanics :wtf: So really, an asteroid/ice/debris/whatever field is a project manager for a sub-component of the 'universal project' -- it gets a steady drip of credits flowing into its wallet, courtesy of the universe, (until said wallet reaches a cap, same as in the auto-manage component of the projects mechanic), and expends credits as it sees fit, depending on the type of field, to bring new resources into the universe. Think of that when you approach an asteroid field in LT...he's actually a high-pay-grade manager! :lol:

What am I doing. How did all of this start from your post about module instancing. I must be in the theorycrafting zone right now :monkey:

*opens vim and wanders off further into gameplay theory lala land*




1 Technically, to truly be gone from memory, all existing copies of the BP must be destroyed AND all objects built from it, AND the BP must have no surviving children (all BPs that were ever derived from it, if any, have been wiped from existence as well). So it's a very unusual event for a highly-utilized BP to be wiped from memory, since it will often have objects in existence that were built from it, hence carry a reference to it, and will often have child BPs that were derived from it (who will often have objects in existence that were built from them)...and so on. But that's exactly the behavior one would expect. It all 'just works' in refcounting :) An remember, BPs are very small memory-wise. I won't do the exact calculation, but a single planet's texture takes the same memory as roughly a bajillion blueprints.

Ahhh what is wrong with me why is this post so long :shock: :ghost:


Link to original: viewtopic.php?f=2&t=5060&p=124867#p124867
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4261
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

The Limit Theory Email/RSS News Thread

Postby JoshParnell » Fri Aug 12, 2016 12:53 pm

August 12th, 2016

Yikes, a few things certainly need to be said:

1. I've not abandoned you all. You may not see me, but I am with you! By that, I mean, I'm in this office working on LT.

2. I sincerely apologize if anyone feels that they've gotten the middle finger from me. Not my intention, ever.

3. Didn't pull a fast one on you all -- I know I'm going to have to keep saying this until the substantial update that you're all waiting for (good lord, with the amount of tension that's going to be relieved, I should start a poll to vote on a name for it...it needs some kind of epic, cathartic name that will go down in LT history) -- but there's really no option for me other than to finish LT. Don't take that the wrong way...I want to finish LT, I need to finish LT because it means so much to me and, after all, if you give up on your dream, then what do you have left? For me it would be devastating. Not to mention the devastation of letting you all down, which I already feel quite sharply due to how far overdue the game is. But it would be unbearably sharp were I to simply call it quits. And of course, it would be downright stupid, considering how far we've come, how many hours, how much energy has been invested -- there's no turning back!! Sometimes I really do wonder how anyone can think that I would actually consider that :monkey:

So, yes, these threads I imagine will continue until I present the next installment of the 'State of the Limit Theory' address. But said address is getting closer each day. Sorry for the angst, I guess it's worse for the people who don't know the relative closeness :shifty:

reegs87 wrote:I do hope that all is well with Josh. I would imagine that the guy has a tremendous amount of pressure on him. That being said it would be lovely to see an update sometime in the near future; this forum was one of my regular haunts....


Thanks, appreciate the sentiment. All is well, although it's been a bit busier than anticipated, but such is life. Recap of life in the past few months:
1. Dad sold house, had to pack up & move all my stuff
2. Femnode got a job in CA, took a trip out there to apartment hunt with her
3. Found and secured a new place to stay here (5 minutes away from the office!! LT Headquarters is just a super short drive away!)
4. Moved and unpacked all my stuff at said new place

But once more I seem to be in a stable life situation and am in the office every day cranking out the LT (and drawing up plans for how to repair the long-since-derailed hype train :ghost: )

kostuek wrote:I'm afraid, he may have hit the same wall as he did with LTSL - his Python business may still be too slow. If this is the case, he is probably back in his corner, thinking about what the hell to do now.


Thankfully not, so far LTEngine 2.0 (pff, I know, there's been a lotttt of refactoring over the years, but this is the first time I actually split the codebase) has been a joy to work in. I'd be lying if I said I wasn't anxious in the slightest about what kind of performance we'll see when I have all the logic running. But, that's kind of the nice upshot of having a hybrid engine -- if performance is problematic, you tell more of the code to come to the dark side...eh, the statically-compiled side. So far, though, all is well and only going to get better as I continue to improve the JIT engine to be able to handle harder cases (hence, be able to convert more of the code).

(Yes, this monkey is entirely aware that he did not give any specific details on the 'State of Limit Theory.' Don't read into it, because unless you read it in a positive way, you've read incorrectly :roll:)

PS ~ In case anyone considers me a lord/savior, which I hope is not the case, please cease said consideration...as we have all seen, I am, regrettably, quite mortal and flawed indeed :oops: However, this mortal still promises to bring you Limit Theory :thumbup: (remember my motto: "I'll ship it or I'll die trying!")

Link to original: viewtopic.php?f=2&t=5356&start=15#p132283

Talvieno's note: I feel like it's important to post this here, given how quiet Josh has been lately, so that no one thinks Josh is missing again or not working on the game. Normally I wouldn't post anything here that had so little info on the state of Limit Theory, but this time I'm making an exception.
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4261
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Mon Jan 23, 2017 12:08 am

January 23rd, 2017
The State of Limit Theory Development, 2017 Edition!

Link to original: viewtopic.php?f=11&t=5659

Hello and good evening lovely Limit Theorists!

Warning: please ensure that your seat-belts are fastened, cockpits fully-pressurized, and pulse weapons overloaded to at least 110% of standard operating power. It's going to be a long ride through this sector.



~ Introduction ~

    First off, I've got to start by saying that this post is coming to you several months too late, but, as I'll discuss later, I'm taking serious steps to ensure that I change my communication habits. I'm sincerely sorry for not being on the forums for several months; I know, coming back and reading some of the intermittent posting, that it's taken a toll on many of you guys' outlook on the game. With this post, I aim to lift that outlook, but also to open up the hood and talk about why things are the way they are at the moment.

    Before we get down to it, a word about my previous updates and the community's response. If you're one of the many posters who has said something to this effect, know that I've heard you when you said "no more vagueness," "concrete details," "more than promises/empty words," etc. I got it, my previous updates were not satisfactory, but give me a chance to prove to you that I can, in fact, change :) On the other hand, a forewarning: to those who have demanded "pictures/videos or LT is kill," let me tell you up-front that you're going to be disappointed. I'll explain in detail why that's not what development is about right now, and this is the part where you all are going to have to meet me halfway in understanding that progress and pretty screenshots are NOT equivalent -- a mistake to which I've fallen prey for much of LT's development.

    The fact that development is not fun, pretty, or exciting (at least to most) is part of the very reason for which I haven't been excited to talk about it, and have even avoided doing so. For years I had beautiful things to throw up in the air and say "LOOK WOT I DID!" and receive praise for making things shiny. Man I miss those days so, so much...but the problems I'm dealing with currently are far more pressing to LT being released than more visual candy. I simply have to push through the problems I'm about to explain if you want LT to be finished. And I have every intention of LT being finished. There will be more pretties, more videos, etc., but let me be very clear: that won't happen before I get past the current roadblock.



~ Development: Where We've Been, Where We Are, Where We're Going ~

    To understand why development has taken the bizarre course that it has, I must take you back all the way to just before the Dark Days, when the 'Road to the Beta' was in full-swing and Josh was burning his candle at 17 different ends. Things seemed to be moving quickly and beta seemed within reach. Then, all of the sudden, the Dark Days were upon us. What exactly happened, other than stress and an unhealthy lifestyle pushing me over the brink? What happened is that I was backed into a corner from which there was no easy escape. I had been backing into this corner for quite some time without realizing it.

    LT was, at that time, a massive codebase. Not massive in relation to how much it did, but still massive with respect to your average programming project. With both C++ and GLSL included, the code was pushing dangerously-close to 100,000 lines. Interestingly, 100K lines is a bit of a special number in programming. It's one of those rule-of-thumb things, but a large number of programmers, now including me, have found it to be roughly accurate: they say that a skilled programmer can manage about a hundred thousand lines of code. What do I mean by that? Think of it like RAM: your computer has a certain amount of RAM. When a program tries to use more memory than you have RAM, typically the program will still function via an OS mechanism called 'paging,' wherein hard drive space is used to effectively extend the limits of RAM. Problem is, doing so comes at a huge cost (less so with SSDs, but still a high cost): RAM accesses now may take orders of magnitude longer due to needing to page in/out memory from/to disk. Sure, your program will still run, but the moment you hit your RAM limit, a typical program will almost immediately incur a huge performance penalty. And so it is with programmers! Good programmers have a 'RAM' of about 100K lines of code. Beyond that, the difficulty of understanding and maintaining said code quickly becomes disproportionately hard, leading to a harsh dropoff in productivity.

    So, I was starting to approach the limits of my programmer RAM. My solution to this problem (as well as modding) was LTSL. LTSL made things gloriously easy to understand, and, for a while, made development gloriously fast in comparison to how C++ development had been. There was a brief period of time wherein I found myself basking in the sun and all was right with the LTverse. LTSL was making development a joy, I was able to keep everything in my mental RAM thanks to the compactness of the language...things were great. Until, one day, they weren't.

    You see, while I had successfully evaded one wall, another was sneaking up on me, and I would soon be caught between a rock and a hard place. LTSL, like all non-compiled scripting languages, was slow. Not terribly slow, but certainly slow in comparison to C++. This was the price I had to pay for offloading gameplay code to LTSL. I didn't anticipate that the price would be too much. On that point, I was quite wrong. It was a rather sudden thing, due to the nature of Limit Theory -- there's so much going on at all times...so many game entities, so many UI components, so much AI, so much physics, so much rendering -- speed degradation of only a small bit of code can cause a frightening FPS hit, since that code may be running every frame on every entity that, just for example, emits an EM signature that can be picked up via scanner. As I pushed into my beautiful new world of C++ & LTSL, I was also pushing closer to the invisible wall of performance problems, and then, one day, I hit it: FPS was down to ~20 even in situations where a relatively small amount was going on (i.e., even without LOD system simulations active, without distant AI, etc.) I knew I was in trouble, because I knew it was just the tip of the iceberg, and that performance was going to quickly go to hell in a handbasket if I continued using LTSL, my saving grace. On the other hand, my inability to mentally manage and make progress on a near-100K LOC codebase in C++ awaited me if I were to turn back. In either direction, I faced a sharp drop-off, either of productivity or performance. I was in a truly impossible situation, and for once, my Crazy-Solution-Generator-3000 failed me. Cornered and out of options, with a community expecting an imminent RTB post, my kernel crashed. And it crashed hard.

    Therein lies the rub, the truly monumental challenge at the heart of building LT (a challenge so formidable that I've given it the title of The Fundamental Problem of Limit Theory; FPLT for short): the completion of Limit Theory demands the utmost efficiency in both run-time performance (a la C++), AND development cost (a la LTSL). These two interests, as most programmers know, are in oppisition to one another in the world of development. In general you can have one, not both. But, you see, with an enormous amount of gameplay logic to crunch on an enormous number of entities, and only one programmer with one mental RAM bank on which to make that all happen, Limit Theory requires both. Voila, FPLT.



~ Solving the Fundamental Problem of Limit Theory ~

    While I've worked on a lot of areas of LT over the past two years (since the Dark Days), >95% of my effort has been devoted to solving FPLT. This has made for a very, very frustrating two years as I've bashed my brain against the hardest problem of my programming career. I've explored numerous solutions. Each prospective solution requires a tremendous amount of effort to even get to a point where I have a simulation large enough to know whether or not said solution will be viable (i.e., I have to build a large-scale, component-based entity simulation with AI to know whether a solution is both compact and performant enough to be an answer to FPLT). So far, the answer has been 'no' to each. Despite all of my accumulated knowledge of code, performance, etc., it's still far too difficult for me to simply intuit whether the prospective solutions will be viable, hence why testing is the only option. It's been a slow, painful process of trial-and-error.

    I promised details, so let's talk about some of my concrete work over the past two years. I've seen posts implying that I haven't been working, and honestly, I can't let that rumor stand. I've been working entirely too hard to get no credit for it, so I'm finally ready and willing to pony up the gruesome details, even though it goes against my long tradition of feeding you all nothing but unicorns and cupcakes :) So here it is...the good, the bad, and of course, the ugly. Put your gloves on.

    First, naturally, there was an attempt to rescue the existing LT by amping up LTSL with a performance overhaul. A practical move at the time. It was a relatively fast failure. I optimized, I changed the way LTSL runs, I molded the memory access patterns from 'my RAM is crying' to 'it could run on punch cards!' Still, I could see that the numbers weren't going up. The gulch between the statically-compiled and intensely-optimized C++ and the runtime-compiled LTSL evaluation engine was too wide to make a dent. Despite my efforts, I'm a bit embarassed to admit that I wasn't ever able to improve LTSL's performance dramatically over baseline, even after blasting it with some of my best optimization cannons.

    With the most practical option shot down, I switched attack vectors and tried what seemed to me the next most practical: splitting the C++ into smaller bits to make it more manageable. I split up what was a 'monolithic' engine, compiled all into one library, into component pieces compiled into different libraries (libltcore, libltrender, libltui, libltai, etc.); this was an attempt to increase both runtime efficiency and my ability to keep all code in RAM. This attempt, luckily, only cost me a few months, as I quickly learned that code is code, and the programmer RAM limit applies no matter how you divvy up the lines. Another solution shot down, another lesson learned.

    I then moved on to a more dramatic change, recognizing that, clearly, the C++ core was simply too big. I was going to have to reduce. Thus ensued one of the most dramatic shifts in LT's development history, as I carefully plucked pieces from the existing codebase and brought them over to a new, 'minimal' C core library. The general idea was to do the really performance-critical stuff in C, do the rest in a still-fast but high-level language (side note: I say C because, by this point in my programming life, I had basically abandoned most C++ constructs, so moving completely to C was not a big issue -- C libraries are friendlier when it comes to linking with the outside world, which is what I was planning to do with respect to the high-level language). My language of choice? Python. It took quite some time, but I eventually engineered the salvaged 'minimal' core of LT, which was a very reasonable 20K lines, along with a cluster of (concise and easy-to-write) Python scripts that could leverage the LT core library for all of the performance-critical and gpu-related stuff. It was quite beautiful. Of course, I knew what I was getting in to: Python is slow. On the surface, it might not look too different from the old LTSL solution. But, unlike LTSL, Python has a massive ecosystem surrounding it. For performance, there's pypy, Cython, Psyco, Pyrex, etc. Much to my chagrin, none of them suited my needs (pypy looked promising, but simply didn't give a big enough speed boost). So I pursued a custom JITing solution. I wrote an intermediate-representation, an AST translator that took Python code to my IR, and an emitter that took my IR to C, which could them be JIT compiled in-memory and executed as machine code by the wonderful TCC. It was a beastly feat of code. In the end, it broke my heart when this solution also failed, not due to performance reasons, but because, as it turns out, compiling Python to a static language (or an IR intended for emission to static language) is indeed near-impossible. Too difficult for me, at least. My JIT worked for simple and moderately-complex logic, but it simply couldn't handle the 'meaty' gameplay logic of LT, as the problem of type inference and other fun things started to become intractable.

    On the bright side, not all was lost with the valiant Python attempt. I did lose quite some time (perhaps on the order of six months) to it. From it, however, I had gained the minimal core onto which I have continued to build (with minimalism!) to this day. The C core will be part of the solution to FPLT. I now have that piece of the puzzle. For the other piece, the 'driver' that will control the core, I'm still searching.

    That brings us up to the present. At this time, my current attack vector is twofold.

    I've been building a solution involving LuaJIT for several months now and am nearing the point where I can do a feasibility test. While I've learned to temper my expectations and wouldn't say that I'm confident that LuaJIT will be the answer, I can say that it's by far the fastest scripting solution I've found. My initial tests, indeed, the only reason I began pursuing a solution in Lua (which wasn't really on my radar), demonstrated that LJ achieves incredible speeds on limited-scope performance tests, even nearing the speed of my own, hand-written C. That's not terribly surprising, since LJ is a genuine JIT, meaning that it's effectively turning Lua into assembly/machine code. It seems to be darn good at doing so. That being said, we'll see how it goes with a full-blown test. I have my reservations about Lua being garbage collected, and I fear that memory access, if anything, may make LJ infeasible. Nonetheless, LJ itself is a formidable feat of engineering, and even if it doesn't end up solving the second piece of the FPLT puzzle, I have a feeling it may still contribute somehow to the final answer (this has happened with so many other 'failed' ideas throughout the course of LT that these days, I almost expect a failed attempt to end up being useful in some way in the future).

    On the other front, I've been building a solution that involves generating C++ via a meta-language based on Python. This would provide a means of performing a lot more of the game logic in ultra-fast compiled code without me having to write 100K lines. In order to get LT working in as few lines as I did previously, I had to use a HUGE amount of complex C++ trickery. While it meant far fewer lines to keep up with, it also made the code more complicated and, in a few cases, sub-optimal in terms of performance. With a meta-language, we can easily solve these problems: I don't care how much code is generated, I only care how much I have to write to generate it. So, for example, whereas setting up the component-based entities in the old C++ engine took several layers of advanced C++, making the component logic hard to understand (boy was this a nightmare with the 'market' component...), I can generate equivalent code in far fewer lines of simple Python. As a bonus, the generated code compiles and runs faster because it's not having to use trickery to reduce code size...I can get Python to spit out tens of thousands of lines in milliseconds, based on ~100 lines of my meta-language. This solution is only a few weeks old, and is sort of a 'nuclear' option in the event that the LuaJIT solution doesn't work out. I'm fairly sure that I can get this approach to work, because it's kind of brute-force. I'd rather be able to find a more elegant way, but if this is what it comes down to, so be it...above all else, I just want to get Limit Theory released at this point, and I'm willing to use whatever I have to use to get there. (By the way, this is totally differenet from the other solution involving Python. Whereas the idea there was to convert Python to machine code on-the-fly, the idea here is simply to use Python as an easier way to write / automate the writing of C++. Kind of like...writing the code...procedurally!)

    So there you have it...the past two years of my life. Has it been shiny? No. Has it been as exciting as the first two years of LT dev? No. But the hard truth is that Limit Theory is an exceptionally-ambitious project, especially considering that we can't just brute force the development with a massive team and all the standard big-budget amenities. Yes, I was young and optimistic when I called it 'easy.' My mistake :? LT is a hard problem. At the center of it all lies FPLT, which encapsulates the problem of needing to build something complex with minimal resources. It's a solveable problem; of this I'm certain. The solution? No, of this I'm not yet certain, but, as I hope to have demonstrated by laying all of my attempts out on the table in detail, I really am attacking it with everything I've got. I can't sugarcoat this situation as I've tried to do in the past, because it just leads to disappointment, so let me be perfectly honest: I don't know when I'll overcome FPLT. I sure as hell hope that it's soon, because two years is already an exceptionally-long time to be beating one's head against the same problem. What I do know is that every failure gets me closer. I learn more about what works and (perhaps mostly..) what doesn't with respect to performance in large-scale simulations. I learn more about low-level performance both of static and dynamic langauges, which helps me to evaluate future solutions. I become adept with more languages, more technologies, hence have an ever-expanding arsenal with which to blast away at development problems (FPLT can only take so much DPS before it breaks. It's a tough tank, but every month my DPS output goes up. It's only a matter of time...) In the mean time, I continue to develop the LT core C library as much as possible to ensure that when I do come to the right solution, all of the underlying machinery is in place for me to quickly and easily plop the existing LT functionality into the mold of said solution.

    Is anyone still listening? :shifty: :ghost:



~ Re-Establishing Communication with the LTVerse ~

    Clearly, something needs to be done about my communication habits. Here's the good news: a large part of what's been causing me to not want to communicate with you all is, as mentioned above, the rather unglamorous nature of my work. I wanted to come back to you guys with guns blazing, yelling "YESSS LOOK AT THIS! AND THIS!! AND BETA IS COMING UP!" But I'm ready to accept that I can't work that way anymore. Showman Josh and Perfectionist Josh are mostly dead at this point, and I think you'll all be pleased to hear that Pragmatic Josh is now in full effect. At this point, my mentality on any given day is 'find the shortest path between where I am and LT being released. Follow it.'

    So from this point forward, I will plan to use my Sunday evenings to write a dev log -- however brief, unglamorous, or unexciting it may be -- to inform you all of my work over the past week. I'm not yet sure whether I want to do this every week or every other week, so I'll have to start with it and just go with what feels best.

    Again, I'm trying very hard to not make the same mistakes that I've been making with communication and end up disappointing you guys. To that end, I'm promising a regular but modest communication, something that will ensure you all that I'm still working to get LT in your hands without requiring as much of my time and effort as the daily dev logs, the monthly videos, RTB, etc. With FPLT I'm taking battleship-sized steps, but with communication we need to start back with fighter-sized steps :)

    (Oh, and yes, this does count for my first log...I'm not going to post more of my life story today :P)



~ Gratuitous FAQ / Autointerview ~

    I thought it might be interesting, if a bit strange, to ask myself questions from the perspective of forumers and squeeze some answers out of myself. In the event that someone actually does ask one of these questions, now I've already provided an 'official' answer. Maybe you will find it enlightening. Or perhaps just strange. Either way, enjoy :ghost:

Q. Thanks for joining us Josh.
A. Thanks for having me, imaginary forum member. What can I answer for you?

Q. Well, I read your post. I honestly did. But all I really heard was 'code code code C++ Python code Lua code JIT beep boop mainframe.' So...what's going on with Limit Theory, you know, that GAME that I'd like to be playing right now?
A. Yeah...it frustrates me too. I don't enjoy working on the technical side of it all nearly as much as I used to; frankly I would much rather be tying off loose ends on the AI, zone control mechanics, simulation LODing, unstable wormholes mechanics, etc. But, as I explained, this problem must be solved before I can finish LT, and I would rather throw myself entirely into solving it than split my time between solving it and trying to write gameplay code when the platform for said code isn't yet a solved problem.

Q. So it sounds like you've made no gameplay progress then.
A. Not true, but it is true that I haven't made nearly as much gameplay progress as you would want from two years of work (see above: >95% effort dedicated to the big problem).

Q. Oh? So then, what part of the game has changed the most since we last saw it?
A. AI has progressed quite dramatically. The top-level AI action ('Action_PlayGame', heh) is much more fleshed-out than before, meaning AI players are capable of literally 'playing' Limit Theory to a greater capacity than before. There are still missing elements, mostly because of missing logic for actions that deal with higher-level concepts like war, research, and construction. There's still a good bit to do here, but, having written some AI now in both Python and Lua, I've gotta say that, if we can get a fast scripting solution to work, AI development is going to go much faster when resumed full-time!

Q. Cool. You mentioned some stuff in the post that kind of scared me and made me think you're basically re-writing LT. Is that the case?
A. Definitely not. I'm changing the foundation on which LT rests, but by no means have I torn down the building. As stated above, the 'core' pieces of LT, i.e., much of the actual 'engine,' is still present in the C library. GPU code is all present and doesn't require porting. As for gameplay code / stuff that used to be in LTSL, I've ported it enough times now to know that it's a quick task. I've said it before and I'll say it again: it's the ideas beneath the code that the majority of the time to come up with. Typing code is easy. Coming up with an idea that will make the code do something wonderful isn't. Very little is 'lost' in this process.

Q. Hrmmm....so then, you've got it rendering and such?
A. Yep, I've got it rendering under both the current attack vector (LuaJIT) and the older one (Python). It's not really a big task with the library in place.

Q. Sweet, so, screenies?
A. Don't...even...!

Q. But I... :3
A. It looks the same as what you've already seen. I haven't worked on graphics! Seriously, the graphics monkey is in the cage. Gameplay, yes. FPLT, very yes. Graphics, no. I literally haven't changed the LT graphics in two years. It's been hard for me to control myself like that, but yes, I'm ready to accept that the current graphics are absolutely good enough for release. LT 1.0 is going to look very much like the latest screenshots you've seen, except with the missing content filled in. No more 'new metal shader version 298.0'.

Q. Fair enough. I'm glad you managed to control the monkey. Speaking of monkies, when beta?
A. When I solve FPLT and steamroll the remaining gameplay content.

Q. Ugh. I hate this 'FPLT' thing. You know, now that I think about it, what if you just ... ? Wouldn't that work?
A. (Depending on nature of ...): No, sorry. / It's a good idea but I've tried it and it didn't work out. / Well, that's essentially what I'm trying to do. / Dear God, why didn't I think of that, you're a genius, FPLT IS SOLVE \o/ *hands over bank account while weeping tears of joy*

Q. Okay then...back on-topic, why did you desert the forums?
A. I honestly didn't mean for it to become a 'thing.' It snowballed out of control: I just wasn't on the forums for a while, not for any particular reason, and then one day it was like a 'Josh is gone' thing, at which point I could have chosen to say "No I'm not," which would have pretty much done the trick. But instead I chose to give in to anxiety and was all "oh no, I'm gone!? Well now I have to come back with fireworks! :?" I really regret making that decision and letting it stagnate for so long.

Q. You should, a lot of us pretty much lost faith in you.
A. I understand. To be honest I can't say that I deserve more. All I would ask is that you give LT the benefit of the doubt in the sense that when it comes out, judge it as Limit Theory the game, not as Limit Theory the product of a guy who let you down.

Q. We'll see. What about Kickstarter though? I mean it's been even longer in the dark for them...
A. It's been a general pattern throughout development that I post more frequently to the forums than to KS. Historically, I've only put up KS posts when I felt that I had 'very substantial' material to post. As per above with the whole 'not a glamorous era in development,' I felt that I'd be letting everyone down with a text-based post, even if it's the best way to bring everyone up to speed. Once again, a regrettable choice made by Showman Josh, but now attempting to be set right by Pragmatic Josh.

Q. I like the sound of this 'pragmatic' Josh guy. Maybe he can finally get us a game?
A. Yes.

Q. Again, we'll see. I'm almost ready to dip my toes back into the water, but the truth is, you seem pretty good with words, yet have let us down quite a few times with those words.
A. Which is exactly why I've not sugarcoated this situation. Did you get the same 'thrill' reading all this that you did when I released an update video? Most assuredly not. But what I'm good at is raw honesty, which sometimes makes it hard for me when I honestly believe in my capability to do things at which I end up failing, or when the situation is honestly not so great. But I think you all, at this point, would much rather hear not-so-fun truths than nothing at all, which are the only two choices I've got. Notice the lack of glitter, unicorns, and grandiose promises in this whole thing!

Q. I certainly noticed the lack of glitter. Speaking of which, why on earth did you not provide a TL;DR for what may be your most obnoxious wall of text ever?
A. I considered it. I even wrote a TL;DR. But I realized very quickly that doing so was completely counter-productive to the point of this whole post. I took a lot of time to try to explain the current state of Limit Theory development; to try to get everyone to understand precisely what I'm doing from day-to-day right now and why it's necessary. If someone truly wants to understand it, they need to read this post. If not, then 'LT is not kill' should suffice. I can't explain it all and make my explanation super-compact. I could do that back when everything could be explained with images or videos. I can't do that at the moment.

Q. Indeed. Well, I'm rather exhausted from reading, and it's apparent that you're not going to give me screenies, so I'll end it here. Thanks for the answers, and good luck with LT. I look forward to that (semi)weekly log.
A. Thanks for listening, imaginary forum member.


<3 Josh

Link to original: viewtopic.php?f=11&t=5659
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4261
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Wed Jan 25, 2017 4:35 pm

January 25th, 2017
Hey guys,

Excuse my brevity but I'm in an incredible flurry of trying to get things ready right now for...my upcoming trip to San Antonio this weekend for PAX South!

I wish I had announced this earlier so that those who really wanted to show up and come hang out and chat about LT / see a demo could do so, but I'm afraid that anxious Josh was nervous for a long time about whether or not he'd have anything to show. Quite frankly, what will be shown is still a bit up in the air, but I can tell you that Limit Theory will have a demo booth on the showroom floor! I'll be there Friday, Saturday, and Sunday for the whole shebang. I'm going with my fellow game makers who work here at the Tech Park.

So, this means a few things:

    1. If you happen to be nearby and really want to drop in, grab a pass now because I think they're almost gone (I think Saturday is already sold out). But even if you don't get a pass, if you're very close by I'd be happy to arrange a meet & greet. If the LT sign comes out half as good as it looked when I rendered it last week (in-engine), you'll be able to find me easily :squirrel:

    2. LT will have a playable thing as of this weekend. I'm in the depths of using my (imperfect) LuaJIT solution to build it right now, and things are going well despite me not having enough hours in the day (as always). I think this is good timing: a lot of you were pleased with the last post, but a lot were also cautious because I still didn't show anything. I don't think I can give more proof that I'm making progress and working hard than showing up at an expo and letting people demo it in person :)

    3. I say 'playable thing' because it won't be the full LT (obviously...otherwise we'd be in beta). What I'm cooking up is essentially a 'slice' of LT gameplay, intended to give PAX attendees a fun, fast way to get a glimpse of the LT feeling, see the procedural universe, etc. Besides not having LT finished, frankly, even if I did, I don't think it'd make for a good demo. Not the kind of thing you can just spend 5 minutes on :P But I am excited to be able to show you all the fruits of this endeavor. This is also going to put me at least a month or two ahead of schedule in terms of being able to evaluate whether the LuaJIT option is going to work. If it holds up well for this demo, it will be a good sign.

    4. I believe it was Flatfingers who said 'show me the smoldering corpse of perfectionist Josh.' :lol: Well sir, I'm going to put a controller in the hands of total strangers and let them play what will undoubtedly be a (pretty, but) highly imperfect demo. Perfectionist Josh wants to write a demo that lets them choose between like 10 different scenarios, each one showcasing a different piece of LT gameplay. Pragmatic Josh is gunning for one solid thing without too many bugs. After this weekend, I hope I will have presented you guys with this smoldering corpse :ghost:

    5. I'm extremely, extremely crunched from now until Sunday. This means I won't be available for communication, and, apologies to the pledge transfer people, I am going to have to deal with them next week rather than this week. Hope you can understand :)

    6. I definitely want to do a weekly log this week, because I'll have a lot to say after such an exciting week/end! But I will still be at PAX on Sunday, so expect a log early next week instead.

    7. Wish me luck. The code has been flowing very rapidly since last week, but I still need torrential amounts of it! Story of my life.

(Oh, and speaking of Pragmatic Josh, I'll be running on modern hardware just in case LJ gets slow on me. New laptop with i7 and 1060GTX (a very gracious Christmas present from my family, who, like you all, thought I was embarrassingly behind the times for a game developer..) who cares about performance! \o/ :ghost:)

Do let me know if you're already planning on going or are somehow going to be able to make last-minute arrangements to do so! I'd love to meet anyone who is going.

~Josh

Link to original: viewtopic.php?f=11&t=5674
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4261
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby Talvieno » Thu Feb 02, 2017 4:57 pm

Tuesday, January 31, 2017

I'm going to interrupt the stream of copy-pasted Josh-posts with a bit of an announcement. Two announcements, actually.

First, Josh made a post to the Dev Log forum - the first in over two years. Note that you can actually reply to this one, unlike the others in the past. Most of the post is a retrospective look at his stay at PAX South. I've put a copy of the post in the spoiler below. He should be updating it either weekly or bi-weekly; I'll continue posting copies of the updates here when we have them.
Spoiler:      SHOW
JoshParnell wrote:Tuesday, January 31, 2017

Good evening and welcome to the 'new' Limit Theory development logs! As per my promise to get back on the communication train, with this post I'll be kicking off the new logs (which may casually be referred to as The LT Fortnightlies), in hopes of ushering in a much-needed era of sane, stress-free, non-showmanly communication to you all. Note that, unlike the past devlogs, I am (at least tentatively) leaving these open to comments, until it proves to be a bad idea.

Now...that being said, it was a really busy and exciting week for LT, what with me attending PAX South in San Antonio with a 'kind of LT' demo. Due to the unusually-exciting week, my first fortnightly is going to be a lot longer (and perhaps more shiny) than the average one. Please do not take this first log as 'the standard,' because I will certainly not be attending something like PAX / creating a new LT demo every week. But for this week, I have a lot to write about and I want to, so I will :geek:

Last week was a ridiculously-busy flurry of code, electronic equipment, failures thereof, migraine-inducing bugs, triumphant successes, very little sleep (followed by far too much sleep), and, in the end, a lot of good feels. When I signed up to be a part of the Louisiana Tech Park's PAX group about three months ago, I had only one intention: to light a fire under myself to get something playable in a booth. At the time, that seemed impossibly far away. As of Sunday, it was reality :) Mission accomplished!



The Process

The demo that I showed off was, for the most part, built in about a week. More specifically, the bulk of it was built between last Tuesday and Sunday. As usual, I managed time poorly and would have had more features in the demo had I started earlier. Still, I don't think I'd change the way it played out, since the process proved to me a few things.

1. Perfectionist Josh is dead. There can honestly be no question about it. This puppy was rough. To even get it working in time, I had to muster duct tape and hacks like I've never mustered before. I cheated physics in ways that would have never even occurred to my little elegance-loving brain in the past. I cut my original grandiose goal down to the bone, literally re-organizing my 'critical features' priority list every hour or so as it came down to the wire, and slowly accepting that more and more things would not make it. Sounds bad, right? But in the end: there was something playable. :thumbup:

2. The current system of core C library + high-level scripting language (Lua / LuaJIT being the current flavor) works very well for building stuff quickly. Not surprising, since LTSL was similar. Still, it's nice to have confirmation on this front. I effectively built a small, arcade-ish space shooter in a little under a week. Huzzah, turns out I can still code real things!! \o/

3. (I will discuss this more later) My hopes that LuaJIT will yield the performance I need are cautiously growing. I was quite surprised at how well it held up.



The End Result

It took me until noon on Sunday (on Friday I alternated between a screenshot slideshow and a random nebula generator with a slowly-rotating camera; on Saturday I stayed in the hotel and coded all day because I couldn't handle not having something playable), but a playable demo finally arrived.

The demo was very simple, consisting of nothing more than a massive, free-for-all battle of 100 to 500 ships (I varied it through the day), many of which were hunting the player. The player had a small (box) ship, equipped with two pulse lasers. Controls were simple and included your standard LT-esque 6-DOF flight, shooting, switching between 1st/3rd person camera, (implemented later on in the day) a thrust booster, and (my favorite) a 'generate a new system' button (which, of course, included generating a new procedural nebula, starfield, configuration of ships, etc). As you can probably imagine, the original plans included way more (two warring factions, a 'home base', purchasable upgrades, etc...), but...*snip snip*

When I first arrived with the demo, it immediately became clear that it was too difficult. People couldn't hit ships at all. I guess I've become a bit too good at leading ships properly in order to hit them, and forgot that it's quite hard ;) So I pulled up gvim and, ten minutes later, aim assist was making the game all too easy as people simply held down the fire button and watched the enemies explode :lol: Five more minutes of gvim later, we had a 'partial' aim assist, which required people to aim relatively close before the targeting computer would help them. Finally, the difficulty was about right and people started to enjoy playing.

Interestingly, I think I garnered a lot more attention at the booth during those times when I was live-coding more features into the demo and gvim was up on the booth's screen :monkey: While I was implementing a boost feature later on, one guy told me he'd buy the game just because I knew what I was doing instead of 'copying and pasting from the internet.' Perhaps I should have just coded at the booth on Saturday instead of in the hotel :roll:

All-in-all, it was quite a simple demo, but it did have the taste of Limit Theory to it, and it was, in my estimation, quite a decent bit of fun for a relatively small amount of implementation time. Yesterday when I got back, my roommate asked me how the weekend went. I let him play the demo and he was surprisingly upset when he finally died, and told me that it got his adrenaline going (in fact, I think my roommate currently holds the high score for kills...), so that was nice to hear :D



The End Result (Meta)

While the concrete result of the process was the demo, I think the tertiary fruits of the week ended up being far more valuable than the demo itself:

1. The core C library gained a lot of features. To me this one of the most 'concrete' wins, because I consider that library to be the one permanent fixture of all this development. It's still unknown whether LuaJIT will pan out, therefore, whether the existing Lua code will be used in the final Limit Theory isn't known at this time. The C library, however, will be used, and it gained a lot of functionality this week. Loads of new matrix math routines, Mersenne Twister RNG implementation, heavily-expanded gamepad support & functionality, new fast noise library, some basic physics functionality (basic intersection tests, impact point calculations), and more.

2. I got a lot of inspiration and rekindled enthusiasm out of it. Truth be told, I've been down in the dungeons battling FPLT for so long that I've forgotten to come up for fresh air. When I first drove an ugly little box ship around a system again, it felt really good, good enough to inspire me for a moment or two. I've always had a problem with remembering to play and have fun with the GAME I'm making. I'm going to try to do a better job of that, because it's no fun feeling uninspired.

3. The LT community is now fully convinced that I'm intent on doing things differently for the better. Right? Right?? :angel:

4. I gained some important perspective on what matters. I know I've had trouble with this in the past, but again, I'm making a big effort to move to the land of pragmatism. During the development of the demo, I scrapped features that it almost killed me to scrap, out of pure necessity. I didn't have time to bring over a ship-generating algorithm, so my placeholder boxes never got replaced. I didn't have time to bring over my entire physics engine, so the player could fly right through other ships. These things killed me deep down. But how many people made an issue of them when playing? Zero. Instead, I heard things about targeting feeling clunky, movement not feeling 'tight' enough, wanting more weapons, etc. Throughout the day I tweaked these things with some live-coding to improve the experience, and it made all the difference. The few people that did figure out that they could fly through other ships didn't bat an eye, and just went on shooting. Granted, I'm not saying that I plan on ripping physics out of LT 1.0, or that I'm going to ship it with box ships. But it was an important reminder for me that I have a very skewed perspective as a developer, and that having people play the game and remark on what needs work is critical.

5. I got to meet some people who have been following LT (including a few forumers like Flatfingers, Skyfligher, Sebby McWester), and the experience of talking to them about LT and the things that made them excited made me really, really happy. I started Limit Theory to build my dream game. I'm still doing that, but as I've grown up a bit since the beginning, I think seeing the excitement and the smiles on other peoples' faces as they talk about LT gets me going more than it used to. All you guys who share my dream and have supported me in the journey to make it a reality are very much an inspiration to me at this point.



Moving Forward

Thanks to last week, I have a pretty strong feeling about where I need to go from here. As I mentioned, LuaJIT performed quite well the whole time. I never had issues with CPU bottlenecks, I seldom had issues with the GC (and was able to mitigate them substantially by tweaking the GC's settings). In the end, I was, frankly, amazed to be able to run 500 ships in a system with thousands of pulse lasers flying through the air at any given moment at a decent framerate. The game logic was simple compared to full LT logic, BUT not so simple that it can't be used as a rough measuring stick.

Before PAX, I was on the fence about LuaJIT. I was cautiously optimistic. After PAX, I am now even more on the fence, and even more cautiously optimistic. In other words, I have pushed much deeper in to doing things with it, and the situation with respect to whether or not it will work has remained unchanged. LuaJIT has thus made it farther than any of the previously-explored solutions.

This week, I intend to launch an intensive study into the performance characteristics of the PAX demo + features that I'll be adding in an attempt to get a better sense of what happens when the logic gets heavier. I basically need to prove that LuaJIT will work, or prove that it won't. That's my current goal: to hop off the fence.

To do so, I'm going to have to break out the measuring tools and do some serious profiling to figure out exactly where LJ is going to start hurting me. It's going to be tough, because JITs have a totally different set of performance characteristics from static languages, and I've been pretty off in my previous assessments concerning LJ and performance. I need to become an expert on LJ performance in the coming weeks.



Screenies!

Yep, I actually come bearing screenshots this week. Again, please don't be fooled by this first log: it was clearly an unusually exciting week!

LT @ PAX South 2017 Album

(Yes, my cellphone camera is awful)

Some of my favorites:

Image

Image

Image

Image


Second, and more importantly, Josh updated the Kickstarter page today. I won't copy-paste that, but you can feel free to go look at it yourself.


I'll continue posting updates here, but it's worth noting that they should be coming more regularly from now on. :)
Image
Have a question? Send me a PM!
User avatar
Talvieno
Community Manager
 
Posts: 8214
Joined: Fri Apr 11, 2014 6:50 pm
Location: North GA, USA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Mon Feb 06, 2017 12:14 pm

Monday, February 6, 2017

Whew, roller-coaster of a week. Ultimately, a pretty crushing ending, sad to say. As stated in the last devlog, my plan for the week was to allocate 100% of effort to exploring the performance characteristics, and, ultimately, feasibility of the current LuaJIT solution.

The tentative conclusion, after getting a finer handle on where I lose microseconds with LJ, was...LJ is really close to being feasible. With the current tests, it is performant enough. But don't get too excited yet. Despite injecting some more logic, my tests aren't yet representative of the full scale of LT simulation. As such, I would have considered passing tests with 'good headroom' as a green light for LJ. In the perfect escalation of tensions, LJ did pass, but not with much headroom. What I mean by 'headroom' is basically 'spare time.' Essentially, the current solution manages to 'just scrape by' the current tests, which puts me in a very nerve-wracking situation. LT will quite obviously be more intensive than my tests, which means, presumably, LuaJIT would not perform well (FYI, I was not running these tests on my older computer, I was running them on a rather powerful laptop, so that contributes to the nerve factor). On the other hand, I haven't explored all optimization routes yet -- I could push more logic into C, I could continue to learn more about how to squeeze the most from LJ, etc.

Well, some of you will be happy to know that my first thought (believe it or not) was threading. Despite how frequently I complain about the difficulty of threading game logic, I actually made a great design decision when building the PAX demo: unlike all of my previous work, I (somewhat impulsively) decided to implement a 'data-driven' design for the component-based entities. I think I did so initially to minimize function call overhead (with this design, I write update & draw functions that take lists of specific object types and perform the necessary logic on them; this is in contrast to simply having 'monolithic' loops that call update/draw functions on one entity at a time). I quickly realized this was a great decision for graphics optimization, as it allowed me to eliminate a lot of OpenGL state change calls. In fact, the simple PAX renderer (theoretically) runs substantially faster than the old C++ LT renderer. But, most importantly, this design ended up being critical to opening a path for a limited amount of game logic threading.

Now, in C(++), this would have been a fairly simple ordeal. But we're in LJ. Lua has no native support for threading, but it's designed such that there's nothing stopping you from running multiple interpreter instances on multiple threads. The real problem, however, is that each such instance has its own data -- there would be no direct 'sharing.' Shared memory is absolutely necessary to getting any performance gain out of threaded logic. The alternative (and the mechanism that existing Lua threading packages use) is to send data as necessary between the different interpreters via communication channels. Frankly this is a waste of time for most high-performance code. In my case, the time it would take to send even a basic snapshot of the state of some objects to another thread in order to have it process them would vastly outweigh the time it would have taken to simply perform the logic on one thread. I don't need to do a perf test to know that non-shared-memory multithreading is a waste of my time :P OTOH, Lua isn't built for shared-memory threading. Two different interpreter states cannot get at one another's data.

Anddd that's how my Friday was spent, striking out the 'cannot' and replacing it with 'should not try to' :lol: I devised a decent little hack to force the interpreters to share memory. I had no illusions about how dangerous this was. Nonetheless, I basically lept out of my chair with joy when a simple test actually showed my mechanism to be working, and showed threads smoothly sharing memory. It was fast. It was correct (no memory corruption). I built a threadpool utility and built some functions to help me control arbitrary amounts of worker threads splitting parallel code paths among themselves. Around midnight, I had finally put enough gears in place to try it out on the real thing: threading the logic for ships in the PAX demo.

And that's when everything caught on fire. :cry:

It took about five minutes to come to the definitive conclusion that shared-memory threading of any real complexity (e.g. accessing tables within tables) isn't possible in Lua. Best guess is that the garbage-collected architecture screws everything up. Instead of pointers, tables likely use state-relative indices, breaking any attempts to access memory from other interpreters. Whatever the case, I'm nearly certain that the GC is to blame. I can't possibly overstate how much I hate garbage collection.

All-in-all, a fairly heart-breaking week. Threading would have pushed me over the edge into the green -- I would have had a comfortable amount of headroom. Alas, 'twas not meant to be.

I'm not ready to give up on LJ yet. It has still come closer than any other solution. I will not discard it lightly. This week, I will delve even deeper into learning performance characteristics of LJ -- in particular, there is a built-in profiler that is (apparently) very good. I need to get that running so I can see what's going on and where my opportunities for buying time are. Catch is, to run the profiler I have to be running my program directly from LJ, not launching LJ from my program. A little finessing of the C engine will be necessary for me to convert it into a shared library, and then a little time will be required to make a Lua script that does the (basic) functionality that the C core did with respect to starting things up. Shouldn't be long before I can grab some nice profile results. After that? Who knows. I will try whatever I can. The goal is to buy headroom. The goal is to push LJ into the green.

I believe it can be done. Whether I can figure out how is another matter, and I suppose we'll all have to stay tuned to find out :geek:

---

PS ~ In case anyone was going to point it out, the 'thread' construct in Lua is deceptively-named, and has nothing to do with true hardware parallelism. It's not helpful in this quest, sadly. Also, LuaLanes, luasched, LuaTask, etc, etc etc etc...none of the existing packages for Lua threading implement the kind we need (shared-memory, preemptive). 'LuaThread' claims to be shared-memory and preemptive. Sadly it has mysteriously disappeared from the internet, and the only similarly-named impostors I can find do not implement true shared-memory/preemptive (likely because, as I found, it is not possible :ghost:)


Link to original: viewtopic.php?f=30&p=141771#p141769
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4261
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Sun Feb 12, 2017 6:33 pm

Sunday, February 12, 2017

55 89 e5

Very productive week, and, overall, quite a bit more satisfying than last week. The week was split straight down the middle, 50/50, as I will explain.

The first few days were dedicated, as expected, to continuing my analysis of LuaJIT performance, getting profiling tools in place, and performing profile-guided optimization to get a better feel for where the milliseconds were going. I'm pleased to report that this all went quite well!

LuaJIT's built-in profiler is indeed really nice. It didn't take much effort at all to set up, and the results helped me get a much better perspective on the existing code's performance. Unfortunately, the results of several profiling runs of the PAX demo weren't exactly what we like to see in performance optimization. Frequently, something like the '80/20' rule will apply in profiling and optimization: it's common to find that a small minority of the code is consuming a majority of the CPU time (hence the oft-abused quote "premature optimization is the root of all evil"). This is actually what we want to see, since it means concentrating effort on that small minority can yield big gains. Alas, it wasn't the case with the demo. The profiler reported a lot of very small percentages accross the entirety of the code, with the best 'bottlenecks' peaking at maybe ~5%. The meaning of such a result is basically: there's not much we can do to reap big performance gains, at least, not much we can do easily.

Still, the profiling was helpful enough for me to weed out the peaks that did exist (mostly by transferring them to the C library), resulting in a decent gain of somewhere between 25% and 40% perf. At the end of it all, my heavy-duty machine was able to run a 1000-ship battle at 60 fps with a millisecond or so to spare. Remember, of course, that not all logic is implemented, so this isn't representative of what you'll be able to do in LT, although it's encouraging. The goal for me has always been to allow ~100-ship-battles at decent framerates, and at this rate, I believe we'll be able to do that.

That result, among others, has convinced me that we're safe to move forward with LJ. I'm not able to hop off the fence and plant my feet as firmly as I'd like to in the "this will work" grass, but for now, what I've seen has allowed practical me to conclude that, with continued profiling and keeping a careful eye on the milliseconds, we will be able to proceed with LJ. This is also taking into consideration that, if things do start to get too heavy, I'll likely be able to make enough little scale cuts here and there to push out a still-high-quality LT 1.0. In other words, I really just wanted to remind you all that I'm keeping practicality in mind and am open to, for example, having to slightly lower the scale of system economies or out-of-system simulations, etc, if it means the difference between 60 and 10 fps :)

Quite exciting :D I'd be using more smileys and squirrels if I weren't so scared of being excited about solutions :P

---

Now, in complete contrast to the first half, the latter half of my week was spent hedging my bets. Several people have asked the question "what happens if LuaJIT falls through"? The answer is that I do have quite a few options remaining in my mental priority queue. I explained in my 'State of Limit Theory' post that, at that point, I was splitting my effort roughly 50/50 between LJ and a different solution involving code generation from script. Since then, the prior has escalated to taking most of my time, what with PAX and the excitement thereafter of having something I can play and iterate on. Now that I've decided to move forward cautiously with LuaJIT, I do intend to resume my 'hedging' efforts. Although I'll continue allocating the majority of my time to 'LT in LJ' in hopes that it'll pull through for us, I'll still be giving ~20-30% to R&D for the next potential solution in the priority queue. This week, in an attempt to spin up that next solution and give it some momentum, I gave it a one-time boost of about half my time.

So, currently sitting at second place in the queue is what I think of as a 'nuclear' option -- a sort of fail-safe, brute-force option. Indeed, it may be a bit scary that a nuclear option is next in line behind LJ, but Practical Josh is thinking of it this way: "I'm getting really tired of solutions failing; had I gone with a nuclear option in the first place, I'd be done by now." Flawed logic, because I didn't have the know-how to pursue this two years ago (again, the knowledge gained from those failed attempts is nontrivial!) But, now that the option is accessible to me, I intend to give it a low-priority JoshOS thread as my hedge against LJ failing, at least until something really promising displaces it in the queue.

Over the past few days I've worked hard to spin up this solution, and have already got quite some results to show for it. I built a working x86 assembler/linker (of course, not full-featured yet, but working as in 'capable of generating in-memory programs & functions using a restricted instruction set.' Just a few hours ago I had my first successful test run in which I used it to create some simple math functions at run-time. They worked! :) It was an intense few days of reading hundreds of pages of Intel's software developer guide to better understand CPU architecture and, more importantly, to understand enough to translate assembly to machine code (that's what an assembler does, among other things). Let me re-iterate since some of you are probably scared that I've lost my mind now: this is for plan Z, in the event that LJ fails (and fails hard enough for me to walk away from that substantial time investment) and no other solution manifests in the mean time. We should hope that it ends up being nothing more than a nice learning experience for me. Which, by the way, is very refreshing to have every now and then. My brain feels like it got to go on vacation during the second half of the week thanks to this work (yes, I love learning enough that reading Intel manuals and looking at opcodes in hex feels like a vacation). I was surprised at the relative ease of doing it -- I initially assumed we were looking at months until any results at all; turned out to be a few days.

This so-called nuclear option allows something along the lines of 'compiled LTSL,' i.e. LTSL running at nearly the same speed as the pre-compiled code. Some people have asked about the feasibility of such a solution before, since it seems like a somewhat-straightforward way of solving the original problem of having run into one kind of limit with C++ and another kind of limit with LTSL. I honestly didn't have the knowledge to do that kind of thing before. But, having spent so much time with intermediate representations (thanks, failed Python solution), JITs & asm (thanks LJ + failed Python solution), codegen (thanks, C++ codegen along with the 32 other metacompilers I've written in my life..), I now do. Best of all, I have a number of attack vectors for doing so (TCC, LLVM's IR/code generator), but the most appealing to me (for plan Z) is the solution involving the minimal number of intermediate pieces that could fail: my own direct-to-machine-code, in-memory compiler/assembler/linker. I already wrote an LTSL 'compiler' that takes it as far as an expression tree (essentially an executable AST), and now I've got the humble yet promising beginnings of the latter parts. It might sound like a monumental task, but the reality is that I only have to implement a tiny fraction of what 'general-purpose' tools do. In other words, when I first created LTSL and the compiler for it, I didn't worry about making it a feature-complete language, I worried about making it super easy to write ship generation algorithms, UI, gameplay code in. Similarly, for assembler/linker, I am not concerned with outputting executables, nor shared libs, I'm not concerned with PLTs and GOTs, nor many of the other complexities. I'm concerned with writing a relatively-small subset of ops to memory and executing them from a program that's already running (the LT core) in order to quickly evaluate things (like LTSL expression trees). When you cut the problem down to the core, it's a lot less scary. Still not easy, of course, but totally within the realm of feasibility.

(I know that I'll still catch flak from some people on this, despite stressing that it's plan Z and not by any means the focus of the coming weeks / months of development...but hey, I said I was going to be honest about the good, bad, and ugly :ghost:)

---

Finally, the other 10% ( :crazy: ) of my week was spent creating the beginnings of a benchmarking script utility to help me be more precise about my observations and decisions when it comes to all these different solutions and their respective performances. This was, in part, motivated by a simple test that I performed pitting C, unoptimized C (i.e. no compiler optimization), D, Lua (the standard interpreter), LuaJIT, and Python against one another in a small bit of code. In doing so (and finding a few interesting oddities along the way), I realized that, for someone who spends so much time thinking about a very hard problem related to perf, I've got a startling deficit of objective, quantitative data to back me on my calls. It's been a good week for quantitative measurement, what with the profiling runs and all. I decided that I need more of that in my life :lol:

It's a very simple little utility (written in Python!), but with a few more hours of work it'll help me record concrete information about relative performances in the face of many variables (language/solution, piece of code, machine, OS, CPU architecture, GPU, etc.) As I continue development, I plan to toss new benchmarking tests into the mix to help me stay abreast of FPLT in a precise way. Hopefully I'll be able to quote precise figures in the future instead of just "too slow, close but not quite, good enough, really fast." I'll also have a way to be less on-the-fence about things like LJ since I'll have hard data to say what is and what isn't working.

---

This coming week, I'm excited to say that we resume 'development as usual,' at least in some sense. The majority of time is to be devoted to implementing more LT in LJ rather than scrutinizing the existing code for performance, which should really prove to be a relief from walking on eggshells with a profiler in-hand! Of course, the minority is to be devoted to 'assembling' ( :roll: ) a nuclear warhead, so that's a rather fun contrast.

In all, I'd estimate a 100% chance of fun! :geek:

PS ~ I'm still working on the whole brevity thing when it comes to my logs. They look a lot smaller in full-screen vim and somehow become startlingly-long when I paste them into the forum's post editor :oops:

b8 39 05 00 00 5d c3



Link to original: viewtopic.php?f=30&t=5736



Note from Talvieno:
If you haven't seen it yet, you might want to check out Skyfligher's interview of Josh from PAX South.
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4261
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: The Limit Theory Email/RSS News Thread

Postby JoshParnell » Tue Feb 21, 2017 9:49 pm

Tuesday, February 21, 2017

Excellent week of work. Excellent in the sense of no sudden big problems, changes in course, crazy ideas that cause straying from existing courses, or agendas that obliterate all notions of course entirely, etc. Just your standard, solid week of good work. The kind that we need a couple <insert order-of-magnitude estimate here> more of and LT will be done :geek: This means we may finally see a reasonably-sized dev log! ( :ghost: )

---

On the LTLJ front (Limit Theory in LuaJIT; yes I like abbreviations): good week of getting more content brought over and sanitized. PlateMeshes are on the way back (the model format used to generate stations, ships, turrets, and most other artificial objects in LT), this time being promoted to the LT core. With them, all pm-based generators are on the way back as well (ship, station, and the various little hardpoint mesh generators). I'm interested in doing a better job with the implementation of this type of model than I did in LTC++, since it ended up being a really good, heavily-used piece of tech. When I wrote PlateMeshes in the original engine, it was more of a prototype that ended up seeing a huge amount of use. I never did replace the occlusion generator with something robust. Though I'll first port over the old solution, I'd like to upgrade it to a sub-quadratic algorithm...it was very slow to generate HQ meshes with occlusion in the old engine! Perhaps more importantly, I never did come up with a good texturing solution (by 'good' I mean nice, clean UV-mapping rather than doing a bunch of expensive pixel shader hacks to get a decent-looking plating pattern). PMs really should be quite trivial to UV-map, so I'll give it a shot once ships and stations are 100% back.

In other content news, planets are back and almost at 100%, including atmospheric scattering. The only thing lacking is the actual atmosphere mesh (which will take all of thirty more minutes). Asteroids will be back as soon as scalar field surface extraction makes it into the LT core, which, again, is more just a matter of me spending some hours on it than any kind of real intellectual work. The LTLJ renderer (a remarkably small piece of code compared to in LTC++) now respects opaque vs. blended passes.

In summary, a lot of stuff is coming back very quickly, as I promised it would, since the real work of thinking through problems is already done in many cases. It feels so nice to get to focus on pulling content back in. It's getting more and more fun to hop into the growing PAX demo and shoot at baddies :D It feels a lot more epic with a nice, big planet in the background. This time around, I'm not going to make the mistake of going for long periods without a playable version of LT. I intend to keep the PAX demo in working shape and updated with relatively-recent content, both so that I can continue to feel the inspiration of being able to play LT, and because I'll be pushing myself to go to more local tech events where I'll demo the game (I'll give you all a better heads-up next time if I happen to go to another major event, though I think PAX South will be the last for the foreseeable future; the other opportunities available to me are local to Baton Rouge or New Orleans).

Some more good news: this week I experienced one of the happiest development moments in memory when I got to see my universes in VR for the first time!! For Christmas I treated myself to an HTC Vive (with which I'm in love), and subsequently implemented VR support in the LT core. However, I had never taken the time to wire up the PAX demo to the VR functionality before this week. All I did was add support for rendering to a VR headset and using the headset's tracking information to allow looking around; I didn't actually spend time making controller mappings or making the demo appropriate for VR in any way -- I just wanted to see it. And whew, did it blow me away. (Allow me to be nostalgic for a moment?) It reminded me of why I became a graphics programmer: because I wanted to build worlds. I always dreamed of getting to build worlds and then explore them. When I was 17 I pledged to myself that I would stop dreaming about it and do what I could to make it a reality, which is when I started the long process of learning 3D programming (which I assumed, at the time, I'd never really be able to do). Around this time I also looked into the other component -- exploring the worlds. But the hardware just wasn't around back then. I saw how I might be able to build worlds with enough learning, but I didn't see how on earth I could learn enough to build the technology that would let me experience those worlds. Fast-forward less than a decade, and a bunch of really smart people have done exactly that. I never dreamed it would happen so soon...if you had told me back then that in eight years I'd be able to code planets, nebulae, stars, ships, AND that I'd have access to hardware that would let me SEE them (almost) as I see objects in reality...no way I would have believed it. But there I was, this week, floating above a planet, staring at nebulae and stars, watching the occasional pulse laser fly by my face. Switching to first-person and flying by big ships (even though they're just big blocks pending the transfer of the ship generating algorithm) is thrilling. Even the most trivial of experiences, like slowing down time and watching a pulse fly by, thinking "my god, that's just a big equation I wrote...and here it is, flying by my face..." can be rather amazing :squirrel:

Though the VR experience isn't the primary focus of LT, I'm planning to do a lot more visualization and testing in VR, simply for the inspiration factor. It's one thing to see my changes on a screen, but it's 10x more inspiring seeing it 'for real.' After two years of relatively-uninspiring work, I'll take all the motivational boosts I can get. This week's LTVR experiences were an unexpected but welcome dose of fire :geek:

---

On the nuclear front: not a lot of progress (I'm doing a good job of keeping it low-priority!) However, I did implement address patching for jmp, jcc, call, etc. This means that I can effectively use labels, which means I can implement function calls, conditionals, loops, etc...things involving code addresses that have to be patched-in during relocation. I tested this out by writing and emitting some basic loops. It was surprisingly-exciting to see the results, simply because it's a great feeling to see zero-overhead loops. When I say zero-overhead, I mean...I can literally take the # of loops executed per second, multiply by the number of ops in the loop's emitted machine code, and get a number in the rough ballpark of my processor's clock speed (to be perfectly honest I'm not sure if that math makes sense given my still-limited knowledge of micro-architecture...but...I do get numbers close to my clock speed... :3 ) Cool stuff! :monkey:

I also made it through chapters 7 & 8 of the Intel Manual (Vol 1), which I'm slowly digesting for fun and the betterment of my brain. 7 (programming with general-purpose instructions) was mostly review, although I was unaware of the string instructions like MOVS, LODS, STOS, etc. Interesting stuff to know. "A REP STOS instruction is the fastest way to initialize a large block of memory." Neat tidbit, always wondered if that kind of thing could be done in asm without a bulky loop. 8 (programming with the x87 FPU) was fairly new to me, and was a lot of enjoyable (and relevant) information. I never knew that the FPU used a stack-based register model..interesting. Also didn't know there was an instruction for loading a very-high-precision pi constant (FLDPI). It was also nice to solidify my understanding of FP precision, a common source of confusion and misinformation from what I've seen.

---

Next week will hopefully see more of the same: old content revitalized and brought back to life with renewed vigor in the LT core or the LJ code, more functionality in the PAX demo (I would especially like to see some more AI work, although it may be premature at the moment), more inspirational moments staring out into space with a Vive on my head, and more assembler functionality (I'd like to see basic FPU usage and machine code generation for some basic 'real-world' math, the likes of which would be used in LT tight loops).

Seems I failed again at a short log. Ah well, I guess this is just how things are now...I seem to enjoy my work and talking about it! I suppose it's better than the alternative, which has been the case in the recent past :)

:wave:


Link to original: viewtopic.php?t=5792&p=142644#p142644
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4261
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Next

Return to Scripting & Modding



Who is online

Users browsing this forum: No registered users and 2 guests