• First off, I've got to start by saying that this post is coming to you several months too late, but, as I'll discuss later, I'm taking serious steps to ensure that I change my communication habits. I'm sincerely sorry for not being on the forums for several months; I know, coming back and reading some of the intermittent posting, that it's taken a toll on many of you guys' outlook on the game. With this post, I aim to lift that outlook, but also to open up the hood and talk about why things are the way they are at the moment.
  
  Before we get down to it, a word about my previous updates and the community's response. If you're one of the many posters who has said something to this effect, know that I've heard you when you said "no more vagueness," "concrete details," "more than promises/empty words," etc. I got it, my previous updates were not satisfactory, but give me a chance to prove to you that I can, in fact, change On the other hand, a forewarning: to those who have demanded "pictures/videos or LT is kill," let me tell you up-front that you're going to be disappointed. I'll explain in detail why that's not what development is about right now, and this is the part where you all are going to have to meet me halfway in understanding that progress and pretty screenshots are NOT equivalent -- a mistake to which I've fallen prey for much of LT's development.
  
  The fact that development is not fun, pretty, or exciting (at least to most) is part of the very reason for which I haven't been excited to talk about it, and have even avoided doing so. For years I had beautiful things to throw up in the air and say "LOOK WOT I DID!" and receive praise for making things shiny. Man I miss those days so, so much...but the problems I'm dealing with currently are far more pressing to LT being released than more visual candy. I simply have to push through the problems I'm about to explain if you want LT to be finished. And I have every intention of LT being finished. There will be more pretties, more videos, etc., but let me be very clear: that won't happen before I get past the current roadblock.

• To understand why development has taken the bizarre course that it has, I must take you back all the way to just before the Dark Days, when the 'Road to the Beta' was in full-swing and Josh was burning his candle at 17 different ends. Things seemed to be moving quickly and beta seemed within reach. Then, all of the sudden, the Dark Days were upon us. What exactly happened, other than stress and an unhealthy lifestyle pushing me over the brink? What happened is that I was backed into a corner from which there was no easy escape. I had been backing into this corner for quite some time without realizing it.
  
  LT was, at that time, a massive codebase. Not massive in relation to how much it did, but still massive with respect to your average programming project. With both C++ and GLSL included, the code was pushing dangerously-close to 100,000 lines. Interestingly, 100K lines is a bit of a special number in programming. It's one of those rule-of-thumb things, but a large number of programmers, now including me, have found it to be roughly accurate: they say that a skilled programmer can manage about a hundred thousand lines of code. What do I mean by that? Think of it like RAM: your computer has a certain amount of RAM. When a program tries to use more memory than you have RAM, typically the program will still function via an OS mechanism called 'paging,' wherein hard drive space is used to effectively extend the limits of RAM. Problem is, doing so comes at a huge cost (less so with SSDs, but still a high cost): RAM accesses now may take orders of magnitude longer due to needing to page in/out memory from/to disk. Sure, your program will still run, but the moment you hit your RAM limit, a typical program will almost immediately incur a huge performance penalty. And so it is with programmers! Good programmers have a 'RAM' of about 100K lines of code. Beyond that, the difficulty of understanding and maintaining said code quickly becomes disproportionately hard, leading to a harsh dropoff in productivity.
  
  So, I was starting to approach the limits of my programmer RAM. My solution to this problem (as well as modding) was LTSL. LTSL made things gloriously easy to understand, and, for a while, made development gloriously fast in comparison to how C++ development had been. There was a brief period of time wherein I found myself basking in the sun and all was right with the LTverse. LTSL was making development a joy, I was able to keep everything in my mental RAM thanks to the compactness of the language...things were great. Until, one day, they weren't.
  
  You see, while I had successfully evaded one wall, another was sneaking up on me, and I would soon be caught between a rock and a hard place. LTSL, like all non-compiled scripting languages, was slow. Not terribly slow, but certainly slow in comparison to C++. This was the price I had to pay for offloading gameplay code to LTSL. I didn't anticipate that the price would be too much. On that point, I was quite wrong. It was a rather sudden thing, due to the nature of Limit Theory -- there's so much going on at all times...so many game entities, so many UI components, so much AI, so much physics, so much rendering -- speed degradation of only a small bit of code can cause a frightening FPS hit, since that code may be running every frame on every entity that, just for example, emits an EM signature that can be picked up via scanner. As I pushed into my beautiful new world of C++ & LTSL, I was also pushing closer to the invisible wall of performance problems, and then, one day, I hit it: FPS was down to ~20 even in situations where a relatively small amount was going on (i.e., even without LOD system simulations active, without distant AI, etc.) I knew I was in trouble, because I knew it was just the tip of the iceberg, and that performance was going to quickly go to hell in a handbasket if I continued using LTSL, my saving grace. On the other hand, my inability to mentally manage and make progress on a near-100K LOC codebase in C++ awaited me if I were to turn back. In either direction, I faced a sharp drop-off, either of productivity or performance. I was in a truly impossible situation, and for once, my Crazy-Solution-Generator-3000 failed me. Cornered and out of options, with a community expecting an imminent RTB post, my kernel crashed. And it crashed hard.
  
  Therein lies the rub, the truly monumental challenge at the heart of building LT (a challenge so formidable that I've given it the title of The Fundamental Problem of Limit Theory; FPLT for short): the completion of Limit Theory demands the utmost efficiency in both run-time performance (a la C++), AND development cost (a la LTSL). These two interests, as most programmers know, are in oppisition to one another in the world of development. In general you can have one, not both. But, you see, with an enormous amount of gameplay logic to crunch on an enormous number of entities, and only one programmer with one mental RAM bank on which to make that all happen, Limit Theory requires both. Voila, FPLT.

• While I've worked on a lot of areas of LT over the past two years (since the Dark Days), >95% of my effort has been devoted to solving FPLT. This has made for a very, very frustrating two years as I've bashed my brain against the hardest problem of my programming career. I've explored numerous solutions. Each prospective solution requires a tremendous amount of effort to even get to a point where I have a simulation large enough to know whether or not said solution will be viable (i.e., I have to build a large-scale, component-based entity simulation with AI to know whether a solution is both compact and performant enough to be an answer to FPLT). So far, the answer has been 'no' to each. Despite all of my accumulated knowledge of code, performance, etc., it's still far too difficult for me to simply intuit whether the prospective solutions will be viable, hence why testing is the only option. It's been a slow, painful process of trial-and-error.
  
  I promised details, so let's talk about some of my concrete work over the past two years. I've seen posts implying that I haven't been working, and honestly, I can't let that rumor stand. I've been working entirely too hard to get no credit for it, so I'm finally ready and willing to pony up the gruesome details, even though it goes against my long tradition of feeding you all nothing but unicorns and cupcakes So here it is...the good, the bad, and of course, the ugly. Put your gloves on.
  
  First, naturally, there was an attempt to rescue the existing LT by amping up LTSL with a performance overhaul. A practical move at the time. It was a relatively fast failure. I optimized, I changed the way LTSL runs, I molded the memory access patterns from 'my RAM is crying' to 'it could run on punch cards!' Still, I could see that the numbers weren't going up. The gulch between the statically-compiled and intensely-optimized C++ and the runtime-compiled LTSL evaluation engine was too wide to make a dent. Despite my efforts, I'm a bit embarassed to admit that I wasn't ever able to improve LTSL's performance dramatically over baseline, even after blasting it with some of my best optimization cannons.
  
  With the most practical option shot down, I switched attack vectors and tried what seemed to me the next most practical: splitting the C++ into smaller bits to make it more manageable. I split up what was a 'monolithic' engine, compiled all into one library, into component pieces compiled into different libraries (libltcore, libltrender, libltui, libltai, etc.); this was an attempt to increase both runtime efficiency and my ability to keep all code in RAM. This attempt, luckily, only cost me a few months, as I quickly learned that code is code, and the programmer RAM limit applies no matter how you divvy up the lines. Another solution shot down, another lesson learned.
  
  I then moved on to a more dramatic change, recognizing that, clearly, the C++ core was simply too big. I was going to have to reduce. Thus ensued one of the most dramatic shifts in LT's development history, as I carefully plucked pieces from the existing codebase and brought them over to a new, 'minimal' C core library. The general idea was to do the really performance-critical stuff in C, do the rest in a still-fast but high-level language (side note: I say C because, by this point in my programming life, I had basically abandoned most C++ constructs, so moving completely to C was not a big issue -- C libraries are friendlier when it comes to linking with the outside world, which is what I was planning to do with respect to the high-level language). My language of choice? Python. It took quite some time, but I eventually engineered the salvaged 'minimal' core of LT, which was a very reasonable 20K lines, along with a cluster of (concise and easy-to-write) Python scripts that could leverage the LT core library for all of the performance-critical and gpu-related stuff. It was quite beautiful. Of course, I knew what I was getting in to: Python is slow. On the surface, it might not look too different from the old LTSL solution. But, unlike LTSL, Python has a massive ecosystem surrounding it. For performance, there's pypy, Cython, Psyco, Pyrex, etc. Much to my chagrin, none of them suited my needs (pypy looked promising, but simply didn't give a big enough speed boost). So I pursued a custom JITing solution. I wrote an intermediate-representation, an AST translator that took Python code to my IR, and an emitter that took my IR to C, which could them be JIT compiled in-memory and executed as machine code by the wonderful TCC. It was a beastly feat of code. In the end, it broke my heart when this solution also failed, not due to performance reasons, but because, as it turns out, compiling Python to a static language (or an IR intended for emission to static language) is indeed near-impossible. Too difficult for me, at least. My JIT worked for simple and moderately-complex logic, but it simply couldn't handle the 'meaty' gameplay logic of LT, as the problem of type inference and other fun things started to become intractable.
  
  On the bright side, not all was lost with the valiant Python attempt. I did lose quite some time (perhaps on the order of six months) to it. From it, however, I had gained the minimal core onto which I have continued to build (with minimalism!) to this day. The C core will be part of the solution to FPLT. I now have that piece of the puzzle. For the other piece, the 'driver' that will control the core, I'm still searching.
  
  That brings us up to the present. At this time, my current attack vector is twofold.
  
  I've been building a solution involving LuaJIT for several months now and am nearing the point where I can do a feasibility test. While I've learned to temper my expectations and wouldn't say that I'm confident that LuaJIT will be the answer, I can say that it's by far the fastest scripting solution I've found. My initial tests, indeed, the only reason I began pursuing a solution in Lua (which wasn't really on my radar), demonstrated that LJ achieves incredible speeds on limited-scope performance tests, even nearing the speed of my own, hand-written C. That's not terribly surprising, since LJ is a genuine JIT, meaning that it's effectively turning Lua into assembly/machine code. It seems to be darn good at doing so. That being said, we'll see how it goes with a full-blown test. I have my reservations about Lua being garbage collected, and I fear that memory access, if anything, may make LJ infeasible. Nonetheless, LJ itself is a formidable feat of engineering, and even if it doesn't end up solving the second piece of the FPLT puzzle, I have a feeling it may still contribute somehow to the final answer (this has happened with so many other 'failed' ideas throughout the course of LT that these days, I almost expect a failed attempt to end up being useful in some way in the future).
  
  On the other front, I've been building a solution that involves generating C++ via a meta-language based on Python. This would provide a means of performing a lot more of the game logic in ultra-fast compiled code without me having to write 100K lines. In order to get LT working in as few lines as I did previously, I had to use a HUGE amount of complex C++ trickery. While it meant far fewer lines to keep up with, it also made the code more complicated and, in a few cases, sub-optimal in terms of performance. With a meta-language, we can easily solve these problems: I don't care how much code is generated, I only care how much I have to write to generate it. So, for example, whereas setting up the component-based entities in the old C++ engine took several layers of advanced C++, making the component logic hard to understand (boy was this a nightmare with the 'market' component...), I can generate equivalent code in far fewer lines of simple Python. As a bonus, the generated code compiles and runs faster because it's not having to use trickery to reduce code size...I can get Python to spit out tens of thousands of lines in milliseconds, based on ~100 lines of my meta-language. This solution is only a few weeks old, and is sort of a 'nuclear' option in the event that the LuaJIT solution doesn't work out. I'm fairly sure that I can get this approach to work, because it's kind of brute-force. I'd rather be able to find a more elegant way, but if this is what it comes down to, so be it...above all else, I just want to get Limit Theory released at this point, and I'm willing to use whatever I have to use to get there. (By the way, this is totally differenet from the other solution involving Python. Whereas the idea there was to convert Python to machine code on-the-fly, the idea here is simply to use Python as an easier way to write / automate the writing of C++. Kind of like...writing the code...procedurally!)
  
  So there you have it...the past two years of my life. Has it been shiny? No. Has it been as exciting as the first two years of LT dev? No. But the hard truth is that Limit Theory is an exceptionally-ambitious project, especially considering that we can't just brute force the development with a massive team and all the standard big-budget amenities. Yes, I was young and optimistic when I called it 'easy.' My mistake LT is a hard problem. At the center of it all lies FPLT, which encapsulates the problem of needing to build something complex with minimal resources. It's a solveable problem; of this I'm certain. The solution? No, of this I'm not yet certain, but, as I hope to have demonstrated by laying all of my attempts out on the table in detail, I really am attacking it with everything I've got. I can't sugarcoat this situation as I've tried to do in the past, because it just leads to disappointment, so let me be perfectly honest: I don't know when I'll overcome FPLT. I sure as hell hope that it's soon, because two years is already an exceptionally-long time to be beating one's head against the same problem. What I do know is that every failure gets me closer. I learn more about what works and (perhaps mostly..) what doesn't with respect to performance in large-scale simulations. I learn more about low-level performance both of static and dynamic langauges, which helps me to evaluate future solutions. I become adept with more languages, more technologies, hence have an ever-expanding arsenal with which to blast away at development problems (FPLT can only take so much DPS before it breaks. It's a tough tank, but every month my DPS output goes up. It's only a matter of time...) In the mean time, I continue to develop the LT core C library as much as possible to ensure that when I do come to the right solution, all of the underlying machinery is in place for me to quickly and easily plop the existing LT functionality into the mold of said solution.
  
  Is anyone still listening?

• Clearly, something needs to be done about my communication habits. Here's the good news: a large part of what's been causing me to not want to communicate with you all is, as mentioned above, the rather unglamorous nature of my work. I wanted to come back to you guys with guns blazing, yelling "YESSS LOOK AT THIS! AND THIS!! AND BETA IS COMING UP!" But I'm ready to accept that I can't work that way anymore. Showman Josh and Perfectionist Josh are mostly dead at this point, and I think you'll all be pleased to hear that Pragmatic Josh is now in full effect. At this point, my mentality on any given day is 'find the shortest path between where I am and LT being released. Follow it.'
  
  So from this point forward, I will plan to use my Sunday evenings to write a dev log -- however brief, unglamorous, or unexciting it may be -- to inform you all of my work over the past week. I'm not yet sure whether I want to do this every week or every other week, so I'll have to start with it and just go with what feels best.
  
  Again, I'm trying very hard to not make the same mistakes that I've been making with communication and end up disappointing you guys. To that end, I'm promising a regular but modest communication, something that will ensure you all that I'm still working to get LT in your hands without requiring as much of my time and effort as the daily dev logs, the monthly videos, RTB, etc. With FPLT I'm taking battleship-sized steps, but with communication we need to start back with fighter-sized steps
  
  (Oh, and yes, this does count for my first log...I'm not going to post more of my life story today )

• I thought it might be interesting, if a bit strange, to ask myself questions from the perspective of forumers and squeeze some answers out of myself. In the event that someone actually does ask one of these questions, now I've already provided an 'official' answer. Maybe you will find it enlightening. Or perhaps just strange. Either way, enjoy

Q. Thanks for joining us Josh.
A. Thanks for having me, imaginary forum member. What can I answer for you?

Q. Well, I read your post. I honestly did. But all I really heard was 'code code code C++ Python code Lua code JIT beep boop mainframe.' So...what's going on with Limit Theory, you know, that GAME that I'd like to be playing right now?
A. Yeah...it frustrates me too. I don't enjoy working on the technical side of it all nearly as much as I used to; frankly I would much rather be tying off loose ends on the AI, zone control mechanics, simulation LODing, unstable wormholes mechanics, etc. But, as I explained, this problem must be solved before I can finish LT, and I would rather throw myself entirely into solving it than split my time between solving it and trying to write gameplay code when the platform for said code isn't yet a solved problem.

Q. So it sounds like you've made no gameplay progress then.
A. Not true, but it is true that I haven't made nearly as much gameplay progress as you would want from two years of work (see above: >95% effort dedicated to the big problem).

Q. Oh? So then, what part of the game has changed the most since we last saw it?
A. AI has progressed quite dramatically. The top-level AI action ('Action_PlayGame', heh) is much more fleshed-out than before, meaning AI players are capable of literally 'playing' Limit Theory to a greater capacity than before. There are still missing elements, mostly because of missing logic for actions that deal with higher-level concepts like war, research, and construction. There's still a good bit to do here, but, having written some AI now in both Python and Lua, I've gotta say that, if we can get a fast scripting solution to work, AI development is going to go much faster when resumed full-time!

Q. Cool. You mentioned some stuff in the post that kind of scared me and made me think you're basically re-writing LT. Is that the case?
A. Definitely not. I'm changing the foundation on which LT rests, but by no means have I torn down the building. As stated above, the 'core' pieces of LT, i.e., much of the actual 'engine,' is still present in the C library. GPU code is all present and doesn't require porting. As for gameplay code / stuff that used to be in LTSL, I've ported it enough times now to know that it's a quick task. I've said it before and I'll say it again: it's the ideas beneath the code that the majority of the time to come up with. Typing code is easy. Coming up with an idea that will make the code do something wonderful isn't. Very little is 'lost' in this process.

Q. Hrmmm....so then, you've got it rendering and such?
A. Yep, I've got it rendering under both the current attack vector (LuaJIT) and the older one (Python). It's not really a big task with the library in place.

Q. Sweet, so, screenies?
A. Don't...even...!

Q. But I... :3
A. It looks the same as what you've already seen. I haven't worked on graphics! Seriously, the graphics monkey is in the cage. Gameplay, yes. FPLT, very yes. Graphics, no. I literally haven't changed the LT graphics in two years. It's been hard for me to control myself like that, but yes, I'm ready to accept that the current graphics are absolutely good enough for release. LT 1.0 is going to look very much like the latest screenshots you've seen, except with the missing content filled in. No more 'new metal shader version 298.0'.

Q. Fair enough. I'm glad you managed to control the monkey. Speaking of monkies, when beta?
A. When I solve FPLT and steamroll the remaining gameplay content.

Q. Ugh. I hate this 'FPLT' thing. You know, now that I think about it, what if you just ... ? Wouldn't that work?
A. (Depending on nature of ...): No, sorry. / It's a good idea but I've tried it and it didn't work out. / Well, that's essentially what I'm trying to do. / Dear God, why didn't I think of that, you're a genius, FPLT IS SOLVE \o/ *hands over bank account while weeping tears of joy*

Q. Okay then...back on-topic, why did you desert the forums?
A. I honestly didn't mean for it to become a 'thing.' It snowballed out of control: I just wasn't on the forums for a while, not for any particular reason, and then one day it was like a 'Josh is gone' thing, at which point I could have chosen to say "No I'm not," which would have pretty much done the trick. But instead I chose to give in to anxiety and was all "oh no, I'm gone!? Well now I have to come back with fireworks! " I really regret making that decision and letting it stagnate for so long.

Q. You should, a lot of us pretty much lost faith in you.
A. I understand. To be honest I can't say that I deserve more. All I would ask is that you give LT the benefit of the doubt in the sense that when it comes out, judge it as Limit Theory the game, not as Limit Theory the product of a guy who let you down.

Q. We'll see. What about Kickstarter though? I mean it's been even longer in the dark for them...
A. It's been a general pattern throughout development that I post more frequently to the forums than to KS. Historically, I've only put up KS posts when I felt that I had 'very substantial' material to post. As per above with the whole 'not a glamorous era in development,' I felt that I'd be letting everyone down with a text-based post, even if it's the best way to bring everyone up to speed. Once again, a regrettable choice made by Showman Josh, but now attempting to be set right by Pragmatic Josh.

Q. I like the sound of this 'pragmatic' Josh guy. Maybe he can finally get us a game?
A. Yes.

Q. Again, we'll see. I'm almost ready to dip my toes back into the water, but the truth is, you seem pretty good with words, yet have let us down quite a few times with those words.
A. Which is exactly why I've not sugarcoated this situation. Did you get the same 'thrill' reading all this that you did when I released an update video? Most assuredly not. But what I'm good at is raw honesty, which sometimes makes it hard for me when I honestly believe in my capability to do things at which I end up failing, or when the situation is honestly not so great. But I think you all, at this point, would much rather hear not-so-fun truths than nothing at all, which are the only two choices I've got. Notice the lack of glitter, unicorns, and grandiose promises in this whole thing!

Q. I certainly noticed the lack of glitter. Speaking of which, why on earth did you not provide a TL;DR for what may be your most obnoxious wall of text ever?
A. I considered it. I even wrote a TL;DR. But I realized very quickly that doing so was completely counter-productive to the point of this whole post. I took a lot of time to try to explain the current state of Limit Theory development; to try to get everyone to understand precisely what I'm doing from day-to-day right now and why it's necessary. If someone truly wants to understand it, they need to read this post. If not, then 'LT is not kill' should suffice. I can't explain it all and make my explanation super-compact. I could do that back when everything could be explained with images or videos. I can't do that at the moment.

Q. Indeed. Well, I'm rather exhausted from reading, and it's apparent that you're not going to give me screenies, so I'll end it here. Thanks for the answers, and good luck with LT. I look forward to that (semi)weekly log.
A. Thanks for listening, imaginary forum member.

JoshParnell wrote:Did you get the same 'thrill' reading all this that you did when I released an update video?

Flatfingers wrote:
4. FIREWORK POSTS ARE NEVER REQUIRED. SIMPLE IS ENOUGH.

Please print out the above statement in a suitably large typeface and pin it immediately above your main working computer.

Seriously, how can we help you understand that this is the truth, for both Kickstarter and the forums?

0111narwhalz wrote:

JoshParnell wrote:Did you get the same 'thrill' reading all this that you did when I released an update video?

...
Maybe?

Fatfingers wrote:I sincerely ask you to please let us know what things we out here can do to help you succeed. For all the complaints, justified or not, you -- LT and you personally -- still have fans. If there's some way that we can help you, all you ever have to do is ask.