Sunday, June 9, 2013

Summary

Great day, but not particularly coherent. One of the more exciting things I did was build a small console program designed to run the bare-bones universe simulation, stripped of all graphics / user interaction code. Not only is it a good test to ensure that my engine architecture has properly separated the graphics and simulation to the extent that it should, but it will also be invaluable in testing my simulation code. With only the simulation running, I'll be able to achieve simulations that run way faster than real-time, meaning I can watch the universe evolve in hyper speed to quickly see the impacts of my changes. Some preliminary measurement using this new program shows that I can currently simulate at about 50x real-time (so in one minute I can simulate almost an hour of game time). At the beginning of the day that number was 20x, so you can tell that I did some nice simulation optimization today

I think my target will be 100x. That should be enough to quickly test my changes. Imagine, I'd be able to simulate more than 4 days of solid game time in a single hour! Mind you, that's full-precision simulation. In other words, it's exactly the same as if you left the normal game on for 4 days. I could watch factions rise and fall, planets change ownership, AI players work from rags to riches...and others work themselves into the grave. All in a few hours or so. Neat

There was a lot of other good work today, but since it was so scattered, I can't remember it very well, and I'd rather leave some room to explain the final change that happened today...something that is exciting, relieving, and scary all at the same time!

As you know, I have recently become a significantly more multiplatform kind of fellow What you may not know is that Dell is currently repairing my laptop, and I also recently purchased an AMD machine which should be arriving soon (no more breaking compatibility with AMD cards ever again!) which will yield a grand total of 3 development machines. And what you may have guessed, but not necessarily known, is that the current hourly logging system is kind of a major pain, both in that it does not support other platforms (forcing me to make educated guesses), nor does it allow easy usage across multiple machines (there is no automatic merging of the hours...so I have had to do it myself in the past when I split time between my laptop/desktop). Finally, what you definitely know is that, when we have a problem around here ... we rip it out

Today I introduce a new logging system! This new one, called "RescueTime," works on all platforms. It also automatically merges data between all of your computers. As if this wasn't already enough reason to switch, it also generates embedded charts automatically, with zero interaction from me!!! Um, where do I sign up!?!?! Just kidding. I already did.

So, the bottom line is that you won't be seeing hourly tallies in the dev logs anymore, but you can visit:

devtime.ltheory.com

In some ways it gives you more detail than the old logs, and in some ways less. It's a compromise, but it provides me with so many benefits that I can't pass it up. Now I can just go about my business and these charts update automatically 24/7 without taking any time away from development!! Awesome It's also neat how it rates your efficiency based on application usage. So driving that number up is going to be fun. Off to a pretty good start to the week at 80%

Monday, June 10, 2013

Massively intense day. Ended up with something that is hardly even the same engine that is was yesterday! That can only be a good thing

Today I went to war with hordes of enemies. The primary target was the serialization engine. I had somewhat of a revelation involving templates, late binding, boost's usage of operator & for archives, and life in general. The first three led me to a completely redesigned load/save system that is cleaner, faster, and more flexible than ever before. Nothing like shiny new code Although I won't dedicate an entire story to it, the war was truly epic. The compiler desperately did not want to let this system happen...at one point it even started fighting back with this sort of nonsense:
No joke. Compiler got so angry with me that it told the program to run in circles forever, getting absolutely nothing done. Man. Never seen that one before But in the end I was able to calm it down to the extent that it finally started emitting some sensible assembly.

Let's see...I also re-wrote the scheduling engine to unify CPU / GPU / synchronous / asynchronous jobs, so that they're all the same thing now (except that, of course, you can choose to run them asynchronously). Furthermore, they plug right into the function / functional metaprogramming garbage that I've been going on about this week, so the job interface is totally generic now (meaning: less code ).

But that wasn't enough for a day! So I went on to implement a portable backtracer that I can now use to recover a bit more useful information on crashes, as well as help me during debugging sessions. It is nice and pretty, and I can also annotate it so that I can tell exactly what is going on at each level of the stack.

Aaand then went on to implement a memory validation class that I can use in the program to help uncover memory access problems. I'm having trouble with a few crashy things at the moment and feel like I could use some automated help!

Ah, and I rewrote the in-engine profiler to yield much higher performance, so that it is minimally invasive (yielding more accurate profiling results). Really happy with it now, as it captures entire stack frames with high accuracy and high speed. Not a bad little tool.

Probably missed a few things ...but...you get the picture...it was a really good day!

PS ~ Changed the second devtime chart to show application category instead of application title...not quite so "Big Brothery" anymore.

[ You can visit devtime.ltheory.com for more detailed information on today's work. ]

Tuesday, June 11, 2013

Not gonna lie, today wasn't all that I wanted it to be The code just wasn't flowing in the right direction and the compiler was even less happy with me today than yesterday. I spent all my time before bed dealing with some annoying framebuffer-related issues that resulted from an optimization/simplification that I made yesterday. And here I was thinking that we had heard the last from those nasty Framebufferians...

After that, I made the mistake of reading some code-related stuff when I woke up this morning, and I found an incredible little nugget that I have never seen anywhere before - a trick for overloading preprocessor macros. Reading it was a mistake, because as soon as I saw it, I lit up with ideas about how I could expoit this to take automated code generation to a new level. And there went my day.... Indeed, in the end I came out of it with a pretty sweet system that will allow me to define data types using the preprocessor in a very natural and clean way, thanks to this new trick. The ultimate result of this is that I could automatically inject whatever convenience code I needed into all of the types defined this way. For example, serialization would be free for these classes. I could even do some basic reflection, getting the number, types, and names of fields, etc. Perhaps more importantly, this could really reduce the number of lines that I have to write to define functionality, and, in doing so, reduce the opportunity for errors.

I'm all about code reduction, but I'm not happy that it took me the greater part of the day to explore this idea. I'm also hesitant to get excited about this, because I've been known to get a little too macro-happy in the past, and it tends to end in a catastrophic collapse The last time it happened (before the KS), I was exploring the idea of an automated type information system primarily driven by macros, and it ended in so much blood and tears that I had to give it up. Playing with fire, no doubt. On the other hand, macros already define all of the code that holds together the component system that drives game objects. Doing it manually would result in a lot more code, or a loss of efficiency...but with macros I have the best of all worlds. So there's obviously the potential for great power. Only time will tell whether this endeavor will end in happiness or collapse...

I'm hoping for a bigger Wednesday - and a bigger second half of the week, hopefully. We'll see!

By the way, we now have two fantastic new smileys I recommend everyone use them to their fullest extent

[ You can visit devtime.ltheory.com for more detailed information on today's work.

Wednesday, June 12, 2013

Pretty nice day Once again, hammered on the old framebuffer code in the wee hours of the morning. Fixed up the problem that caused me grief yesterday, this time with a solution that will hopefully hold for life. Then again, I've obviously learned my lesson when it comes to thinking that I won't have to deal with those nasty Framebuffer villagers again I do hope this is the last of it.

Went on to do a bunch of seemingly-random things. Implemented real primary stars again, so that's a nice little touch. Started working on generating ore-laden asteroids, but didn't get very far since mining is still not really solidified in my mind. I have some asteroid objects with a new "extractable" component...that's a start. The ore is there (not really coherently generated yet, though). But how do you access it This is the question. Also need to generate visible ore on the roids. I think I've settled on a combination of visible and invisible ore. I'm thinking some will have chunks that are visible from the exterior that can be mined off, with other "invisible" contents on the inside of the roid, that require "drilling." Some will only have interior ore, hence will be invisible to the Mk. I - eyeball miner, unless he's good at guessing. This feels good to me, as it gives everyone a chance to get some ore - all it takes is a gun and finding some exterior ore to haul in a bit. But it also affords serious miners significant advantages over casual ones. Getting good scanning equipment will pay off, as it will significantly multiply your opportunities for striking something. And of course, I'm still set on having specialized mining beams that extract the ore efficiently (much higher yield than blasting away at it).

How ore density is represented and visualized is an open question. The idea of spherical harmonics briefly crossed my mind (think orbital lobes on an asteroid as the density patterns of ore...maybe a stupid idea, but interesting nonetheless). But maybe scattering simple spherical densities with noise perturbation would suffice. And how to visualize? Heat map on the surface? Volumetric render? We'll see Either way, I'm pretty set on having nonuniform density within asteroids. Quite boring if every rock is just one big chunk of ore!

To be honest, I'm feeling a very uncomfortable pull these days from my clean coder side...a pull towards functional programming. I think I've come to a point where most of the engine code that can be minimized and cleaned with ease has been taken care of. Unfortunately, I'm starting to see a whole new class of simplification. And it's killing me. I'm seeing patterns that I couldn't see before, and starting to realize that there is even less code than I think. At the end of the day, honestly, the whole game is just compositions of operations on really basic pieces of data. But the operations aren't random, they're structured. I am duplicating structure when I should be unifying it, but I couldn't see it in this abstract way before. The engine is 57K lines of code at the moment, but I can't help but wonder, if I could have seen all of the patterns that I see now from the start, how short would it be today? 20K? 10K? Less??? And if I were to go down the functional road now...this road that would lead to a greater simplification than the LT engine has ever seen...how long would it take me? Am I overestimating the opportunities for structural reuse? Too many questions. So hard to find the right balance between coding and thinking. Should I set a limit on my desires for code cleaning? Maybe at some point I need to say, OK, that's enough, we have to lock it in now even if it means sub-optimal code. But then, what doors of opportunity am I closing for LT if I don't make it in the best possible way that I am capable of?

Tough decisions need to be made. I think I may take a day or so to play around with these ideas and see how quickly I can solidify them. If I can start to collapse code in a day, maybe it could be worth it. There are certain code files that currently overwhelm me, and I need to collapse them, but I don't think I can do so without a new toolset...a new and more powerful level of abstraction.

Codemonkey confused

[ You can visit devtime.ltheory.com for more detailed information on today's work. ]

Thursday, June 13, 2013

Whew. What an intense day. So many brain cells lost, but so much knowledge gained

I said I would explore alternative toolsets for a day or so, and I did that. I explored the possibility of using some really heavy c++ concepts to build abstractions that could potentially allow me to do more in less time and with less work. The result was both extremely positive and somewhat negative at the same time! The good news: I never even realized how much power this language has..I never realized all of the amazing things that can be done with the right knowledge (in this case, functional template metaprogramming). The ability to express super-high-level concepts in a really natural way is there! The bad news is that utilizing these concepts exacts a very, very high mental price. The level of programming that I did today was almost too much to handle. A single miss-step would lead to pages and pages of compiler errors (no joke), to the point that I was basically on my own to figure out what went wrong...and when dealing with high-level abstractions, it's not fun to be on your own... This not only tires me out mentally, but means that I can't make very fast progress when it comes to building these tools. It seems that, basically, with this technique I am offloading more work to the construction of tools, and removing work from using them. In general that's a good strategy...you know, third little piggy built his house out of bricks and whatnot But it does mean harder work up-front.

That being said, it seems like the best way to go forward is to continue to work on my little functional toolset for an hour or two a day, but go back to programming as normal until the tools are finished. It'll certainly take more than a day, I'm afraid. Still, after seeing the raw power of the types of constructs that one can build...I am inspired, to say the least, at the potential!!!

If you're a programmer and are interested in seeing a bit more of what I'm talking about, have a look at the boost pheonix library. I had already started implementing a functional language in c++ via metaprogramming when I stumbled upon this library. My mind was so far blown that I had to spend several minutes just collecting the small chunks of grey matter from around the room. Seriously...this library is pure insanity. Every c++ construct, written in a totally natural syntax, converted to an object, allowing it to be passed around, composed, etc. Amazing. I took a glance at the source code and quickly decided that I couldn't use this library as it is (a single error would probably cost hours to debug, as the headers are just unreadable). But I was able to glean some key concepts from looking at the code that really enlightened me. I restarted my own implementation and got a lot further this time, to the point that I am super excited!!!

Ok...so...yeah, back to working on things that people care about soon! But really, I do believe that this toolset, when it's finished, is going to allow me to build LT faster than ever before

PS ~ Yesterday I seem to have given the impression that these alternative coding solutions that I am considering were going to be a matter of speed vs. quality...i.e., that this would cause a delay in producing the game. I would like to clarify that such is absolutely not the case! On the contrary, the reason that I consider stuff like this is that I am trying to find the easiest / fastest way to build LT! If and when I do adopt new programming constructs, it will only be in the interest of being able to work faster / be more productive, not just to make the code look cleaner (although, not surprisingly, cleaner code usually means faster and more productive development). So yeah, no worries there

[ You can visit devtime.ltheory.com for more detailed information on today's work. ]

Friday, June 14, 2013

Very exciting day, maybe not so much for LT but for me

My new rig came in today, so naturally I had to spend most of the day getting it setup and playing with it! I am even more happy than I expected to be. As I mentioned earlier, my intent was to purchase a lower-end machine with an AMD graphics card so that I could A) test on AMD cards (and never repeat the pre-prototype incident), and B) have a lower-spec desktop to work on (developing on a laptop isn't that fun).

It's actually a bit comical. This machine is *beautiful*. It's the first time I've ever ordered from Maingear, because I was tired of having problems with other custom rig vendors (I won't call names...), and apparently Maingear seems to be the choice for "quality." So I took a risk. Man, I am extremely impressed. It gets a lot of things right. But the interior is just...Godly. The cabling and layout of all the components is beautiful. It's a little sad though, because when you look inside this thing, it looks like a rig where you would expect to see a massive liquid cooler on the CPU and two raging double-decker, footlong graphics cards running in SLI, taking up the whole interior. Instead, there's a whole lot of empty space and a single little 1-PCI-slot, economy AMD card going "hallo! play with me!!" and an unobtrusive CPU with a standard fan. Yeah, I guess the guts don't do justice to the exterior. But when LT comes out, maybe as a treat I will fill it up with nicer components

So, I promptly got all my dev stuff set up on this puppy to see for myself how LT fares on AMD cards. One of the big benefits to all of this multi-platform, and now multi-PC work that I've been doing lately, is that the LT Engine, build environment, and development tools are all becoming more and more simple / minimal, so that it takes less and less time to get it set up. I'm a big fan of simplifying, in case anyone hasn't noticed So it didn't take long to get everything going on Windows. BUT. I didn't want to run it on Windows. Why not go all the way and try for the biggest challenge?

I got myself a Linux partition set up all nice and cozy, and built LT on it. This presented significantly more of a challenge. Unlike my previous Linux install, this install is 64-bit. Interestingly, 64-bit Windows is perfectly content to build and run 32-bit programs, which is what LT is designed to be. Linux, apparently, is not. 64-bit Linux wants to build a 64-bit program, which is not really what LT was designed for. That being said, good programming is architecture-agnostic, so, in theory, it shouldn't make any difference. Of course, in practice, it broke in several places. Specifically (and I was well aware that this would be a problem if I ever decided to try a 64-bit build), my promiscuous use of the size_t type where it doesn't belong got me into some serious 64-bit trouble. It's one of those bad habits that you know is bad but you just can't break It feels so darn good to type size_t. Much better than typing unsigned int, and even if you typedef it to uint..well, there's still just something sexy about using size_t Problem is, size_ts are usually 8 bytes long on a 64-bit system, and 4 bytes long on a 32-bit, whereas unsigned ints usually stay the same. Voila, millions of compiler errors about loss of precision and what-have-you.

But, after getting back in-line with some proper type usage, my 64-bit build came through! This is another feat that I was not planning on accomplishing (especially not this early, if ever). I was going to stick to 32-bit builds, but since that won't fly for 64-bit Linux users (I was not aware of that), well, just gotta suck it up and do it. But it's done!

So, how did this fancy 64-bit Linux build on an AMD card perform, you ask? Terribly!!! Uninstalling the AMD drivers and installing the "open-source" ones (written by non-AMD people, presumably?) improved the situation a bit. Jeez, sorry to say it, but that's embarassing...third-party people writing better drivers for your own hardware than you?? Come on now...! The game is now playable. I have already discovered a major performance problem with the bloom implementation, which will need to be fixed sooner than later. I also fixed a shader bug that caused planets to show up half-black (presumably just on AMD cards). With bloom off, I am getting 60+ FPS, which is OK I suppose. Unfortunately, the procedural algorithms are absolutely KILLING the card. Generating system backgrounds at high-res takes like...a minute on this card!?!?! A cursory bit of inspection suggests that the problem is in the driver's compilation performance. I was always a bit worried about this part. The shaders that LT uses to compute the procedural stuff on the GPU are very math-heavy. A poorly-optimized driver will be murdered by the complexity of it all when trying to compile the shaders. Sadly, I'm not sure how much power I have over this Seems like my only option would be to thread the compilation, but I honestly don't even know if that's possible. Multithreaded GL contexts is NOT something that I want to get into..

Despite all this apparent negativity, I'm more excited than ever!! I finally have a desktop rig on which I can be comfortable developing. If there's a performance problem with LT, this poor little graphics card is going to cry about it, and I'll know in time to fix it! Should be the same situation for the CPU. If I can get LT to perform well in a 64-bit, Linux, underpowered-AMD environment, then I think I can safely feel good about it (high-end Windows Nvidia users will probably get billions of FPS )

PS ~ There was a lot of time that didn't get logged today while I was setting this thing up!

PPS ~ Sorry about the size of the log. So much nerdy excitement today

PPPS ~ Proofreading this log, it occurs to me that I sound like someone who is being paid by both Maingear and Nvidia to spread propaganda. (Sadly) This is not the case. If either of them prove to be unreliable, I will be the first to say so

[ You can visit devtime.ltheory.com for more detailed information on today's work. ]

Saturday, June 15, 2013

Looking at yesterday's log, I'm pretty sure I earned a free dev-log day today Not that they aren't fun to write (I obviously have a lot of trouble coming up with what to say )

I guess it's a teensy bit sad, but the most exciting thing that happened to me today was fixing the bloom performance issue. May seem like a small deal, but this simple effect was eating up literally 50% of the frame time on good-old Kubert (that's what I'm naming the new AMD graphics card. Just because.), and probably about 30% on the GTX (he doesn't get a name, he's too powerful. And sassy.) BUT an hour or so later, bloom was taking less than 10% on Kubert, and is probably now immeasurable on the beast. Nice! So it turns out that flushing the GPU's command buffer 4 times in the middle of rendering is actually not a good idea. I'm not that stupid, I swear!! It was sneakily nested deep inside a texture generating function...had I seen it earlier, I would have ripped it earlier...

More work on setting up asteroids for mining! Not much to say about this at the moment...hopefully next time you hear about it I'll be collecting chunks of some awkwardly-named ore

I also spent some time attempting to upgrade the forum's theme today. Although I rolled it out briefly, I quickly changed my mind about it after agreeing with a few people that it really wasn't as functional as this thing. Truth is, the current forum theme is very easy to read and navigate. That's why I picked it in the first place. However, I do want something prettier eventually. I'm going to keep working on it. Sadly, that took a good bit of time out of my day..but a failure is never a waste of time I learned plenty about phpbb, joomla, and forum customization in general. Much to my weekly efficiency's chagrin, RescueTime looks down upon all things browser-related, so I think my final weekly efficiency dropped to 83% in the last few hours, when I did a lot of online work Oh well, let's aim for at least 85% next week

[ You can visit devtime.ltheory.com for more detailed information on today's work. ]

Week Summary

General Sentiment

I don't know what's going on these days. In all honesty, every week I'm feeling worse and worse about the accomplishments of the week, but every week I look at the weekly summary and realized that it wasn't nearly as bad of a week as I imagined it to be. Perhaps my standards are constantly raising, and it's getting to the point where I'm having trouble following them. Not exactly sure, but it would be nice to feel good about a week again...and I'm not sure I have enough energy / time to meet whatever my standards have risen to Well, let's try to do it next week...maybe I'll feel bad throughout the week, but I'm sure that, like this week, I'll be redeemed when I look at the weekly summary...

Major Accomplishments

• Implemented a console (no graphics) version of LT to use for fast simulations and optimizing the simulation engine
• Implemented a system for automatically defining new classes with built-in functionality like serialization, started using it to reduce code in certain areas of the engine
• Totally overhauled serialization system for faster and easier data loading/saving
• Equipped engine for 64-bit-friendliness and built first successful 64-bit version of LT
• Improved and unified scheduling engine architecture
• Improved in-engine profiler, as well as in-engine memory checking tools
• Started implementing ore / mining
• Started implementing a functional programming library for c++ (will it take us to a new level of powerful/clean code??)
• Fixed several issues with the framebuffer implementation
• Fixed several AMD-specific issues using my new AMD card
• Massive performance increase for the bloom effect