Return to “[Archived] Daily Dev Logs, 2012 - 2015”


Week of June 2, 2013

Sunday, June 2, 2013


Strong start to the week :) I am setting very high expectations for June. You can fire me if I don't hit them!

The biggest advancement of the day was making system properties continuous over space. Several people asked whether the system backgrounds would look coherent throughout a region, and I answered yes. Today I made it happen! I was initially thinking coherent noise, and didn't think twice that this would be the correct solution. But after implementing smoothly-varying properties via coherent noise, I realized that it was rather dumb to be generating properties based on some arbitrary partitioning of space when, in reality, I have already created a very strong partitioning via the hierarchical universe structure. Instead of praying that the relatively large distances between regions would cause them to have distinctive feels, why not generate the properties directly from the region itself? This led me to a new, recursive formulation of spatially-varying properties that looks and feels almost exactly like traditional fractal noise, except instead of being tied to a regular grid structure, the noise is based structurally on the actual hierarchical universe structure. Dunno if that made any sense at all, but the end result is good, and that's what matters :clap: Regions feel very coherent now. Quite fun to explore :D

I wrapped up the day with an unexpected code-cleansing session that targeted some of my math code in the engine. This came about as a result of my desire to move to a double-precision engine. Since I got so much flak about the size of the planets in the last update, I realize that I'm going to have to upgrade the engine sooner or later. The reason that I didn't make the planets huge (and the reason, for that matter, that planets in Freelancer aren't huge), is that a single-precision engine (i.e., one that uses 32-bit floats to keep track of positions) is insufficient for massive planets. Even with the system scale shown in the demo, we are already pushing the limits of single precision. If I increase the size of the planets a whole lot more, I will have to make the systems essentially only have one massive planet and not much else, since I cannot expand the system bounds much further without causing the graphics to become unstable. The solution is to move to double precision. If and when I do so, the engine will be able to handle massive planets and still have massive distances between them, so we will have some awesome systems with awesome scale!! Initially I had not planned on doing this for LT, but the demand is obviously high, and I am beginning to believe that it could be fun to have this kind of epic scale in the game :)

Moving to double-precision is a big switch, though, and requires some technical heavy-lifting. Today I converted my matrix math to generic format so that I'm ready to support double-precision matrix math. All my other math classes are already generic and can support any precision. The next step is to figure out the minimal required set of changes to switch. Contrary to popular belief, a double-precision engine (if implemented efficiently), doesn't use double-precision calculations everywhere. For example, there is absolutely no point in storing the rotation of an object in double-precision. Same for forces / velocities as well as a great deal of other calculations. Really, just about the only thing that needs the extra precision is anything dealing with world-space positions. The only real trickery here is that the double-precision world-space positions need to be converted back to single precision for use on the GPU. If done naively, this would result in no gain whatsoever. If done cleverly (the classic camera-at-the-center-of-scene approach), it provides virtually all the gain of double precision...but requires cleverness. So I just need to figure out how to be clever. And then we can have big planets :thumbup:

Hour Tally

Coding: 6.30
Internet: 2.66
Testing: 1.92
Thinking: 0.85
Total Logged Time: 11.73
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford

Re: Week of June 2, 2013

Monday, June 3, 2013


You know, every once in a while, I think it's important for a project to light fire to its codebase. It keeps both the codebase and the programmers excited and on the move. Remember the attack on Framebuffer last week? In truth, it was a mere drop of blood next to the ocean that was spilled today.

I'm not really sure how it got so out of hand: my day started off with a nice lunch in the park. Just some harmless thinking time as I tried to build a more memory-efficient engine architecture in my head. Watching the water flow over the rocks made me think of asteroids, which made me realize that asteroids outnumber other objects by somewhere between three and four orders of magnitude. I realized that, in reality, memory is basically no concern whatsoever if we just apply some cleverness to the few objects that will be extremely numerous (asteroids....erm...not sure if there's anything else that falls into that category). So when I got home I set to work on the asteroids. Except, to implement my memory-savings, I needed a mechanism for determining whether a given object was being simulated in the "active" system or outside of it. A remarkably simple requirement. But of course, as simple things have done so many times, this made me realize that my whole simulation architecture was wrong and that it needed to be immediately uprooted and replaced :roll:

I spent hours gunning down the simulation engine. There is no simulation engine. There is only the world. This is a theme that keeps repeating itself in my development process. I externalize certain concepts in some weird way like an "engine" or "manager" or what have you, only to realize later on that the concept can be more naturally implemented as part of the world. Simulation is no different. You don't need someone else to keep track of all the contexts that need simulation / to handle the LOD balancing and so forth. You just need to make this logic part of the natural flow of the world, and let the LOD simulation merge with the world itself so that there is no separation. Easier said than done.

Naturally, as I gunned down the simulation engine, I realized that the simulation engine had been conflated with a completely separate idea: frame interpolation for decoupling simulation and rendering. So in order to properly destroy one piece of code, I realized I also needed to destroy a lot of the existing rendering architecture. Yayyy ripping out code :clap:

Well, you know, sometimes you pull up a weed and you get a forest :shock: I am pretty convinced that I set a personal record today for how many changes I made all at once without testing. Usually I try to test things as incrementally as possible to ensure that I'm getting them right. But if you have to uproot and replace both the simulation AND rendering architecture at the same time, you basically have to finish everything before you can test any piece of it. So that was scary and fun....I went through about five hours of pure coding, no compiling whatsoever, before I started the long process of watching the compiler spit out trillions of errors and whittling them down. In the end, I was pleasantly surprised that there were no major issues, and that the new architectures worked well. Of course, I'm sure I've jinxed myself, and have probably actually broken thousands of things :shifty:

By the way, I swear this was relevant. I'm trying to do some final cleansing to make universe simulation as silky-smooth as possible :) Many lines of code dead....more to come? Most likely :twisted:

Hour Tally

Coding: 7.75
Internet: 2.24
Testing: 2.05
Thinking: 1.10
Total Logged Time: 13.14
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford

Re: Week of June 2, 2013

Tuesday, June 4, 2013


I needed to give my code some time to heal after yesterday's massacre, so I decided to stay away from the core simulation and/or rendering areas today. They deserve a day of rest :roll: So I ventured into graphics land once more! I didn't really mean to spend so much time on graphics today, but you have to take the inspiration when it comes! I made a lot of advancements in the quality of the ship surfaces, as I turned my attention to both the plating algorithm and the metal shader. Working from reference images of ships, I finally made some real progress on the plating! Actually, so much so that I'm to the point of being happy with it, which is way more than I could say yesterday.

I also spent a good bit of time working on the HDR / bloom implementation. I've been ignoring it for a while, which means it has become fairly broken. But, inspired by the liberal usage of all things glowy in CryEngine tech videos, I got my act together and upped the glowiness of the engine. Bloom is somewhat of a notorious effect...people tend to overdo it, and nothing ruins graphics like overdone bloom. Really. But when I watch CryEngine videos, for some reason, I am significantly more attracted to that bloom (despite how exaggerated it is). Took me a while to realize it, but it would seem that the big difference between good bloom and bad bloom is that good bloom is wide and soft, whereas bad bloom is narrow and strong. After tweaking settings and playing with the whole HDR pipeline for quite some time, I confirmed that this is indeed the case. Soft, wide bloom is very appealing. Subtle, but conveys the radiance of an object within a wide radius :thumbup:

Wrapped up the day with a few more hours on the porting effort. Major progress today as I got somewhere around 50% of the engine to compile! That's significantly more than the previous 0%. Slowly but surely, I'm beginning to understand all the little nitpicky things that gcc wants from me (and wow, there are a lot of them... :cry: ). I'm almost scared to get to 100%, though, because I fear that compiler errors will be the easy part. I know the first thing I'm going to see when I start up the engine is going to be ... "Segmentation fault." And then the fun will begin :problem:

Hour Tally

Coding: 3.04
Internet: 2.21
Logistics: 3.10
Testing: 2.19
Thinking: 0.92
Total Logged Time: 11.44
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford

Re: Week of June 2, 2013

Wednesday, June 5, 2013


Another one of those days where successful compilation just didn't happen :roll: I'm on a quest to fix saving / loading (yep, broken already...), the issues have led me to some serious problems with the way items are implemented (this manifested itself in LTP as identical items sometimes failing to stack, and no doubt is to blame for some of the crashes). It could have been a one-hour fix if I had taken the easy route. But, as always, the simple issue started me down a path of asking too many philosophical questions. And, as usual, the philosophical questions led me to philosophical answers that required sweeping changes. This shouldn't surprise anyone anymore :geek:

I was doing a lot of thinking about a lot of really general and abstract concepts, like the relationship between data and the source of the data. More and more I am coming to the realization that data is just a temporary thing that we cache for performance, and that the only real thing of interest is the source of the data. Really, if I had infinite CPU time, I would never store anything. There would just be code. And everything would be ultra-beautiful. This is what led me to the nice dynamic resource infrastructure that I mentioned a week or two ago. I think I hit pretty close to the mark with that system, but today I wiped it out and hit even closer to the mark :D I generalized the work I had done to an even more abstract and modular level, with the hope that this work will carry over seamlessly to fix the item implementation. Ultimately, I want an engine that is entirely driven by algorithms, not by data. Yet, at the same time, I want to specify those algorithms in a very natural and compact data form...kind of a small irony. Today I developed some more tools for doing so. Sorry it's so abstract...but the work was definitely out there :wtf:

Very deep c++ work today. Had to pull some template tricks that I've never touched before (heading into metaprogramming land). Dark magic....but powerful dark magic. Example of the result of this dark magic:

Code: Select all

ReferenceT<RefCountedT<CachedT<AutoPtrT<GeneratorT<RenderableT> > > > > renderable;
Whoa. That's one heck of a variable. It's the variable that stores whatever rendered form an object in the game will take. At first glance the code looks ugly. But then, at second glance, if you understand these identifiers (which are fairly simple), you can see that this single line is actually doing what could otherwise be pages of code. I'm not proud of the syntax of it, but I'm proud of how much it does in one line. This line alone tells the engine that an object's rendered representation will be...first, reference counted. Ok, that's a pretty standard thing, no surprises there, it just means the memory will stay in scope for as long as objects still need it. And we're holding a reference to a refcounted type, which means that we're wrapping a type that wasn't previously equipped for reference counting. But what are we actually referencing? A generator of renderables, meaning something that is capable of handing us a renderable. A renderable is what it sounds like: something that can be rendered. But why do we have a generator for it? Well, we may not ever need the renderable, so why load it if we don't need it? But we also own the generator, so we've got an autopointer to it, which makes sure that we dispose of it when we're done. Finally, there's a caching layer that means we'll save the result of the generator and use it next time instead of calling the generating algorithm again. The impressive thing about this is that, when renderable is accessed inside the code somewhere, the compiler is actually going to generate the code to do all of that. With just this one line! That's the kind of territory I want to keep pushing into. Huge amounts of work being done with tiny pieces of code :thumbup: Maybe the final game will be just one line of code. I would like that :)

On a side note, I was actually working in gcc today because Visual Studio doesn't have a break-on-first-error option, which makes it nearly impossible to debug sweeping changes to the codebase. With gcc you can compile just one object file, and work on them one at a time! As a bonus, we're up to 75% compilation on linux now. I can almost smell the disappointment when I finally get to run it :lol:

Hour Tally

Coding: 6.73
Internet: 1.65
Testing: 0.16
Thinking: 1.32
Total Logged Time: 9.86
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford

Re: Week of June 2, 2013

Thursday, June 6, 2013


So...guess what game I played today? Yeah, I played Limit Theory...

...ON LINUX :ugeek: :ugeek:

I know, I know...I was talking about only taking a few hours here and there to work on this. But I got carried away. It was an entertaining battle with gcc, and then with ld, and then with gdb. Killing the issues one at a time was too much fun, and the hours were slipping away, but I couldn't help it! Getting the 100% compilation was fairly easy. Then came the massive wave of linker errors. Linker errors are always harder than compiler errors. Here's the gist of it: Compiler error says "Hi, I think you made a mistake on line 67 of HappyAsteroid.cpp, I expected to see a typename at character 20, but instead saw a + ." Linker error is more like "YOU MADE A MISTAKE SOMEWHERE IN ONE OF THE 500 FILES OF CODE BC I CAN'T FIND SYMBOL X$::$XXkjEFE#&:::~VTBLE@# REFERENCED FROM ::#%#?LIB@%#$ OK PLZ FIX IT." Obviously those are just so much more fun :crazy: After hacking and slashing through the cryptic linker errors, I proceeded to do the old ./Launcher. I knew what I would see, so it was not at all surprising when I saw "Segmentation Fault (Core Dumped)" (Linux equivalent of a hard crash, if you didn't know.) Called it from day one :( But, surprisingly, it only took me about 10 minutes in the debugger to fix all the issues...and I was up and running, watching the launcher appear before my very eyes!!!

So things actually went smoother than I had hoped! After a few brief bugfixes, I was staring at the real deal, Limit Theory running on Linux (xubuntu if you care to know)! Framerate was about half as on Windows, even with the latest beta NV drivers installed. I guess I have some work to do there, but I'm actually glad that I have a less-performant system to run LT on while my laptop is out of commission. It scares me to be developing on this desktop, because I fear making performance mistakes without realizing it (Mr. 560GTX eats performance mistakes for breakfast).

Anyway, I know most of you don't give two ounces of Bantha poodoo whether LT runs on another OS or not, but for me it's just an exciting accomplishment, and also a reinforcement that the LT Engine is clean and versatile. It just feels good :) And I'm also excited that I have another toolchain that I can develop on. The Linux build actually compiles a lot faster than Windows, thanks to multi-threaded compilation. Then again, Visual Studio can do that as well, but I turned it off in favor of whole-program optimization, which may well be one of the reasons that the Windows build is kicking the Linux one's butt in performance... :?

Well, maybe this is more relevant to you if you're a Mac person...I guess there are probably more people out there who care about the Mac port than the Linux one. And for those people, I say: we are much, much closer to a Mac port! I estimated previously that this would get me 50% of the way to the Mac port. Now, I'm going to adjust that estimate and say it's probably closer to 60-70%. All I really have left to do for the Mac version one, and then figure out the shared library convention. I think it's a pretty good sign that porting is already so far underway at this point in the development cycle. It definitely makes me feel more comfortable about being able to deliver on all those promises :thumbup:

Sorry to bore you today Windows folks...hopefully I will have something for you tomorrow! :wave:

PS ~ Logistics = working on an interview for a Baton Rouge magazine.

Hour Tally

Coding: 6.72
Internet: 2.44
Logistics: 0.98
Testing: 0.87
Thinking: 1.79
Total Logged Time: 12.80
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford

Re: Week of June 2, 2013

Friday, June 7, 2013


Not much of a Friday, unfortunately :( I tried to implement an idea that I had yesterday to simplify some of the code surrounding things that change and have to report their changes (inventories, hangar bays, etc). Sadly, c++ really didn't want me to see this idea through, and despite the level of coding cannonry that I pulled out, it just wasn't going to happen the way that I wanted it to. Someday I will probably be able to figure out how to have fields automatically report modifications to themselves without any memory overhead, but today I wasn't able to pull it off, and it cost me several hours.

Not all was lost, though! I spent some quality time acquainting myself with the linux profiling tools. I know, "we've heard enough of this linux business, let's get on with our lives." But profiling is relevant to the quality of the game as a whole, not just a particular port of it! I figure, the more profiling tools that I have at my disposal, the better! I also stumbled upon , which is both funny and remarkably insightful at the same time! After thinking about it for a while, I realize...they're absolutely right, profiling a bottleneck should be almost trivially easy...just break and look at the callstack. Probability says that you are in the bottleneck (if there is one). Do it a few times and probability says even more strongly that you are going to hit the major perf hogs if your program has any. I mean, duh, it's obvious, but at the same time, somehow insightful. After playing with this technique for a few minutes I uncovered what seems to be a performance issue of which I was unaware: SFML text drawing is quite slow. Not sure if it's just on linux or windows as well, but it is probably worth moving to my own text rendering system sooner than later, just so that I can have better control over the text (both in terms of quality as well as things like applying shader effects, drawing it to textures, etc).

By the way, last time I'm going to mention linux today, I promise: it seems that I was wrong yesterday when I said that the linux build was running about half speed. What I found out today is that vertical sync is getting forced for some reason on linux (even if LT doesn't request it), which was causing me to see a 17 ms frametime when I was used to seeing about 8 ms with vsync off. So, it's hard to tell exactly what the performance of this version is since I can't get an accurate fps reading with vsync on. It may well be fairly close in performance to the windows build. It certainly feels smooth :)

I spent a good chunk of time with my family today, as my Dad is currently visiting town. Forgive the hours please :geek:

Hoping for an intense Saturday... :shifty:

Hour Tally

Coding: 4.43
Internet: 2.43
Logistics: 0.55
Testing: 1.14
Total Logged Time: 8.54
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford

Re: Week of June 2, 2013

Saturday, June 8, 2013


Intense Saturday indeed 8-) So intense, in fact, that I decided I would rest my eyes for a few minutes before writing the dev log. That was two hours ago. I hate how time compresses itself and runs away from you if you shut your eyes for just a few seconds :oops:

I started out with fixing yesterday's text rendering performance issue. I decided that, maybe I wouldn't go down the route of managing my own text drawing after all...especially not when it only took 15 minutes and a few lines of code to fix the problem that was causing SFML to lag on Linux (argh! He said it again :( ). Shortly thereafter, I was flying around at top smoothness once again! :thumbup: Now I can comfortably say that the port is almost as fast as, if not as fast as, the real deal! Definitely smooth enough to enjoy.

Next up, reinventing the wheel with items. Lots of code ripping, blood, tears, diet know the story ;) Several hours later, we had a new item-handling core! As a bonus, loading / saving decided to start working thanks to playing nicely with the new system. Nothing better than code rippage that has the side-effect of fixing things! Oh, and it should also be the end of the item stacking bug as well.

I'm a little bit torn about where to go from here...I want to start rolling into some deep simulation and AI, but the world still is further than I'd like it to be from completion, and I'm not sure if it's such a good idea to start working on simulation before the world is finished :think: Choices, choices.

PS ~ Sadly, the hour log for today is mostly a guess. I was in "the other operating system" ( :shh: ) all day, and unfortunately my time tracker does not have a compatible version. I would classify it as a "normal" day for hours, hence the guesses below :geek:

Hour Tally

Coding: 6.09
Internet: 2.63
Testing: 1.19
Thinking: 1.13
Total Logged Time: 11.03
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford

Re: Week of June 2, 2013

Week Summary

General Sentiment

It almost felt like a slow week...but looking back over the accomplishments, I think it's clear that I'm getting better at squeezing more out of the hours, because even without hitting 80 hrs this week...there was definitely a lot of progress! :D Having the first port finished makes me happy. The new simulation architecture and overall improved cleanliness makes me very happy. Overall, a very happy week! The engine continues to get cleaner and more lovable by the minute, and that's only going to make me want to work on it more. It's a lovely positive feedback loop :thumbup:

Major Accomplishments
  • Finished the Linux port
  • Rewrote and cleansed the simulation architecture
  • Rewrote the way items are dealt with (side-effect is fixed savegame loading and fixed stacking bug)
  • Improved performance of text rendering (massive boost for Linux, but should be at least slightly noticeable on Windows as well)
  • Improved the architecture of deferred asset loading, pushing towards more powerful abstractions
  • Significantly improved ship plating algorithm
Combined Hour Tally

Coding: 41.05
Internet: 16.24
Logistics: 4.62
Testing: 9.52
Thinking: 7.10
Total Logged Time: 78.54
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford

Online Now

Users browsing this forum: No registered users and 0 guests