Return to “Dev Logs”

Post

Saturday, July 29, 2017

#1
Saturday, July 29, 2017

Hello hello everyone! Glad to be back to the devlogs. Let's hop right in. It's going to be a long one.

It was a tremendous week. With Sean and I back in the office (and me particularly energized and ready to get back to LT after the stress of teaching), we were already in a position to have a good week. But several major happenings upgraded it the past five days from good to awesome.

---

Adam: (King of) Surface Extraction

This week's award for MVP goes to Adam, about this there can be no question.

Before my three-week teaching leave, I set Adam to the task of studying distance field surface extraction techniques and finding out if there were any good, open-source implementations out there that would allow us to upgrade our distance-field-based mesh pipeline. If you've been following the logs for a long time, you've definitely heard me talk at lengths about distance fields (SDFs) and my various adventures with them. The LT Prototype used SDFs as the exclusive means of generating meshes. Today, they're used for non-artificial constructs like asteroids, while BoxMeshes (the new PlateMesh) are used for ships, stations, and components thereof. Generating meshes from SDFs is a very difficult problem. I've implemented two algorithms for doing so -- marching cubes and surface nets, and currently use the latter to generate asteroids. Both of these are 'naive' algorithms in that they're just about as simple of a solution as you can possibly find. Despite that, they're still quite difficult to understand and implement (SN more so than MC).

I've often pined about not having a great solution to this problem. I'm sure I could find at least a dozen mentions in past devlogs of 'what ifs' with respect to having a really good surface extraction algorithm, and how everything would be golden if we did. Thanks to Adam's work, it's no longer a 'what if' situation :)

Adam not only succeeded in finding an open-source library that has the functionality we need (and more), he also went through the struggle of figuring out how to build, test, use, and evaluate the quality of the code therein. This week, he was able to show concrete timings and outputs that proved the library capable of performing adaptive, high-quality surface extraction with great speed.

The implications are vast. I've only just started working with the library on Friday when Adam handed it off to me, so I can't say with 100% certainty that we won't run into snags. But already I have been able to do enough that I'm extremely optimistic. With luck, it'll solve/enable the following for us:
  • Unified procedural modeling pipeline with everything constructed from distance fields
  • No loss of 'sharpness' for objects that should be sharp (like ships, stations, etc.)
  • Support for far more detail and complexity in our PCG mesh algorithms, translating directly to more creative variation and higher quality for our models
  • Substantial decrease in cost of rendering meshes due to adaptive extraction
  • Decrease in cost of computing ambient occlusion for meshes
  • Increase in ambient occlusion quality, especially for complex meshes
  • Potential solution to the problem of efficient texturing of our meshes thanks to manifoldness (BoxMeshes did not produce manifold meshes)
  • Free LOD meshes with optimal fidelity, leading to even more rendering performance at no dev cost to us
At the end of the day, all of that basically boils down to both a lot of dev time saved, a substantial performance gain, and way more flexibility for our PCG. Yes, I'm very happy about it :squirrel:

Sean: AI Combat, Teams, Hierarchical Fleets

This week, Sean dove deep into AI combat. He's set up a combat sandbox that can run really fast combat simulations and provide us with information about the performance of different combat AIs. At the most basic level, we can run head-to-head dogfight simulations between two AI ships using two different algorithms and determine which is superior (in particular, we can determine the win rate of both ships under randomized starting conditions).

After he had gotten the basics down, I tasked Sean with making the sandbox more general, so as to support 'teams' of arbitrary size and count. After only a short while, we had an even more powerful sandbox that can rapidly simulate entire 'faction vs. faction' battles and give us insights about how fleet composition, starting location, and AI reasoning all add up to prowess on the battlefield. At one point, Sean simulated a 100-faction battle with thousands of ships. It was entertaining but terribly slow since his sandbox isn't using our optimized engine as a backend. Once I finish my scaffolding of Lil'T, we'll move his code into the real engine so that he'll be able to use the ECS to simulate such battles in milliseconds ;)

Sean is now working to implement arbitrary fleet configurations and develop combat algorithms that will behave intelligently within the context of a potentially-massive fleet structure (unit X is part of fighter squadron A, which is flying in v-formation escorting cruiser Y, which is part of offensive line B, which is moving in wall formation 10km ahead of battleship C, etc). This is the pinnacle of the general AI combat problem, and a very difficult one indeed. I don't anticipate a solution to this to come about as quickly as he solved the other problems, but I'm sure he'll still end up surprising us :D

It continues to be really nice that we've got someone working on the hard gameplay problems as Adam and I blow away all remaining technical ones :monkey:

Josh: Lil'T & LuaJIT

As for me, I've been at the helm of Lil'T development this week, trying to get it all to a point where everyone can start working in the same place. Practically-speaking, it means that I have to have 'the way we do things' locked-down in all departments. I've had some really great success in doing so this week!

So, you're all aware that we've struggled with FPLT in the past, but that LuaJIT + high-quality C systems have proven capable of yielding beautiful performance. As I've expressed before, the only caveat is that one must tread lightly with LJ in order to reap that performance. A single mistake in a tight loop will cause LJ to abort the compilation of something that may run far too slowly in the interpreter. I had previously discovered that my code for LTDemo was doing exactly that, and getting almost none of the benefits of JITing (ouch). This week, while building the Lua base of Lil'T, I had to methodically tackle each of my previous mistakes and figure out how to do the same thing without making LJ angry.

After countless hours poured into reading LJ source, performing experiments, watching trace outputs, reading through LJ's generated assembly dumps, and trying various techniques to determine what does and doesn't work, I can say with confidence that I know exactly how to build Lua code that maintains all of the benefit of Lua's flexibility and readability, while still keeping LJ happy and compiling down to pure assembly. One breakthrough, in particular, was the discovery of a pattern that permits me to do almost all of the 'convenient' stuff that I was doing in the LTDemo code while still getting LJ to give us assembly! This means that much of what I thought would have to be 'thrown away' in favor of carefully-tuned calls to the C engine will, in fact, only require minor modifications! (For those of you familiar with Lua, it means that I can associate metatables with raw engine C types without incurring any overhead and still getting optimal assembly output -- meaning I can create wrappers that make our engine super easy to work with, and all at zero runtime cost. While it may look trivial to accomplish via the ffi.metatype LJ builtin, in practice there are loads of ways to botch it and end up paying a huge price for the convenience!)

In a final tour-de-force of this knowledge, I wrote some code that computes distance fields in Lua and uses our own wrapper of the aforementioned library to extract meshes. Even with JITing to assembly, I expected performance to be rather poor, as the code was just about as naive as possible. Much to my surprise, it ran blazingly-fast. Out of curiosity, I profiled it against optimized C (as in, gcc -O3 et al.) that does the same thing but uses smarter techniques to avoid overhead. Even with all those advantages, I found the C version ran only ~25% faster. When one considers how incredibly simple the Lua code looks in comparison, and considers that the C 'cheats' by using a faster method for populating the SDF grid, the performance difference is simply unbelievable. I'm officially and shamelessly promoting myself to the title of 'LuaJIT Jedi Master' :lol: I had entirely too much fun playing with extracted SDF meshes yesterday :)

---

All-in-all, this week was one of the best weeks in recent memory for LT development. Even as I woke up today, I felt more at ease than I've felt in quite a while. We've come so far with solving so many hard problems that lie at the very heart of Limit Theory's feasibility, and finally, after what feels like an eternity, the road ahead is clear.

The coming weeks should be packed with excitement :)

---

Although there's not much 'shiny' about this week, you guys continually tell me to post shots even if they're not pretty so...here you are... :ghost:

Mandatory shot of me being a l33t hacker with all my LuaJIT assembly dumps. Notice LJ is taking almost no CPU and minimal memory. Possibly because there's very little happening...still impressive, though.

Image

Shots of simple test CSG meshes, demonstrating how well the new algorithm works (notice the really nice adaptivity! Triangle size varies hugely depending on how many triangles are required to capture the details):

Image
Image

Sean made a few animated gifs from his AI work. Sadly, he didn't include the runs where he had huge numbers of ships :( I'll get him to make some for next week ;)

Image
Image

The gallery is here (although you've basically seen them all now).

:wave:
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
Post

Re: Saturday, July 29, 2017

#3
That is some beautifully optmised mesh there :D

(If only 3DSmax would produce such statisfying symmetry with boolean operations... main reason i do everything with surface extraction instead of additive/subtractive booleans, dont want to deal with cleaning up the mesh after booleans....)


Also very cool work on the ai combat, cant wait to see the formation thing going on in full 3d environments :squirrel: :monkey:
Post

Re: Saturday, July 29, 2017

#4
:D Love the cube, Josh . :thumbup: Very much appreciate the work Adam is doing . :thumbup: And more than happy with the contribution that Sean is making to the project. :thumbup: As much as I love shiny work I can also appreciate the not so shiny work being done at the moment. :angel:

:shifty: Obviously, I can't fully appreciate the technical stuff involved but I can see progress being made. ;)
Post

Re: Saturday, July 29, 2017

#6
After reading up on SDFs and the difficulty of feature extraction from them, I am very impressed by those results. Also the AI stuff Sean is working on looks like fun. I'm so happy that LT dev is speeding along. I can't wait to be able to play it (though in all honesty, I suppose I can wait since I have been for almost five years)
A life well lived only happens once.
Seems deep until you think about it.
Post

Re: Saturday, July 29, 2017

#7
Adam not only succeeded in finding an open-source library that has the functionality we need (and more), he also went through the struggle of figuring out how to build, test, use, and evaluate the quality of the code therein. This week, he was able to show concrete timings and outputs that proved the library capable of performing adaptive, high-quality surface extraction with great speed.
could we have the name of this magic library ? good open source code should be shared and celebrated! (and it might be relevant to my work, too :-) )
Post

Re: Saturday, July 29, 2017

#11
Josh, I'm really proud of you and Adam :) Tell him I said hi and that I think the work he's doing on LT is super badass.

BFett wrote:
Sun Jul 30, 2017 1:42 pm
Is no one else impressed by how beautiful the background of that mesh is? I love those butt like nebula. That 10v10 fighter Gif is also very neat. Thank you for showing us this neat stuff Josh!
I have a browser extension called Cloud-to-Butt that changes the word "cloud" to the word "butt" that somehow still manages to catch me off guard. LOL.

davidparadis wrote:
Sun Jul 30, 2017 11:33 am
And as is (hopefully) to be expected, my video review of the dev log is available: https://youtu.be/YoLC5tqRLa8
These reviews are really nice, you have a great video voice. Also, it's ver-BAY-tim. ;0
Ship Inspiration Pinterest!! (send me stuff)

"You’ve got to work on something dangerous. You have to work on something that makes you uncertain. Something that makes you doubt yourself... because it stimulates you to do things you haven’t done before. The whole thing is if you know where you’re going, you’ve gone, as the poet says. And that’s death."
- Stephen Sondheim
Post

Re: Saturday, July 29, 2017

#12
AlexWasLike wrote:
Sun Jul 30, 2017 2:47 pm
Josh, I'm really proud of you and Adam :) Tell him I said hi and that I think the work he's doing on LT is super badass.

BFett wrote:
Sun Jul 30, 2017 1:42 pm
Is no one else impressed by how beautiful the background of that mesh is? I love those butt like nebula. That 10v10 fighter Gif is also very neat. Thank you for showing us this neat stuff Josh!
I have a browser extension called Cloud-to-Butt that changes the word "cloud" to the word "butt" that somehow still manages to catch me off guard. LOL.
I... love it.
AlexWasLike wrote:
Sun Jul 30, 2017 2:47 pm
davidparadis wrote:
Sun Jul 30, 2017 11:33 am
And as is (hopefully) to be expected, my video review of the dev log is available: https://youtu.be/YoLC5tqRLa8
These reviews are really nice, you have a great video voice. Also, it's ver-BAY-tim. ;0
Thanks! ver-diddly-BAY-diddly-tim.
Shoot ALL the things!
Post

Re: Saturday, July 29, 2017

#13
First off, squee! Devlog!

Second, all hail Wonderbyrd! o/ o/ o/ o/ o/ o/ o/ o/ o/ o/ o/ o/ o/ o/ o/ o/ o/ o/ o/ o/ o/ o/ o/ o/

Far more detail and complexity AS WELL AS substantial decreases in rendering cost AS WELL AS optimal adaptive LOD? so what you're saying is that we could potentially have things like this
Spoiler:      SHOW
Image
in our limit theory universes? Yes I know that's a model, but it would require pretty efficient and complex meshes to be remotely possible. Do you guys think this new solution is capable of giving us this level of geometric complexity? or something more like this
Spoiler:      SHOW
Image

100-faction battle with thousands of ships.
fuuuuuuuuuuuuuuuck. so you're also saying we'll get battles not only on this scale
Spoiler:      SHOW
Image
but also on this
Spoiler:      SHOW
Image
and this
Spoiler:      SHOW
Image
?
Image
Challenging your assumptions is good for your health, good for your business, and good for your future. Stay skeptical but never undervalue the importance of a new and unfamiliar perspective.
Imagination Fertilizer
Beauty may not save the world, but it's the only thing that can
Post

Re: Saturday, July 29, 2017

#15
Cornflakes_91 wrote:
Sat Jul 29, 2017 10:34 pm
That is some beautifully optmised mesh there :D

(If only 3DSmax would produce such statisfying symmetry with boolean operations... main reason i do everything with surface extraction instead of additive/subtractive booleans, dont want to deal with cleaning up the mesh after booleans....)

Also very cool work on the ai combat, cant wait to see the formation thing going on in full 3d environments :squirrel: :monkey:
Interesting to hear you say that...I have always idolized surface-based CSG (i.e. not volume extraction) as the 'ultimate,' but now that I think about it, I suppose adaptive extraction does make more sense for complex CSG hierarchies. In a sense, it's like lossy compression, and will end up saving a huge amount of relatively-unnecessary tris in the case of a complex hierarchy. Cool :monkey:
vector67 wrote:
Sun Jul 30, 2017 4:25 am
After reading up on SDFs and the difficulty of feature extraction from them, I am very impressed by those results. Also the AI stuff Sean is working on looks like fun. I'm so happy that LT dev is speeding along. I can't wait to be able to play it (though in all honesty, I suppose I can wait since I have been for almost five years)
Yeah, this library is serious stuff. I believe it's using dual contouring with adaptation and manifoldness fix-ups.
nobb wrote:
Sun Jul 30, 2017 4:56 am
could we have the name of this magic library ? good open source code should be shared and celebrated! (and it might be relevant to my work, too :-) )
OpenVDB. It appears to be part of the VFX industry's standard toolkit, which makes us feel pretty good about the quality :)
Victor Tombs wrote:
Sun Jul 30, 2017 2:46 pm
Of course we are, BFett....but too much praise is never a good idea. ;) :)
Thanks for keeping me in line Victor, we all know nothing good comes from Josh getting a big head :roll: :ghost: Also I'm not sure why BFett is impressed considering that's just the same old nebula algorithm, except without the usual nice post-processing effects & HDR (since that screenshot is from a very minimal sandbox). But hey, I'll take the praise... :angel:
AlexWasLike wrote:
Sun Jul 30, 2017 2:47 pm
BFett wrote:
Sun Jul 30, 2017 1:42 pm
Is no one else impressed by how beautiful the background of that mesh is? I love those butt like nebula. That 10v10 fighter Gif is also very neat. Thank you for showing us this neat stuff Josh!
I have a browser extension called Cloud-to-Butt that changes the word "cloud" to the word "butt" that somehow still manages to catch me off guard. LOL.
LOL. Cloud-to-Butt strikes again :lol: And thank you! :)
Hyperion wrote:
Sun Jul 30, 2017 7:32 pm
Far more detail and complexity AS WELL AS substantial decreases in rendering cost AS WELL AS optimal adaptive LOD? so what you're saying is that we could potentially have things like this
<many requests for speculation>
We'll have to wait and see. God forbid I make promises I can't keep :roll: But ideally: yes. Just yes. That is all :ghost:
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford

Online Now

Users browsing this forum: No registered users and 5 guests

cron