Return to “Dev Logs”

Post

Saturday, June 3, 2017

#1
Saturday, June 3, 2017

Really exciting week; the whole team was on-point this week if I do say so myself! Quite some major developments :) With any luck I won't spent so much time on the log this week, since there's valuable work to be done and I'm under a lot of time pressure here with this whole concept of 'getting certain things done by a certain time' -- a strange and foreign concept that Adam has convinced me has substantial value to it :V

---

Josh: ECSv2

In the last log, I talked about how we had to go back to the drawing board with our entity component system, since, unfortunately, things were starting to break down in LuaJIT, necessitating pulling work over to C. Most of my time has been spent this week nailing this problem. On Thursday, a turning point came when I had finished enough of the system to start running initial tests in LJ. For me, it was one of the most exciting moments in months when I saw the results. Basically, our ECS design works really, really well AND interfaces with LuaJIT REALLY well. The C itself runs faster than I expected. But the LJ <-> C interface? Multiple orders of magnitude faster than I thought possible. It blew my expectations clear out of the water. So, let me lay down some fun facts about where we are with respect to the system that will support all of the entity data in LT. First, the C-side:
  • Logic that runs on the C-side is stupidly fast...
  • ..it is also easy to thread. I'm already threading the C-side logic. Yes, LT is actually going to use your cores :)
  • In C we can perform non-trivial, per-frame logic on >100K entities @ 60 FPS with a good deal of headroom on a modern CPU (using threading).
  • Depending on how heavy the logic is, we can perform it on the order of 10ns to 100ns per entity. Kinematics updates, for example, are currently clocking in at ~30ns (3ms for 100K entities).
  • All of this is based on using non-hard-coded entity types. That means new entity types added by mods enjoy the exact same performance characteristics.
  • New component and entity types can be modified and added dynamically. The ECS was designed from the ground-up to ensure that we were not hard-coding anything that we didn't absolutely have to.
  • Creating and destroying entities is virtually instantaneous due to the memory pool design I built for entity data storage. 10K missiles created in one frame is boring work for ECSv2.
Now, the part that has always been more problematic. How about LuaJIT?
  • Logic that runs in LJ is 'on the order of' C speed (!!!!).
  • LJ logic can update tens of thousands of objects per frame without issue. This will improve when LJ gains threading.
  • ..oh yeah, since all data is stored and managed C-side, we can thread the Lua as well ;)
  • Lua code accesses and modifies entity data natively, with zero overhead, exactly like C code would. In other words, myShip.Integrity.health = myShip.Integrity.health - damage. We can write that in Lua, and the instructions emitted by LJ are 'basically' the same as what Visual Studio / GCC / Clang would emit.
  • Speaking of emitting, all Lua code is currently running as JITed assembly in my tests. The new ECS <-> Lua interface was designed to ensure that LJ is able to JIT everything, and never have to fall back to the interpreter (which is why we can achieve C-like speed.)
I realize that's a lot of technical stuff to digest. Let me put it simply: FPLT is solved. Not worked-around. Solved. Those test figures I gave above? They represent substantially more work that will typically be needed in LT. Remember when I talked a long time ago about having performance 'headroom'? Well we've now got plenty of it. All it took was...a better understanding of LuaJIT, a better ECS design, an automatic ECS->LJ binding mechanism, multithreading, and clever, cache-aware memory pools. Yep. Turns out that's 'all there was' to solving FPLT :V

A huge weight has been lifted from my shoulders. I can now say with great confidence that C + LJ is our solution. I can now write gameplay code with confidence that it won't be thrown away when I figure out that it performs poorly. Really, this is just a tremendous leap forward :D :clap: Adam and I are absolutely stoked that our design hit the nail on the head.

Adam: BSP Trees

Adam has spent the week becoming the resident master of BSP trees and narrow-phase physics. He's been really, really thorough about implementing them in such a way as to avoid common pitfalls (BSPs are well-known for their robustness issues -- it's easy to get them to 'mostly work,' another thing entirely to get them to 'really work'). At this point, we've got fast and accurate raycasts on complex geometry thanks to his BSPs. But it's not over yet: with robustness locked-in and hammered-out, Adam's going to start working on optimizing the BSP construction algorithm to ensure that we get the best performance possible for collision checks against annoyingly-complex objects like stations.

I'm really happy that someone on the team is taking the time to understand the nuances of BSPs (and, in general, robustness in bounding volume hierarchies). I'm certain it's going to pay dividends. After all, with the ECS now consuming so little of our frame time, the next-biggest performance hog is likely to be physics, and Adam's on track to make sure we cut it down to the same blazing efficiency that we managed to pull out of the ECS.

It's also worth mentioning that Adam is really good at building small testbeds that visualize results and show us when things are off, even if only by just a hair. This is exactly the kind of thing that I wouldn't do if I thought my implementation was 'mostly working,' and exactly the reason I was impressed with his rigor from week 1. Thanks to these visualizations he's caught a lot of edge-cases and robustness issues that would likely have gone unnoticed otherwise.

Sean: Colony Dynamics and Working Market Simulations

Sean continues to delve into all of the high-level simulation workings of LT, still focusing primarily on colonies. This week, however, I sat down with him for quite some time to explain the way the 'real' markets work in LT (bids/asks), the way it relates to AI, how colonies play into that, etc. I implemented a few basic objects for him in Lua to help with his simulation, the most important of which was a quick implementation of a dynamic market that keeps track of bids / asks, resolves them in the correct order, etc.

From there, Sean has been able to set up his first working economy simulations! At the moment, he's looking at a single, on-colony marketplace and how the colony's supply & demand drives the local market, as well as how external factors influence it. He's also experimenting with the pricing/valuation algorithms and how the AI's ability to changes its own idea of how much any given item is worth affects the global simulation. It may be a small-scale simulation, but to me this is a major step: from here, all he needs to do is continue adding elements to the simulation, studying how they affect it, and adjusting his course appropriately. I'm eager to see this work take off!

---

As you can see, the LT train is showing no signs of slowing down anytime soon! I'm pretty amazed at how much we've burned through in only a few weeks. But the burning must not let up! In the coming week, I'll be attempting to close out ECSv2, with a focus on continued performance studies in LJ. With luck I'll be starting on a new milestone near the end of the week. Adam is looking to have sphere and bounding-box intersection tests working on our BSPs sometime in the coming week, and will hopefully also begin work on optimizing BSP construction. As for Sean, I'm looking to see the market simulation blossom as more and more of the 'real LT' mechanics are factored in and added to the equation!

See you all next week :wave:
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
Post

Re: Saturday, June 3, 2017

#2
I'm gonna need another pair of pants from this log.

>100K entities @ 60 FPS

10K missiles created in one frame is boring work for ECSv2
:wtf: :wtf:
Spoiler:      SHOW
Image
Does threading mean the possibility of using multiple cpus? LT on a supercomputer?

And Get Sean in IRC! We must break him in!
Image
Challenging your assumptions is good for your health, good for your business, and good for your future. Stay skeptical but never undervalue the importance of a new and unfamiliar perspective.
Imagination Fertilizer
Beauty may not save the world, but it's the only thing that can
Post

Re: Saturday, June 3, 2017

#4
Hyperion wrote:I'm gonna need another pair of pants from this log.
:clap:
Hyperion wrote:Does threading mean the possibility of using multiple cpus? LT on a supercomputer?
Yes that's what I was talking about :) SATURATE ALL THE CORES!! :twisted:
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
Post

Re: Saturday, June 3, 2017

#5
JoshParnell wrote:
Hyperion wrote:I'm gonna need another pair of pants from this log.
:clap:
What can I say, you have VERY talented hands ;) :shifty:


Now, with ECP being orders of magnitude faster and more efficient, if you can make the Physics just as efficient, does this open the possibility for orbital mechanics?
If you can get the ECP and Physics bottlenecks out of the way, then what do you think would be the next bottleneck? AI? Graphics? The fidelity of simulation in distant systems?
Image
Challenging your assumptions is good for your health, good for your business, and good for your future. Stay skeptical but never undervalue the importance of a new and unfamiliar perspective.
Imagination Fertilizer
Beauty may not save the world, but it's the only thing that can
Post

Re: Saturday, June 3, 2017

#8
Josh, after LT is done. I am going to have to ask you to go and work with the guy who made Children of a Dead Earth, and make ultra-realistic space combat also ultra-fast.


As for this post?
Well... there aren't exactly words for it. Good job.
A scripting language that runs on the order of a compiled language? THAT is the dream. Can you do it to JS? :V
°˖◝(ಠ‸ಠ)◜˖°
WebGL Spaceships and Trails
<Cuisinart8> apparently without the demon driving him around Silver has the intelligence of a botched lobotomy patient ~ Mar 04 2020
console.log(`What's all ${this} ${Date.now()}`);
Post

Re: Saturday, June 3, 2017

#9
Been a while since I've made a post on the forums, mostly been stalking IRC... but it's great to see everything happening with the development of LT. And dare I say we can now call it a dev TEAM! I'll be lurking as always and I can't wait to see what the team does!
:clap:
The only limit is the one you set.
Post

Re: Saturday, June 3, 2017

#11
WOW! I'm literally blown!

Congratulations on your really great work and I'm REALLY happy for you, that it worked out so good and that it runs faster than you expected!

This post will mark the end of my daily posts here in the forum. I'll only be able to check the forum every Saturday and Sunday starting from now for the next two months as already stated in the "I'm gonna be away..." thread in the "Everything" topic. So, yeah. I wish you the best of luck with your development and I'm looking forward to getting blown away by your great progress news every weekend (no pressure here :ghost:).
Automation engineer, lateral thinker, soldier, addicted to music, books and gaming.
Nothing to see here
Flatfingers wrote: 23.01.2017: "Show me the smoldering corpse of Perfectionist Josh"
Post

Re: Saturday, June 3, 2017

#12
Silverware wrote:Josh, after LT is done. I am going to have to ask you to go and work with the guy who made Children of a Dead Earth, and make ultra-realistic space combat also ultra-fast.


As for this post?
Well... there aren't exactly words for it. Good job.
A scripting language that runs on the order of a compiled language? THAT is the dream. Can you do it to JS? :V
You arent allowed to use the multithreading because you always say "nobody needs multithreading, get a better CPU".

:V

JoshParnell wrote:
Hyperion wrote:Does threading mean the possibility of using multiple cpus? LT on a supercomputer?
Yes that's what I was talking about :) SATURATE ALL THE CORES!! :twisted:
Also, how many threads is "all the cores"?
Number of independent scripts running?
Number of objects on which run scripts? (Whyever that'd be)

Online Now

Users browsing this forum: No registered users and 7 guests

cron