Return to “Dev Logs”

Post

Sunday, April 23, 2017

#1
Sunday, April 23, 2017

It's not a perfect resolution of the grand war against entities, but this week has brought out the fiercest practicality that I can possibly muster, and for that I think we'll all be grateful. It is time to move on, dear friends, from this monstrous black hole of time and effort. Brighter prospects lie ahead.

---

Entity Component System: So...we're all tired of it by now, right? Me too. After many more hours drifted by this week with no sign of bringing me the end-all-be-all solution I've been wanting, I made what I consider to be a very practical compromise: I said "screw it." Since such crass language isn't allowed in my treasured notebooks, I instead wrote something marginally more profound: "design for the rule, not the exception." I'm sure someone has said that before. But now I'm saying it again. After enough neural burning on the problem, I decided it's rather absurd to continue this temporal bonfire (yes, LT development sometimes involves lighting time itself on fire) when the edge cases, not the majority ones, are holding back the design. I banged out an ECS that covers the 90% and gave myself permission to worry about thruster trails later. Here's my theatrical rendition of the past two weeks (Boss is, of course, practical Josh in disguise):
Boss : "Build me that ECS."
Josh : "Aye sir."

< 1 week passes. >

Boss : "Where's my ECS."
Josh : "Working on it."
Boss : "Get it done kid."

< 1 week passes. >

Boss : "Where's my ECS."
Josh : "I have a lot of good designs."
Boss : "Neat. So where's my ECS."
Josh : "Well sir, currently none of them can elegantly accommodate the fact that thrusters have a key part of their rendering behavior tied to one little piece of state that should, in earnest, live on the Lua side, as it's really not 'intrinsic' to anything. Sadly, in the case of thrusters, there are enough of them that the data martialling will become problematic and result in framerate loss. On the other hand, it's difficult to see where this state should live on the C side of the system."
Boss : "...that right? So...just...thrusters..."
Josh : "Well sir the general problem is likely to crop up in a few other pla--"
< Boss raises hand so as to silence Josh. >
Boss : "It's all so...incredible. So fascinating. But see here, look Josh, let me ask you just this one thing."
Josh : "...yes sir?"
Boss : "WHERE. IS. MY. <expletives censored> ENTITY COMPONENT SYSTEM?"
Josh : *Quickly hands over imperfect but functional system*

< Boss storms away angrily. Cornelius Evazan enters from stage left. >

Cornelius : "You just watch yourself."
Josh : "I'll be careful."
Cornelius : "You'll be dead!"
And there you have it. Kinematics and a few other pieces of physics-related information, along with graphical information, live in C. The rest (requiring no per-frame updating) lives in Lua. Components are Lua-side. So really, it's just an ES that lives in C...an Entity System without the components, since I decided these are, in the 90% of cases, lightweight enough to be handled by Lua without issue. And so it will be until something starts causing major issues.

RenderQueues: A nice advancement necessitated by this week's ECS work was a core rendering feature missing from all previous version of the LT Engine: the 'RenderQueue.' Truthfully, the RQ type is little more than a 'modern' implementation of something OpenGL has had since the dawn of mankind (GL called them 'Display Lists'). Since modern man has decided that OpenGL's display lists were a bad choice to put in a 3D API, we've deprecated them (along with 90% of the GL API). Hence why I implemented them as a first-class engine feature (driver support for display lists, unlike with many deprecated features, actually is a problem). The feature is little more than exactly what it sounds like: a queue of rendering commands; essentially a data structure that encodes a miniature rendering program. This allows rich flexibility to specify rendering operations in Lua by building a RenderQueue, but allows the operations to be executed at the full, blistering speed of a tight C loop. In fact, I've some notion that, since RQs are basically implemented as mini bytecode interpreters (i.e., a decode loop with a switch statement), they might outperform my previous implementations of certain rendering operations (thanks to the fact that all of the relevant GL code is sitting right inside that hot loop; external dispatches that aren't GL calls are nowhere to be seen). But all that aside, the point is that I needed a flexible and fast way to specify rendering operations from Lua, and I've now got it.

I previously had an engine construct called "Model" that handled this type of thing; it was basically a list of Material/Mesh pairs, telling the engine what meshes to render (e.g., for a ship), and what material to use for each mesh (a material was itself a combination of a Shader and various shader parameters). Upon starting to write my Model class into the LT Core, I saw the light, realizing that all of these various constructs that I used to have (Model, Material, ShaderState (a bunch of shader parameters)) could be unified by a structure that encapsulates exactly what they are, down to the core: just some rendering operations and various shufflings of state. Now I can think of a Material as being a RenderQueue that binds a shader and sets up some shader parameters specific to that material. I can think of a Model as being an RQ that calls a sub-RQ corresponding to a material, then draws a mesh, etc. RQs can be nested within one-another to encapsulate the exact kind of sharing that I used to think of in terms of all these disparate constructs that I no longer need! Yay unification. Rendering life is quite simpler now. To think the GL guys had this insight in 3000 B.C. or whenever it was that GL came about :monkey:

I have some dangerous notions about how we might get cute with dynamically building RQs to achieve various optimizations. I'll try to steer my mind away for now.

Mods, Content, The Works: As you'll recall, my objective at the moment is to write the Lua-side LT code in 'mod' format, so as to make life easy on myself while simultaneously ensuring that I know how everything is going to come together with respect to actual mods. I've made a good many changes and simplifications to the format this week to help myself pump out that core gameplay code. I was honestly getting a little overwhelmed and turned around with my previous system of hooks and so forth. I've simplified the mod format to be much more like a 'data description,' which makes things cleaner and easier to reason about (without any sacrifice in flexibility). This is one thing I really love about Lua. It's a beautiful language for describing data, given that the one and only 'advanced' datatype is a table, and data is, after all, just tables inside tables inside tables. Throw in the ability to bind table fields to actual code/functions, and you've got all the power you need to build a piece of 'data' that tells the engine how to handle whatever it is that the data is defining.

I now just have a 'content' sub-table within my mod description table that contains any number of entries. Those entries represent...well...the content of the mod! Right now, they can be entity type definitions (ship, station, planet, etc.), event listeners (to add functionality in specific places), component definitions (define new components for use with the ECS), and miscellaneous type definitions (I'm still not sure what to call things like 'Item' and 'Corporation' and so forth...they're datatypes of course, but don't need to be handled as 'entities' by the ECS).

Stupid example of what I'm working on right now:

Code: Select all

return {
  id      = '_LTBE_SYSTEM',
  name    = 'Limit Theory Base Entities: System',
  desc    = 'Defines the System Entity',
  author  = 'Josh',
  version = 0.01,

  content = {
    {
      type = 'Entity',
      id = 'System',
      components = {
        'CollisionDetector',
        'EntityContainer',
        'NaturalResources',
        'Zones',
      },
    },

    {
      type = 'Listener',
      id = 'System.onCreate',
      fn = function (Game, e, seed)
        print('Looky, a system!')
        local rng = RNG.Create(seed)
        e.name = 'Omicron ' .. tostring(rng:getInt(10, 20))
        e.starDir = rng:getDirection3()
        e.starColor = rng:getVec3(0.5, 1.0)
        -- Populate the system with...you know...lots of stuff
      end,
    },

    {
      type = 'Listener',
      id = 'Engine.onInit',
      fn = function (Game)
        local seed = 1337
        local system = Game:createEntity('System', seed)
        -- TODO: Implement Limit Theory
      end,
    },
  }
}
Once all content is loaded from mod files (or base files, in this case), the engine pulls together all these bits of content and resolves everything to figure out exactly what every entity consists of, what events should be fired when, etc. Less manual labor than before. FYI the content creation code there is obviously placeholder :P

It has a ways to go to be sure; we can certainly pretty up the syntax later. Right now I'm interested in getting all of this stuff working. It honestly strains my brain a bit learning to think in this new manner of events and handlers rather than the old, linear update logic. But the straightforward 'dumbness' of the old update logic is exactly what killed performance, so it's time I learned to man up and think asynchronously.

---

In the coming week I have another LT showing at LSU on Friday, so my goal will be pretty straightforward: I'd like to see LTDemo running on the 'real' systems: using the C entity code, event-driven rather than monolithic update loops, etc. I'd honestly be happy to have that all working with the same level of functionality as before, because that would mean that I'd be in good shape to rapidly move forward with all systems sitting on a solid foundation. It'd be a major step. At present I'm not entirely sure how many man-hours of work that will entail, but I do intend to find out :)

Hey everyone! Talvieno here, with another week's edition of The Non-Technical LT Devlog.

This week, Josh focused hard on the whole ECS problem and got it solved - at least to the point that it's "good enough". He now has a single unified structure with which he can handle ships, stations, asteroids, planets, and other objects. This will make implementing content much faster. OldJosh is not pleased with this development and would rather he have a complete solution, but PracticalJosh - or NewJosh - has won out here. I think this is what everyone was hoping for. :)

RenderQueues: Exactly what it sounds like. Josh has implemented a unification of rendering commands that'll make it a lot easier to progress in the future. You can think of it like a to-do list for the engine's rendering component - the part that draws everything to the screen. Everything has its own little to-do list now, and can be modified/controlled via Lua - which means it can be modded.

Mods, Content, The Works: Essentially, this is Josh talking about how he's simplified the modding format to make it easier to use. There's not really much else to be said; I think that just about sums it up. At any rate, this next week he's going to try to focus on getting the LT code up and running for another showing at Louisiana State University.

tl;dr: Finished the entity component system, and set up the framework to make it much easier to add content, as he's getting to that point.
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
Post

Re: Sunday, April 23, 2017

#5
I cannot tell you how happy I am to hear this description of the mindset for dealing with the ECS.

I understand, really I do, how painful it feels to just know that with a little more time, the conceptual breakthrough will come that finally allows the tidy unification of all the...

NO.

Fight that! Punt it into next week, then set next week on fire with a temporal flamethower.

Yes, it is probably true that stopping before achieving a completely coherent solution for a core system probably does mean deferring some hard problems to later. There may be some technical debt that must be paid.

But fully solving every single problem as it comes up misses the opportunity to let several accumulated problems solve each other (which does occasionally happen), or to find solutions that answer multiple problems at once. So there's a practical benefit to moving on after reaching a reasonable "good enough" point.

Maybe even more importantly at a psychological level, I firmly believe that "design for the rule, not the exceptions," or a "90/10 rule," or "the engineering mindset," or whatever you want to call it, is exactly the right way to be thinking. It's a tradeoff: it gives up the very intense but low-possibility satisfaction of a unification win in exchange for the less intense (but still pleasurable) and more likely win of making visible progress toward completing the whole project.

Or as engineers have put it for many years: "A good solution today is better than a great solution tomorrow."

It sounds like Pragmatic Josh has carried the day here. That is the best possible news for Limit Theory The Project. :thumbup: :clap: :angel:

Good luck getting the new demo ready for the LSU showing!
Post

Re: Sunday, April 23, 2017

#6
Just want to quickly say how much I'm enjoying Josh 3.0's Dev Logs: I don't agree with so much of the work and decisions being made, and I love that - it makes me think, makes me consider other people's viewpoints more, gives me insights into new techniques I haven't experienced directly, and it demonstrates the real underlying nature of the 3.0 version's philosophy of pragmatism.
--
Mind The Gap
Post

Re: Sunday, April 23, 2017

#7
Flatfingers wrote:I cannot tell you how happy I am to hear this description of the mindset for dealing with the ECS.

I understand, really I do, how painful it feels to just know that with a little more time, the conceptual breakthrough will come that finally allows the tidy unification of all the...

NO.

Fight that! Punt it into next week, then set next week on fire with a temporal flamethower.

Yes, it is probably true that stopping before achieving a completely coherent solution for a core system probably does mean deferring some hard problems to later. There may be some technical debt that must be paid.

But fully solving every single problem as it comes up misses the opportunity to let several accumulated problems solve each other (which does occasionally happen), or to find solutions that answer multiple problems at once. So there's a practical benefit to moving on after reaching a reasonable "good enough" point.

Maybe even more importantly at a psychological level, I firmly believe that "design for the rule, not the exceptions," or a "90/10 rule," or "the engineering mindset," or whatever you want to call it, is exactly the right way to be thinking. It's a tradeoff: it gives up the very intense but low-possibility satisfaction of a unification win in exchange for the less intense (but still pleasurable) and more likely win of making visible progress toward completing the whole project.

Or as engineers have put it for many years: "A good solution today is better than a great solution tomorrow."

It sounds like Pragmatic Josh has carried the day here. That is the best possible news for Limit Theory The Project. :thumbup: :clap: :angel:

Good luck getting the new demo ready for the LSU showing!
Yup. I really like that little story with "Boss" and Josh up there.

I know that exact problem. I get caught in trying to do my coding overly perfect. Means that I take a function and try to tailor it for every possible future use of this function, instead of just saying "Will I really EVER use this function again? Is it worth all the work?".

And I think I heard that saying "A good solution today is better than a great solution tomorrow." before... ;)

We also have another saying in electronics, as well as in measurement and control technology: "As exact as possible, as exact as needed." meaning you should keep the Pareto principle in mind.
You do 80% of the work in 20% of the time and the remaining 20% of work eat up the other 80& of your time. And you just gotta set a stop after a certain amount of time and say "It's good. It does what its's supposed to do. It may not be perfect, but it works for the majority of its uses. And for everything else, I'll approach it when it's really needed."
Automation engineer, lateral thinker, soldier, addicted to music, books and gaming.
Nothing to see here
Flatfingers wrote: 23.01.2017: "Show me the smoldering corpse of Perfectionist Josh"
Post

Re: Sunday, April 23, 2017

#15
Thank you Flatfingers and others who are encouraging 'practical' Josh...I do find it's getting easier with time and practice. In fact, I'm finding that being practical is quite less torturous than being a perfectionist...I think my life will be orders of magnitude easier if I can master it :ghost:

As to why thruster trails are special / what makes qualifies an entity as being 'annoying' from the viewpoint of the ECS:
  • We have some state that we can agree all entities need, or at least a big enough percentage that the state should live in the C structure that we call an 'entity.' Right now this state is all about position/orientation/velocity/force/mass/angular equivalents (so kinematics-related information), renderer information (a pointer to a renderqueue), hierarchy information (some pointers to other entities), and physics engine information (collision-detection related stuff).

    So far so good. This information can live alone in the C world and be content. RenderQueues have all the data they need for most objects (from Lua we can set the 'material' and mesh to use by building the RQ, within C the RQ can access the Entity's C data to build things commonly-needed for rendering, like a world matrix). There are some annoying exceptions. These annoying exceptions occur when rendering requires 'extra' data -- in particular, extra per-entity state data. Thrusters, for example, have some notion of 'activation' or 'output': a float from 0 to 1 that signifies how much force the thruster is outputting relative to maximal output. When rendered, thrusters have several components: the base model, the lens flare, and the short exhaust 'trail.' The latter two change depending on how much the thruster is outputting. There is no flare or trail for a thruster that's idle. This ramps up smoothly all the way to a rather bright flare and blue exhaust when the thruster is at max. So naturally, the shaders involved need to know about this 'output' variable on the thruster.

    In theory it's not a big deal: the base part of the thruster, the model, doesn't change regardless of activation. It's the 'special effects' that do, and we can consider special effects to be...well...'special', so we're OK with the notion that these should be rendered via Lua since they have 'special' logic. That's fine for most special effects. Wormholes, planets, warp nodes, explosions, etc -- all these have 'special' rendering logic and we're fine with relegating that to Lua. There aren't many of those things. But now coming out of theory land and into practice: there are a lot of thrusters. In fact, there are significantly more thrusters than ships. So this is one special effect that is particularly 'special' because it requires special logic but still needs to be handled in a performant way.
Honestly, I'm sure that I can solve it without too much trouble. It was my mistake in the first place to let this type of detail interfere with ECS design. Although it's important to consider all usage cases, as I said, I shouldn't be letting the 10% cause me to question a design that works for the 90%. I have plenty of ideas concerning how we can handle thrusters and any other 'high-volume' special effects that may crop up. The first one is: do nothing. Indeed, I've never actually profiled the overhead of drawing thruster trails in Lua, so why am I assuming that it will be a problem? Because I'm a silly, silly little Josh :shifty: It may well turn out to be no issue at all. And if it does cause problems, we can apply incremental optimizations that will almost certainly be 'good enough' long before having to consider such drastic things as changing ECS design!

Sometimes I'm dumb, sometimes not. I just hope for more of the 'not' days :ghost:

Re: Cornflakes / Particle Emitters: Ewww particle emitter trails are ugly and inefficient :P Yeah, they're fine for lower-graphical-fidelity games. It wouldn't look right in LT (I've tried). In order to get something that looks decent, you end up having to use a lot of particles, making it way less performant than necessary; even then, it won't look as good as a custom-made effect that uses only a few quads :) Really it's the same story for all effects. Usually you can get something that looks OK with particles (and with a lot less dev effort), but in the long run you can almost always get higher quality and higher performance with an effect tailored to the use case. (Remember when I was lazy with warp nodes and made them using a ****-ton of particles? Guess what was causing a nasty little FPS hit when I was near warp nodes....bingo... :V )
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford

Online Now

Users browsing this forum: No registered users and 7 guests

cron