Hello,

I have not read each topic, but i have not found the answer,

Will the LT Engine be multicore based ?

Almost certainly, since most computers on the market now are released with dual- or quad-cores.

It'd be a shame to squander all of that power.

Phibrizo wrote:Hello,

I have not read each topic, but i have not found the answer,

Will the LT Engine be multicore based ?

Yes and no. The primary simulation loop will not use multiple cores, because doing so would increase the complexity of the code immensely. Procedural asset generation will, however, be threaded. So while only one core will really be "playing" the game, other cores will be working in the background to make sure that loading screens are minimized.

Ok, thank you for the answer

COOL! I have a six core 2.7G system. This game should rock on this computer.

JoshParnell wrote:Yes and no. The primary simulation loop will not use multiple cores, because doing so would increase the complexity of the code immensely. Procedural asset generation will, however, be threaded. So while only one core will really be "playing" the game, other cores will be working in the background to make sure that loading screens are minimized.

I'm actually kind of surprised at this. In much the same way that blender can split up graphics rendering, I'm surprised you don't split up the simulation loop rendering to make that work faster. Maybe something like using a binary tree and assigning different parts of it to a separate core. I smell recursive multi-threading (something that I have fun playing with when programming threaded fractals).

But you're the boss, Josh. I understand the complexity involved. I've written some programs for clusters and unless you have a very tight grip on what you're doing, you're having memory leaks up the wazoo. I've been scolded by my college multiple times when I accidentally brought their cluster to a halt with an accidental fork bomb.

DWMagus wrote:I'm actually kind of surprised at this. In much the same way that blender can split up graphics rendering, I'm surprised you don't split up the simulation loop rendering to make that work faster. Maybe something like using a binary tree and assigning different parts of it to a separate core. I smell recursive multi-threading (something that I have fun playing with when programming threaded fractals).

But you're the boss, Josh. I understand the complexity involved. I've written some programs for clusters and unless you have a very tight grip on what you're doing, you're having memory leaks up the wazoo. I've been scolded by my college multiple times when I accidentally brought their cluster to a halt with an accidental fork bomb.

The only thing that I could possibly consider is to put simulation on a different core from rendering. I could never consider splitting simulation across several cores, as this would mean requiring a lock every time I access an external object, which would slow things down immensely, not to mention increase code size by at least 10%. If you have an easily-visible dependency hierarchy like a binary tree, you're right, things are easy. But my entities are nowhere near close to that dependency-free..they just can't be. For example, many entities need to keep track of which other entities are nearby. That functionality alone creates an inseparable coupling between all entities that require such functionality. Which completely deletes any tree structure that might have existed, and would necessitate an absurd amount of locking to allow concurrency.

As for simulating and rendering on different cores, I will keep this in mind...it may be less painful than I think. It would just require locking the object that's being rendered, which doesn't sound too terribly painful.

But getting threading right, as you probably already know, is one of the most painful things one can try to do in programming. And also one of the fastest ways to introduce incredibly nasty bugs. Threading bugs are the absolute worst, because they may occur all the time, 10% of the time, or .001% of the time. It's virtually impossible to debug a bug that happens .001% of the time, but that kind of thing becomes possible the moment you introduce threading, since runs are no longer deterministic.

So, I have to be extremely cautious when dealing with threading, because I simply don't have the manpower to debug it, unless it's a very simple, very tightly-controlled system like "send an asset request to the generator" / "get an asset back from the generator". But, I promise that I will use it where it's practical/possible!

PS ~ As an aside, it's not even clear to me that splitting the simulation would yield higher performance. As I said, the amount of object-object dependency would necessitate multiple lock acquires/releases for each entity, for each simulation tick. Which means multiple (evil) kernel calls for each entity, for each simulation tick. Off the top of my head, I'm going to say that the overhead of that will far outweigh the gain of threading..

What about threading not based on system, but based on distance? Assuming you're using a sector or system based model, the player obviously is only able to "see" a tiny part of the universe (that needs to be simulated at high fidelity). Everything that is not going on in the current system can be simulated at a much lower degree of fidelity and larger time scales (synchronization/locks every few seconds for example), and in that case could be ameable to multicore. Decision routines, strategic planning, etc -- all that stuff can occur in the background, at long time scales with comparably long intervals until synchronization is necessary. Right?

Yes, I agree that multithreading rendering does not make sense (procedural generation does)

JoshParnell wrote:PS ~ As an aside, it's not even clear to me that splitting the simulation would yield higher performance. As I said, the amount of object-object dependency would necessitate multiple lock acquires/releases for each entity, for each simulation tick. Which means multiple (evil) kernel calls for each entity, for each simulation tick. Off the top of my head, I'm going to say that the overhead of that will far outweigh the gain of threading..

Ah, locks. I completely forgot about those.

I also forgot regularly about that during runtime on the cluster and sometimes wondered why my code worked on my (single-core) old PC and didn't on the cluster. I think I'm a danger to myself sometimes when it comes to coding.

Thanks for the more in-depth explanation, Josh.

JoshParnell wrote:But getting threading right, as you probably already know, is one of the most painful things one can try to do in programming. And also one of the fastest ways to introduce incredibly nasty bugs. Threading bugs are the absolute worst, because they may occur all the time, 10% of the time, or .001% of the time. It's virtually impossible to debug a bug that happens .001% of the time, but that kind of thing becomes possible the moment you introduce threading, since runs are no longer deterministic.

So, I have to be extremely cautious when dealing with threading, because I simply don't have the manpower to debug it, unless it's a very simple, very tightly-controlled system like "send an asset request to the generator" / "get an asset back from the generator". But, I promise that I will use it where it's practical/possible!

This is a wise decision!
I lost 2 weeks of life to find a bug in multithreading , and my project was not nearly as huge as the Limit Theory.

jimhsu wrote:What about threading not based on system, but based on distance? Assuming you're using a sector or system based model, the player obviously is only able to "see" a tiny part of the universe (that needs to be simulated at high fidelity). Everything that is not going on in the current system can be simulated at a much lower degree of fidelity and larger time scales (synchronization/locks every few seconds for example), and in that case could be ameable to multicore. Decision routines, strategic planning, etc -- all that stuff can occur in the background, at long time scales with comparably long intervals until synchronization is necessary. Right?

Yes, I agree that multithreading rendering does not make sense (procedural generation does)

It sounds more feasible, and I will certainly try to keep this in mind when it comes time for universe simulation (which is going to be a royal pain in the rear any way you slice it).

However, the ultimate problem of globalness is simply not solvable. If you want to run a semi-global simulation, locks are going to be required all the time. Basically, the size of the simulation set that each processor handles is the factor that limits the "globalness" of the simulation. Because if I want my traders to consider trades in the next system over (which is being processed on a different core) - and I certainly do - then they will need to acquire a system-wide lock on that system, just to be able to obtain a list of tradable entities in the system. Sounds more complicated than it should be, right? But if you don't do it, you'll run into exactly the kind of .001% bugs that I was talking about. It may work 99.999% of the time without a global lock. And then a missile being processed on one core goes out of existence at the exact same time that a trader on another core decides to consider trading with the containing system...and we get a crash. Highly improbably, highly unrepeatable, and highly untraceable.

Now, if we restrict the simulation to "regions," for instance, by saying that factions will never consider expansion beyond their local region, traders will never consider trades past their local region, and then we run entire region simulations on a core...then we might be in better shape. There's still the problem of warping into/out of systems, but that happens at a low enough frequency to be acceptable for system-wide locking. I'm still considering how global I want the simulation to be, and your suggestion may work nicely if I decide on something like a local-region-only simulation.

So. Thanks

How about ... a snapshot? Similar to what Windows Shadows Copy does, do a snapshot of the local system at minute resolution (or whatever timescale) and have the threaded off-system AI work on that. This is actually the more realistic approach, since the AI shouldn't have omnipotent knowledge of conditions in all systems -- only based on data that it gathers. So, if instance, the AI is scheduled to attack a player ship (in system) from off-system, it's going to use a snapshot with the player ship in the current system. If you on the other hand warp out (whether by coincidence, long-range jump sensors, etc), the AI doesn't know. Only when it arrives in that system, doesn't find your ship, does the eval routine run again. If an AI trader is scheduled to dock at a station in your system, and you destroy the station, again it wouldn't know ... until it arrives at the destination with a burning hunk of space debris.

Using your regions example, you could make intra-region systems synchronize at a fast time scale, and inter-regional systems synchronize only very slowly. And have inter-regional distances be VAST compared to intra-regional distances (which actually makes sense in real life). After all, there aren't that many people updating starmaps all day. And it makes sense from an intelligence-based perspective.

Conclusion: If you can't make it absolutely concurrent, maybe it shouldn't be that way to begin with. In programmer speak: Implement a system to handle concurrency failures gracefully. (again easier said than done).

Just something to consider. Feel free to use or ignore at your leisure.

PS this dev blog from EVE may be useful. http://community.eveonline.com/devblog. ... og&bid=286

jimhsu wrote:How about ... a snapshot? Similar to what Windows Shadows Copy does, do a snapshot of the local system at minute resolution (or whatever timescale) and have the threaded off-system AI work on that. This is actually the more realistic approach, since the AI shouldn't have omnipotent knowledge of conditions in all systems -- only based on data that it gathers. So, if instance, the AI is scheduled to attack a player ship (in system) from off-system, it's going to use a snapshot with the player ship in the current system. If you on the other hand warp out (whether by coincidence, long-range jump sensors, etc), the AI doesn't know. Only when it arrives in that system, doesn't find your ship, does the eval routine run again. If an AI trader is scheduled to dock at a station in your system, and you destroy the station, again it wouldn't know ... until it arrives at the destination with a burning hunk of space debris.

Using your regions example, you could make intra-region systems synchronize at a fast time scale, and inter-regional systems synchronize only very slowly. And have inter-regional distances be VAST compared to intra-regional distances (which actually makes sense in real life). After all, there aren't that many people updating starmaps all day. And it makes sense from an intelligence-based perspective.

Conclusion: If you can't make it absolutely concurrent, maybe it shouldn't be that way to begin with. In programmer speak: Implement a system to handle concurrency failures gracefully. (again easier said than done).

Just something to consider. Feel free to use or ignore at your leisure.

PS this dev blog from EVE may be useful. http://community.eveonline.com/devblog. ... og&bid=286

Now this is a very interesting idea that I have never thought of before! Obviously it increases the complexity in many ways, but it's a very appealing and elegant solution, probably the least complex possible. This moves the work into the maintenance of shadow copies. I don't look forward to deep copying a region, simply due to the dependency graph. Definitely a non-trivial operation, but no worse than serializing it I suppose.

But it also sparks several ideas in my head, including hierarchical shadow copies. Since we'll be using this for far-off simulation, the simulation will run at a lower LOD and should not need every detail of every entity in the system...so we really only need deep copies of important entities. Furthermore, we can define lightweight versions of all entities, and only shadow copy these, since we won't need ultra-specific details about the entities.

Anyway, thanks again for the quality ideas

The biggest thing we need to consider is mostly Josh's time frame and what he believes he is capable of.

There is a lot of variance between different computers, cpus, cores, threads, etc. The idea behind multi-threading is to break up a large task into smaller tasks so that it is easier to process. This is true for large-scale operations where it is best to complete a task in the fastest way possible.

Where this gets into a problem for LT is that even if Josh is able to utilize multiple cores, and it is the most efficient ever, what it boils down to is that it means that on faster/multiple core machines the simulation will run... better? faster? It becomes a no-brainer.

We already know that a hexacore current gen AMD will run rings around a P4 machine. In the actual amount of 'work' to be done, it will be the same regardless of the machine you're on. i.e. The simulation will run at the same speed with the same LOD regardless of your machine. In that case, the amount of 'work' that needs to be done is static. You're only breaking up a finite amount of work to be processed faster. If you don't, then either the static amount of work gets done faster, or it is broken up to be more efficient.

Either way, it comes down to not whether or not the game is multi core, multi threaded, etc. It comes down to not needing it without severely changing the gameplay depending on your machine.

Not sure if I alliterated that well enough to convey what I'm talking about.