I have to agree with Gazz.
If you model an infinite game like any X-game, you won't have a lot of fun.
X biggest problem is, that everything is always calculated, the hole time. The bigger your game gets, the more assetts an player owns the slower the game will work, especialy if you use heavily scriptet mods and their cpu consuming comands.....

For an infinite game i can only see an form of events. Things are only calculated, when an event happens.
Why should my assetts be calculated the hole time?
If i am browsing my assetts list for example, this is an event where my assetts should be updated, not always!
Or take an players mining station for example. There will be a chance that it gets attacked, higher in outlaw space, lower in more cicilised area. If this attack event triggers, then the combat will happen and will be calculated, but only once for exactly this situation. BTW an good example for modding, to raise or lower the chance for attack modifiers.

LT works on a simulation level-of-detail scheme, such that far away events are simulated at a coarser level of detail than nearby events, and the details are resolved in more and more detail as you come closer to the system.

IMO, simulation time is not as big of an issue as people make it out to be. I am unsure of why X had such performance problem, but my intuition tells me that it must have been due to everything running in a script interpreter. C++ is very, very, very fast. Things that don't get executed tens of thousands of times per frame tend to be of almost no consequence (barring graphics calls, mind you). In a script interpreter, that totally changes, because even trivial things can become two orders of magnitude slower, so it's not just the inner-of-innnermost loops that's bottlenecking anymore, it's the whole thing.

So I don't anticipate that resolving a battle will be too CPU-intensive to do in real time across a large portion of the universe. Even if a battle were to occur every frame (which it won't), it still won't be that big of a deal. It's just some math, which, at the end of the day, is a drop of water in the ocean of physics calculations that are happening to see if your pulses are hitting asteroids or not. Seriously, the math for resolving a battle is probably significantly less intense than the math for checking collision on a single beam weapon. Now, will it be madly frustrating trying to tweak this simulation math to get it right? Oh yes. It will no doubt drive me insane.

But I don't think performance will be the main problem

Thankfully, Gazz is our local expert on OOS simulation, and he has already given me a good bit of insight on this process. So it's nice that we have help

JoshParnell wrote:LT works on a simulation level-of-detail scheme, such that far away events are simulated at a coarser level of detail than nearby events, and the details are resolved in more and more detail as you come closer to the system.

Does this mean that when:
* you travel through a system, finding out details along the way (e.g. specifics about NPC's, space stations)
* then go on to explore at great distances where the first system moves to a coarser level of detail, say a level where it is simulated that a war took place there, but not the details
* that if you travel back to the system as you get closer the back story will be filled in, so that by the time you get there you can find out what happened to the NPC's that you knew earlier etc.

I can imagine the devil will be in the details, but getting this right would make for a very compelling experience.

JoshParnell wrote:LT works on a simulation level-of-detail scheme, such that far away events are simulated at a coarser level of detail than nearby events, and the details are resolved in more and more detail as you come closer to the system.

IMO, simulation time is not as big of an issue as people make it out to be. I am unsure of why X had such performance problem, but my intuition tells me that it must have been due to everything running in a script interpreter. C++ is very, very, very fast. Things that don't get executed tens of thousands of times per frame tend to be of almost no consequence (barring graphics calls, mind you). In a script interpreter, that totally changes, because even trivial things can become two orders of magnitude slower, so it's not just the inner-of-innnermost loops that's bottlenecking anymore, it's the whole thing.

So I don't anticipate that resolving a battle will be too CPU-intensive to do in real time across a large portion of the universe. Even if a battle were to occur every frame (which it won't), it still won't be that big of a deal. It's just some math, which, at the end of the day, is a drop of water in the ocean of physics calculations that are happening to see if your pulses are hitting asteroids or not. Seriously, the math for resolving a battle is probably significantly less intense than the math for checking collision on a single beam weapon. Now, will it be madly frustrating trying to tweak this simulation math to get it right? Oh yes. It will no doubt drive me insane.

But I don't think performance will be the main problem

Thankfully, Gazz is our local expert on OOS simulation, and he has already given me a good bit of insight on this process. So it's nice that we have help

I give you mad props for this. Everything I've seen always takes massive amounts of time. You may want to actually publish some white papers regarding it. I'm sure there are many developers out there who would love to implement something similar, and this is also a way to get your notoriety up.

I believe this is also how some of the other big names got their start too, John Carmack for example, with his shortcutting of 3d mechanics for Doom.

JoshParnell wrote:To enable modding I would have to set up a script interface for pertinent objects and functions, and I don't think I have time to do that! The only other option is to allow DLL modding, but then I have to release all headers and give instructions for compilation and such, and I think it will not be very accessible for most of the audience.

I registered just to ask about the scope for modding then searched and found it. Just one follow up question, given that, as we all know, the X scripting engine is both the X series strongest long term asset and it's Achilles heel, do you think you'll be able to follow up on the second option quoted above? I'm pretty certain that adding a scripting engine would be a disaster at this point, as you'd probably have to violate your code base horribly to make it work, but there are many developers who don't work in gaming but who are always interested in playing around to extend the game. I should say I think that that would be a fine post-release addition: it'll take a few months before people start to say "man, I wish LT had <insert feature here>", whether that's fighter drones, space marines, cloaking devices, or just outputting stats to a G19 keyboard display (or whatever else our crazy minds can dream of).

In the end though, only you can look at the big picture of your code and decide if, even with some of the header files, anything like this is even remotely possible.

JoshParnell wrote:I am unsure of why X had such performance problem, but my intuition tells me that it must have been due to everything running in a script interpreter.

Got it in one.
X3 does not just have a script engine. It has two on top of the "real code". Players only have access to the outermost layer.
What is commonly referred to as "hardcoded" instructions runs on one of the inner layers. This includes navigation commands like "fly to this position", "evade for 300ms", or the single instruction that executes one round of OOS combat between two ships.

All "AI" scripts (few exceptions) of all the ships in the universe are done in the outermost script language, which is the reason why players have been able to change so many high level AI behaviours in so many ways. While the actual OOS calculations run extremely fast, the engine has to dig through most of a 500 line "AI fight script" until it gets to that one line.

I never looked intot he x scripts much. Is it a bunch of if statements, or is there a finite state machine involved?

I'm not sure what you are asking there.
If-then-else constructs have nothing to do with the type of "machine".

Most scripts are written as a FSM because actors have a finite number of properties and states such es "evading" or "attacking".

You could just as well make arguments for a Turing machine because the number of instructions is (virtually) unlimited - only constrained by physical memory. I have written scripts with several thousand lines.
On a macroscopic level, an actor calls scripts in a fixed sequence so they could be seen as instructions on an infinite tape.

I have also written scripts based on fuzzy logic. Mostly AI stuff where finite states would result in (logical) dead ends and predictable behaviour.

In any kind of code, the type of "machine" is not some hardcoded thing but rather how you structure it.
It's basically a philosophical question because those models are extremely abstracted. In actual code you usually get a mix of many "machines".

Thanks for confirming Gazz, guess it's not surprising.

It's a tough trade-off, really :/ Hard to get performance and extensibility into the same engine. Like I mentioned earlier, it's possible that LT could be scripted through native libraries (.DLLs / .SOs / whatever mac is...), which would, conveniently, suffer no performance hit...but I'm pretty sure most people here aren't going to be willing to set up a compiler and jump through all the build hoops, as well as learn c++ just to extend LT.

(well, I'm fairly certain that you would do it, but still...)

Oh, I'm familiar with trade-offs.
When I wrote a complex fuzzy logic based script for X3, I ended up rewriting major parts of it over the course of several months... twice... until I could get acceptable performance. Now it's heavily compartmentalised with up to 7-10 separate tasks (rarely up to 20 on a capital ship) running on a single ship and communicating with each other. Very much mad scientist style but that's how I got it working on an engine that was totally unsuited for it. =)


C++ isn't the big hurdle. I've been working in all sorts of different languages over the years and code always stayed the same.
Data types, expressions, and instructions / function calls. Minor differences in syntax.

So far I haven't used a compiler but many modding systems require you to wrap scripts and content into some sort of "package" that the engine can digest. Same thing from a user perspective.

It's only problematic if the process isn't documented and modders have to figure it all out by trial and error. Or if the designer throws extra wrenches into the works, like a phone-home feature being activated a month after release and invalidating modded content. (XCOM)


Obviously, there is an easy solution to the problem, too.
You just create the perfect game and no one would ever want to mod it.
But, you know, no pressure, okay?

Interesting..that's a lot of scripts! I had to do something similar when I was modding Morrowind...so many hacks just to make basic things work! In some ways, it makes you proud when you step back and say, "yeahhh, I just totally accomplished something that they never intended to be achievable with that system"...and in other ways, makes you cry, because the same thing should have taken about 5 minutes in native code, but it took you a month of work to get the scripting engine to comply...

Anyway, we'll see what happens with modding. I just got an idea that maybe I can make the scripting interface automatically generate itself using preprocessor magic...hmmm..maybe I will explore that. I'm not going to lie, I'm pretty much a beast at exploiting the preprocessor....a macro-magician, if you will

JoshParnell wrote:(well, I'm fairly certain that you would do it, but still...)

The first thing that would happen is somebody writing a script adapter in a native library and then you will end up with the same scripts as if you just integrate lua or something like this from the beginning. I checked what folks from Privateer project do and also what those crazy Russians from FFE D3D project do and they all try to integrate lua for mission scripting. This is one place where native code would probably be less flexible than some script or XML.

Scripts would be a performance problem though, so this might be an intensive to write native code instead. For me it is certainly easier to write a C++ DLL than to start messing with some crappy script engine, but I still did some minor scripting for X3. Sometimes developers think that their scripting will simplify things. It rarely does, but may be this is only for somebody who can write C++ or Java code. I still remember that "OMFG, who wrote this mess?" moment when I tried to do some X scripting first time. And then another "Are you kidding?!" moment when I realized that there is another layer of interpreted code used to provide most of the game functionality, including the scripting available to the modders.

I doubt LT will end up like this, but if you can write it with some interfaces in mind, it would be awesome.

If modding is going to be looked at seriously by Josh, I would almost want it as an external module, either as a DLL or something interpreted as a script engine. This way, you can save resources/processing time if you don't use them, while at the same time being more full-featured than just an interpreter. Kind of like how MineCraft has the different add-ons and mods that are as simple as just copying files into directories (once you get past the modding of the actual jar file that is).

I like interfaces more than interpreters, TBH, but I don't know the difficulty of coding those.

From my perspective, coding an interface would be a lot easier than integrating an interpreter, since an interpreter requires an interface to begin with, and then some. I had not considered kurush's idea that some really skilled programmers might be able to take a DLL-based modding system and use it to construct an interpreted script modding system. That would be quite cool, and could actually work...although I could see it becoming chaotic, with a hundred different types of scripts floating around based on which third-party interpreter you're using...

Still, it's a cool idea, and certainly less work for me. All good things to consider

The only issue with that would be managing the time taken for the decision process. If a script was running the AI for say one given ship (or ships commander more to the point), and that ship required a decision to be made, then the system would hand it off to the script.... which may take some time. During the time that the script took what would the ship be doing? Obviously the rest of the universe would continue to flow on, maybe making the original decision redundant.

Seems like either there needs to be a limit put on the time scripts can take.... or probably much better that the scripts would only be making high level decisions which are not so time dependant, e.g. a script may so to do this trade run, but other internal AI processes would handle things like conflict (though I guess the script could influence that by adjusting the NPC's temperament - fight to the death / run and hide).