Tuesday, February 28, 2017

Logging the Creation of a Beast -- the Good, the Bad, and the Template Metaprogramming.

Tuesday, February 28, 2017

Postby JoshParnell » Tue Feb 28, 2017 10:14 pm

Tuesday, February 28, 2017

A rather solid but unremarkable week; I feel this week could aptly be described as a lingering shadow of last week. For both better and worse, last week's "I started on" could basically be replaced with "I finished" or "I almost finished," and the log would be nearly complete. Thus, as has been falsely stated many times now, we may actually see a reasonably-sized devlog tonight!? (EDIT: at this point, it's a tradition...:ghost: )

---

PlateMeshes: 90+% complete in the LT core. As discussed last week, I emphasized building a release-quality implementation since they became an invaluable tool in LTC++. I now have a high-quality, fast, C library implementation of all of the old PM functionality with the exception of rotation of individual plates. I will need to spend a bit more time in order to finish that bit efficiently, but the rest of the work was by far the bulk. Frankly it required more time than I would have liked, but all of that time was spent adding supporting functionality to the core in order to efficiently support a good PM implementation. The good thing about this kind of time is that it's "permanent progress," as is all progress of the LT core (which is, as a reminder, designed to be the 'etched in stone' part of the LT engine).

Mesh 'Warps': 20% complete (I figured out how to do them). These are the things that allowed me to distort models in cool ways that added curvature and interesting global features to ships and stations. Unfortunately, their implementation in C++ using all that fancy (and incredibly-nasty) template stuff I used to do made it fairly easy. I had some trouble figuring out how to do the same thing effectively in a clean, C API (i.e., making it easy to build warping functions from LuaJIT). I've designed the implementation, about which I'm pleased. I haven't coded it yet. Interestingly, solving this design problem brought me insights on how to make reference-counting work cleanly across API boundaries (for example, to enable both the core and LuaJIT to hold pointers to the same object and still have all memory management work correctly). I've not had this insight before, but when the time invariably comes that I need to do this (beyond warps, which will require it from the get-go), I now know how to do it painlessly.

Networking & 'Control Panel': 100%/90% complete (server/client), and completely new to any of the LT engines! Yes, this is where about 2.5 days of my time went this week, but it's a technology investment that I'm betting will pay dividends. While working with planets, it occurred to me that I would really like a UI for changing the procedural algorithm parameters in real-time to see exactly how they're affecting the final result (with complex procedural algorithms it can be very difficult to reason about the effect of individual parameters). I didn't want to spend a huge amount of time building something just for this task, though, so, inspired by a friend at LTP who does web development, I decided to use a tool that already exists and is very fast and easy for creating quick UIs: the modern web browser. This necessitated networking functions in the LT core, which I had promised I would provide for modders who want to attempt 'multiplayer LT' mods anyway. So, I implemented a multi-platform TCP & UDP socket API, then used it from LuaJIT to spin up a very simple HTTP server run by in the background by the engine that serves up a simple control panel webpage when connected to via a browser. It can change internal engine variables automatically in response to controls like sliders or buttons being changed in the panel.

Though I'm hoping the ability to connect to and communicate with the engine via networking will pay substantial dividends over the remaining development time, having the control panel available is already really helpful: I have a really easy way to expose real-time-tweakable parameters, and, as a bonus, since it's based on sockets & HTTP, I can even connect remotely so that I can use one machine (like my tablet) exclusively as my 'controller' while I develop on the other. I'm also thinking this will be great for monitoring and managing long-running universe simulations (when I get to that point, ideally sooner rather than later)! I'd even like to explore using this for asynchronous resource generation on the GPU, which is traditionally difficult to do in a single process (due to threaded GL contexts being messy). There are a whole lot of ways I can envision using even this very basic networking tech :geek: Also, of course, the tools are now there (in theory) for anyone who wants to give an LT Multiplayer mod a shot at release (in practice, good luck, I recommend putting together a team :ghost:).

---

In the LTASM world, I did exactly what I had mentioned and hoped for last week: basic x87 floating point operations. Quite happy to have that stuff working. I've got enough to, for example, do most if not all of the basic FP math that you might do in the LT logic, although I don't necessarily have all of the opcodes for doing things in the most efficient way. Frankly, though, my first goal in this world of Plan Z is to have a script -> machine code pipeline working at all. That will be amazing enough for me, then we can worry about micro-optimization. My tests this week with FP instructions in simple loops are already blazingly-fast. I am getting a stronger and stronger feeling that this work will play a role in LT even if LuaJIT continues to hold. In fact, I can see it being a beautiful addition to LJ! LJ is basically a dynamic asm generator, why not have a static asm generator for perf-critical code? Man does it make a difference getting down to that raw level (I mean compared to something like Lua, not C/++).

Speaking of which, as usual, I spent some quality time with the Intel manuals this week, covering chapters 10 & 11 (SSE, SSE2), and skimming 12, 14, & 15, (SSE3/4, AVX/AVX2, AVX512). So basically, I learned a whole lot about vector / 'SIMD' instruction sets this week. Fun as always :geek: (Can you believe AVX512 has 2KB of register space for FP math?? Insane...)

---

It's worth mentioning that I spent a good bit of time this week on other, non-coding work. This week saw an abundance of emails, real-life events, and a bit more time spent with the forum than usual. Speaking of which, I assume that most of you have seen the announcement of Talvieno as Community Manager by now, but if not, check it out :) Although it took a bit of time this week to write that post and get everything squared away, his new role as CM will ensure that the amount of non-coding time spent during future weeks is kept to a minimum! I'm really happy to have Talvieno (officially) on board.

---

No, I'm not terribly impressed with this week (maybe I'm just tired right now...looking at the log, it seems fairly substantial?), next week should see less clerical work as well as less of having to finish up things, and more of getting to start new things. Seriously, next week I hope to see ships and stations in their full glory: PlateMeshes at 100% (rotations are definitely needed for stations), mesh warps implemented, procedural algorithms 100% ported, mesh ambient occlusion calculations 100% complete. I'd like to see asteroids as well, meaning scalar field extraction in the C core. If I have all of those things in a week, I'll be happy! I'll also have a good-looking, content-filled system, presumably. On the asm side, perhaps basic SSE functionality; let's say scalar operations at first, so that I can do the equivalent operations that the x87 FPU would do, but using the more-modern SSE (interestingly, the SSE instruction set is not just for vector processing, it has scalar (single-element) instructions as well).

Until some time in the next 7±2 days...! :wave:
“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford
User avatar
JoshParnell
Developer
 
Posts: 4261
Joined: Sun Oct 07, 2012 3:06 pm
Location: Baton Rouge, LA

Re: Tuesday, February 28, 2017

Postby Silverware » Tue Feb 28, 2017 10:31 pm

Well... /me is excite for one.

I had a huge post, about things that would be awesome to do... But I think I'll send that as a PM instead. :V
Image
Image
User avatar
Silverware
Vice Admiral
 
Posts: 2969
Joined: Sun Sep 07, 2014 3:23 pm
Location: Goattown-Three, Sigma Six, Goat Space

Re: Tuesday, February 28, 2017

Postby IronDuke » Tue Feb 28, 2017 10:41 pm

Didn't understand most of that, but
Image

I do love that you yak so much about LT. :thumbup: :D
:squirrel:
--IronDuke
Moving toward the future at 60 minutes per hour.
I-War 2 thread
Epic Limit Theory Limerick
User avatar
IronDuke
Rear Admiral
 
Posts: 1453
Joined: Mon Aug 31, 2015 10:47 pm
Location: The other end of the weapons barrage that's killing you. :3

Re: Tuesday, February 28, 2017

Postby Graf » Wed Mar 01, 2017 12:01 am

I love how Josh is like "I'm not happy with this week" when he kind of just implemented the networking necessary for multiplayer. :shock: Great work as always Josh, it really is amazing to read about your work! :thumbup:
Libertas per Technica
Graf
Commander
 
Posts: 248
Joined: Sun Mar 09, 2014 8:14 pm
Location: Virginia, USA

Re: Tuesday, February 28, 2017

Postby p2k » Wed Mar 01, 2017 12:38 am

I really like the possibility of networking within LT. Would it be feasible to use a dedicated server for the "background" economical simulation while the player uses his client to interact with the game?
User avatar
p2k
Ensign
 
Posts: 17
Joined: Tue Mar 26, 2013 4:49 pm

Re: Tuesday, February 28, 2017

Postby jarl » Wed Mar 01, 2017 2:57 am

LT deep simulation mode, when you play it spins up an Amazon cluster to the run the universe simulation and your client connects in to render the pew pew.
jarl
Lieutenant Commander
 
Posts: 110
Joined: Tue Oct 07, 2014 7:23 am

Re: Tuesday, February 28, 2017

Postby Damocles » Wed Mar 01, 2017 8:26 am

Using the browser as remote control station is also what I have implemented.

Its just much easier to implement a remote-GUI in HTML instead of a native desktop GUI or have to use the command line, and you have the bonus to be able to administer the running application
remotely on a mobile device.
User avatar
Damocles
Lieutenant Commander
 
Posts: 125
Joined: Thu Jan 12, 2017 4:09 pm

Re: Tuesday, February 28, 2017

Postby Two-Tone- » Wed Mar 01, 2017 10:35 am

Hey Josh, you mention that threaded contexts in OpenGL being "a bit messy". Have you considered Vulkan, since it's made from the ground up to be easily multithreaded? Plus I'm sure the performance boost would be nice as it's driver overhead is spread across multiple cores.

Unrelated, but will there be a Linux release of LT?
Two-Tone-
Recruit
 
Posts: 2
Joined: Wed Mar 01, 2017 10:28 am

Re: Tuesday, February 28, 2017

Postby Talvieno » Wed Mar 01, 2017 10:39 am

Two-Tone- wrote:Hey Josh, you mention that threaded contexts in OpenGL being "a bit messy". Have you considered Vulkan, since it's made from the ground up to be easily multithreaded? Plus I'm sure the performance boost would be nice as it's driver overhead is spread across multiple cores.

That's been (seemingly) suggested here, but Josh hasn't addressed it specifically.

Unrelated, but will there be a Linux release of LT?

There will be a Linux release, yes. In fact, Josh is developing it on a Linux system. :)

Welcome to the forum, by the way. :wave:
Image
Have a question? Send me a PM!
User avatar
Talvieno
Community Manager
 
Posts: 8200
Joined: Fri Apr 11, 2014 6:50 pm
Location: North GA, USA

Re: Tuesday, February 28, 2017

Postby Triscopic » Wed Mar 01, 2017 11:01 am

Forget multiplayer, I'm excited by the possibility of being able to leave LT running and remotely checking how my empire is running while I'm away, tweaking prices and ordering ships around via a web page while I'm at work. On lunch, obviously.

Years ago there was a mod that did some of this for (I think) X2 The Threat (definitely one of the games in the series). I found it pretty cool that I could do stuff remotely on my universe sim.
User avatar
Triscopic
Ensign
 
Posts: 29
Joined: Thu Dec 13, 2012 1:41 pm

Re: Tuesday, February 28, 2017

Postby Two-Tone- » Wed Mar 01, 2017 11:33 am

Talvieno wrote:That's been suggested here, but Josh hasn't addressed it specifically.


I hope he does! Vulkan has a lot of benefits he could use; lower frame latency, lower cpu usage, less time spent on the cpu during calls (IIRC, several times lower), VERY multithreadable, handles high number of objects far better, and more.

There will be a Linux release, yes. In fact, Josh is developing it on a Linux system. :)


frakking awesome.
Two-Tone-
Recruit
 
Posts: 2
Joined: Wed Mar 01, 2017 10:28 am

Re: Tuesday, February 28, 2017

Postby Miklos » Fri Mar 03, 2017 4:09 am

Triscopic wrote:Forget multiplayer, I'm excited by the possibility of being able to leave LT running and remotely checking how my empire is running while I'm away, tweaking prices and ordering ships around via a web page while I'm at work. On lunch, obviously.

Years ago there was a mod that did some of this for (I think) X2 The Threat (definitely one of the games in the series). I found it pretty cool that I could do stuff remotely on my universe sim.


This!

Who cares about multiplayer when I can check in via my phone or the game can push messages to me if they are important so I can make a choice while not being at the game :D
procedurally generated comment
User avatar
Miklos
Lieutenant Commander
 
Posts: 107
Joined: Sun Dec 09, 2012 10:58 am
Location: Copenhagen, Denmark

Re: Tuesday, February 28, 2017

Postby JanB1 » Sun Mar 05, 2017 8:07 am

JoshParnell wrote:Tuesday, February 28, 2017
As discussed last week, I emphasized building a release-quality implementation since they became an invaluable tool in LTC++. I now have a high-quality, fast, C library implementation (...)


It may be a dumb question, but did you ever consider the language "D", Josh?
Automation engineer, lateral thinker, soldier, addicted to music, books and gaming.
Nothing to see here

Flatfingers wrote:23.01.2017: "Show me the smoldering corpse of Perfectionist Josh"
User avatar
JanB1
Rear Admiral
 
Posts: 1273
Joined: Mon Mar 30, 2015 1:14 am
Location: Europe

Re: Tuesday, February 28, 2017

Postby Detritus » Mon Mar 06, 2017 8:23 am

JanB1 wrote:
JoshParnell wrote:Tuesday, February 28, 2017
As discussed last week, I emphasized building a release-quality implementation since they became an invaluable tool in LTC++. I now have a high-quality, fast, C library implementation (...)


It may be a dumb question, but did you ever consider the language "D", Josh?

How about we keep all languages out of his knowledge, to prevent further delays. :D :ghost:
Image
User avatar
Detritus
Rear Admiral
 
Posts: 1765
Joined: Mon Nov 23, 2015 10:05 pm
Location: Al Nair Gamma III B

Re: Tuesday, February 28, 2017

Postby JanB1 » Mon Mar 06, 2017 9:43 am

Detritus wrote:How about we keep all languages out of his knowledge, to prevent further delays. :D :ghost:


Buuut how does he communicate with us then? Huh? Didn't think about this one, hmm?
Automation engineer, lateral thinker, soldier, addicted to music, books and gaming.
Nothing to see here

Flatfingers wrote:23.01.2017: "Show me the smoldering corpse of Perfectionist Josh"
User avatar
JanB1
Rear Admiral
 
Posts: 1273
Joined: Mon Mar 30, 2015 1:14 am
Location: Europe

Next

Return to Dev Logs



Who is online

Users browsing this forum: No registered users and 4 guests