while reading the Ship Upkeep Costs thread (and others about ammunition/fuel/etc) the notion of simplifying everything to make it manageable was repeadely brought up.
but i thought:
i want the detail, i want 200 different kinds of missles and bullets to distribute in my fleet and have it manageable
i want dozends of ores i refine into even more metals and process them to platings and such

but usually that means i have to take the burden of micromanagement,
babysitting every single factory and carrier for its stocks

that must change

so i built Pipelines.

General Description

Pipelines are functional constructs intended to visualise streams of commodities and even produced ships between stations and ships and to make the streams easy manipulable and scalable.
They visualise the ware streams with colored lines that span between ships and stations, each ware has its own pipeline and this small lines can be grouped to bigger lines to encompass multiple wares.
The wares are transported by freighters that are allocated to the pipelines and distributed based on cofigruable values.

New Stations and Ships can simply be added to the pipeline by dragging its respective node on the pipeline.

Settings can also be copied by dragging an docked node over another, the dragged node then copies the settings of the non-dragged node.

non-player assets can also be added to build automated trading routes served by the player
in this image are 4 stations, a mine, a refinery, a factory and a storage station.
all are connected to the grouped "Randomium Pipeline", an grouped pipeline which contains the three smaller Randomium Ore, Randomium (Metal) and Randomium Plates Pipelines.

All stations have input and output connections to the pipeline, indicating that they exchange wares via the pipeline.
if one of them gets outside its allocated "comfort zone" of stored resources one of the freighters that's assigned to the pipeline is tasked to correct that state.
either by delivering or carrying away goods.

Edit: Grouped pipelines collapse into a single larger line representing all contained lines, the contained lines get shown when zooming in on the line

Pipelines

[Single as well as grouped Pipelines will be adressed as "Pipelines", as the mechanics do not change]

Pipelines are abstract objects that symbolise "standing orders" for assigned freighters to transfer wares between "docked" stations and ships.
they transfer wares from stations that are overfilled to stations that do not have enough resources.
they do so by ordering their assigned freighters to transport wares from one station to another.

also they can contract external transporters if allowed to do so via a simple checkbox.

they have an "cloud storage" composed of the combined allocated storage of all connected stations and ships, this storage will be used to store resources until they are demanded.

freighters can be assigned to the pipeline as single ships or as groups of ships as described in Gazz' Joint Nodes,
to build convoys which can be everything from a single freighter to an armored battlegroup protecting an resupply convoy.

also there should be "blueprints" of Joint nodes that can also be attached to the freighter node.
when such an blueprint gets assigned freighters will be ordered at an shipyard to be assigned to the pipeline.
such groups will be treated identical to single ships.

Nodes
stations can be coupled to a pipeline and attach automatically relevant inputs and outputs to the pipeline.

Edit: Joint Nodes and JN Blueprints store their Pipeline structure, so that groups of ships and stations can be built with them and do not have to be micromanaged again

for example the refinery shown above imports randomium ore and exports randomium
this links automatically connect to the pipeline and start im/exporting according to their settings.

Edit: storage space of nodes can be allocated to different wares and pipelines,
for example one can allocate 60% of the factories storage for imported wares and 40% for exported wares and only so much would at max get filled with imported or exporting wares

Edit: nodes with the same inputs, outputs and settings collapse into a single node representing any number of nodes.
(for example: if one adds another Randomium Ore mine to the Pipeline, the two get merged into one node saying 2xRandomium Mine)

nodes have essentially 2 different kinds of option bars, which also serve as displays

• First there are the export/import confugurations (the green/red arrows in the illustration)
  • on the top of the bar is the total storage allocated to the ware thats im-/exported
  • The orange box is the "comfort zone" of the respective ware
    it can be changed by dragging the upper or lower bounds
    the node tries to stay in the comfort zone
    if it gets out of the comfort zone it either exports or imports wares.
    
    export goods get exported when the current amount is over the upper bound.
    the station tries to keep some minimum amount of export ware "in house", controlled by the lower bound.
    Edit: this "in house" storage gets tapped if the ware is nowhere else on the pipeline available
    
    import goods get requested if the current store is lower than the comfort zone.
    if it imports the station fills to its upper bound.
    it only fills above the upper bound of the "cloud storage" is otherwise full
  • The same-colored bar represents the current stored amount of wares
• the bar on the far right shows the allocation of the whole storage space of the station or ship.
  the allocation values can be changed by dragging the upper or lower bounds or by typing the wanted amounts in the procentual or absolute textlabels
  the topmost (blue) allocation box denotes storage space occupied by another pipeline not shown in the graphics, this storage can also be partitioned

edit:
wares not used by any production process can be added to any node as imports, exports or storage endpoints.

that makes it possible to set fleets (or more precise their fleet tenders) as imports for ammunitions.
"on the other side" of the node that represents the fleet tender can storage endpoints for the ammunitions be added, that any fleet member can order resupply from the tender.

Storage

Storage Nodes only provide storage space for the Pipeline, they only start to fill up if no node needs resources to fill up its comfort zone.
they also provide means of separating pipelines from each other, for example to separate the production and delivery of munitions.
they can also be configured to sell surplus resources or buy undersupplied wares by configuring its comfort zone.
Edit: maybe add some checkboxes for allowing/disallowing external trade


[currently dont know how to word an TL;DR, maybe added later]


edit 12.8.2014:

i've lately thought in the X:Rebirth esque cargo drones thread that there is a need for handing of the micromanagement in transferring large batches of cargo between entities which are not repeated.
for example a big purchase from an NPC station, or a transfer from a player station to the players capital ship.

from the current knowledge there would be 3 possibillities:

• docking the 2 entities and transfer the cargo directly
• make a contract to have your wares transferred
• micromanage your smaller freighters to carry the cargo to your ship

the first method is possibly not always available due to size restrictions, you might not dock the LT equivalent to an terrestiral container freighter to a small listening post, and the second method seems quite burdensome and unreliable to me if all you want is a quick cargo transfer.

so i propose now an addition to the pipeline system to make the third option much easier to handle


Single run Orders

a single run order is in theory just a pipeline like any else but it does not specify desired, permanent storage states but allocations on how a limited contingent of wares should get transported.


it starts with something like i proposed in quick orders (and thymine expanded upon).

you take a ware (or a bunch of wares), and assemle it into a package.

this package only contains number of type of wares you want to transfer, but no target.

the target of the transfer gets determined by a dragging action, drawing a line from start to end.

this line is now the single run order and behaves like an usual pipeline, you can assign freighters to it, add inputs or outputs to it and distribute the wares on percentages or absolute values.

but it has some special conditions:

• nodes which are "done" get removed from the pipeline.
  once the ship or station has received its share of the wares it disconnects from the pipeline and does not interact with it anymore.
  this effectively removes the pipeline automatically after its work is done
• freighters can get automatically assigned.
  freighters which have the start/endpoint as their homebase and have no orders on them (like serving another pipeline) allocate themself to the pipeline as needed to transfer the cargo as fast as possible.
  there could also be a special roster for "on demand" freighters as a special state.
  whose sole purpose is to wait for such single-run orders and execute them if no other freighters are there to serve them. (this should happen on somewhat intelligent conditions, distance, other freighter wihtout order that is closer etc).
• it does not get executed continually but only once
• cargo drones (if implemented) get used automatically inside their range

i think pipelines can now cater for any kind of ware transfer



edit 3.9.2014:

1. Money
   
   pipelines didnt care about money, until now.
   
   for every player (executive AI or gamer) there is a "main bank account" this account isnt bound to any object but to the player
   
   you can draw pipelines like usual for money, it also gets handled like any other ware regarding to comfort zones.
   altough it doesnt have to be transported physically
   (because a non-executive NPC cannot have money, as they arent part of the economy as per josh
   the money always stays at your immediate disposal, but the allocated funds should be marked somehow
   for example 10000 credits(20000 credits allocated))
   
   setting the comfort zone creates a budget for the ship/station/whatever.
   this budget gets drawn upon when the object buys something from any source, also from your own assets.
   when the object comes out of the comfort zone it transferrs the money like any other ware, to a regional headquarters for example.
   
   for example a few mines could be positioned under the regional headquarters and you can see the money flow from your regional headquarter if the mines produce income or not
   
   but why does it reshuffle the money between your assets?
   
   accounting
   
   you probably want to know if it would be cheaper to just buy the stuff you need instead of producing it yourself, or vice versa
   
   so you follow the flow of the money, and see where it comes from and where it goes to
   
   and for this is the second new idea:
2. Bilances
   
   for every ware thats moved by a pipeline theres a record of it for the relevant node.
   
   so when your mine exports 20 iron ore, your bilance for this station says +20 iron ore.
   
   this could and should work and look like the trading price screen from one of the earlier updates
   
   with positive/negative bilances for certain timeframes and such.
   
   as an example you could take a simple corporate structure
   
   a factory which produces supplies for your military.
   
   so you set a budget for your fleet and the fleet gets its supplies from your factories.
   
   so your factory's bilance turns positive, because it "sells" its supplies to your military, and your militaries bilance turns negative because it has no income and buys from your factory.
   
   this also works with external transactions, for example when you buy or sell wares to NPC's, the bilance for the sold ware is negative, because wares get exported, and for bought wares its positive, because wares get imported
   
   for money there should also always be a pipeline between the player and the main account, to make it easier to see if you spend too much money
   

This is awesome and I support this idea.

ThymineC wrote:This is awesome and I support this idea.

I definitely agree.

This also made me think: since everything in LT is procedurally generated, could it be possible/interesting to make numbers of different ores/alloys as scalable at the start of the game?
This way who prefers a more straightforward production line is satisfied as well, and if you couple this parameter with abundance/scarcity of ores and some kind of "distribution" attribute (something that states like "ores tend to be mixed in every asteroid cluster" or "every cluster has a bigger amount of a set ore and almost nothing of everything else") could lead to A WHOLE LOT of difference between playthroughs...

I had precisely the same thought in another thread, though I compared it more to a scripting flowgraph like Crysis uses rather than buildcraft pipes.

Regardless of the number of commodities we must track, I think this would be a great addition for being able to control your empire in an intuitive fashion.


Though I still think hundreds of commodities would be too much for the average player to comfortably handle, even with such aids.

Though perhaps it wouldn't be so bad if you can depend on NPCs to supply most of your needs.. It didn't work in X3 since NPC production was so minimal in many goods, so you needed a lot of your own factories in order to supply your fleets. Had the NPCs actually had to produce what they used, player use would have been a much smaller share of the economy, and so you could have gotten away with buying more. Like, equipping a battleship took a weeks worth of the entire worlds production of weapons. If such were the case in LT, the player could simply task supply ships to purchase from the NPCs rather than have to be completely self reliant.



There is also the issue of actually getting the materials where they are needed, i.e. to the ships themselves. There is going to have to be some simple method of setting how much stock of an item you want in a ship, and then a simple way to supply it, order it resupplied, or order it to resupply itself.

Edited the OP for clarification

CutterJohn wrote: There is also the issue of actually getting the materials where they are needed, i.e. to the ships themselves. There is going to have to be some simple method of setting how much stock of an item you want in a ship, and then a simple way to supply it, order it resupplied, or order it to resupply itself.

Well, you can simply add an input line to the ship you want to have supplied and configure its comfort zone.

Lovely idea.
I support the implementation of storage. This was something I really missed in the X games.
I like warehousing as a way to influence local economies.

These ideas also tie in well with my thoughts on business management, which I'll hopefully get around to posting about sometime.

made some more edits to the OP, because there were some details missing

Very nice idea. Hoping we have something like this in LT

Cornflakes_91 wrote:Edited the OP for clarification

CutterJohn wrote: There is also the issue of actually getting the materials where they are needed, i.e. to the ships themselves. There is going to have to be some simple method of setting how much stock of an item you want in a ship, and then a simple way to supply it, order it resupplied, or order it to resupply itself.

Well, you can simply add an input line to the ship you want to have supplied and configure its comfort zone.

Needs to be something more robust than that. You might have dozens of ships. Having to attend to them each individually would get tedious, especially considering combat losses means its not a 'set it and forget it' type of action.

If the pipeline for any single trade ship is defined by the form of "From Space Station A headed to Star Ship B" then I think it would be easier to manage. Obviously both underlined names can be changed to create any number of different pipelines between two vessels.

The process could be as simple as selecting the trade ship, clicking a pipeline button, then with two drop down lists the player is given which locations the trade ship will trade between. The items to be traded from one ship to another would also have to be defined on this screen.

CutterJohn wrote:Needs to be something more robust than that. You might have dozens of ships. Having to attend to them each individually would get tedious, especially considering combat losses means its not a 'set it and forget it' type of action.

I think this issue is more indicative of the entire game in general, not the system specifically. We're constantly seeing how we need a way to abstract pretty much ANYTHING from a single ship to multiple so that we don't land in micromanagement hell.

A pipeline should be a standing order. That's abstract enough.
Suppose you have standing orders to:
- Gather resource X to warehouse A
- Keep factory B stocked from warehouse A
- Deliver the end products of factory B to station C

I think you could do pretty much everything with this that you wanted. You could attack a bunch of freighters to all of these pipelines and let the AI figure out what to use each freighter for. Or, you could micromanage and manually assign the best freighter to the job.
Perhaps you need a bigger freighter for stocking the factory, and a smaller faster one to move all end product.

Josh has made it clear that the AI is very adaptable, it scales to the information and freedom you give it. You could give all your freighters access to all your pipelines and let the AI handle it. But this could end up being inefficient. Ultimately, designing specific freighters which are adjusted through technology to fit a certain role will be the micromanage-y, but efficient way to go.

CutterJohn wrote: Needs to be something more robust than that. You might have dozens of ships. Having to attend to them each individually would get tedious, especially considering combat losses means its not a 'set it and forget it' type of action.

Cornflakes_91 wrote: Joint Nodes and JN Blueprints store their Pipeline structure, so that groups of ships and stations can be built with them and do not have to be micromanaged again

from Gazz Node Editing

Gazz wrote: [...]
An entire wing of fighters would be handled / commanded as one object and have health / supply values.
If the creation of such a wing is deliberate, fleet organisation can be as simple or as complex as the player desires.

Also, if a fleet has a TOE (building plan...), you could grab a bunch of fighters and bigger ships and assign them to "the fleet".
According to the TOE they would fill empty slots (losses), bringing the fleet back up to strength, and any excess ships would end up in an automatically generated "reinforcements" container at the fleet's top level.
That way the don't "mess up" your organisation but it may have to be an optional mechanic.

Cornflakes_91 wrote:Pipelines are functional constructs intended to visualise streams of commodities and even produced ships

BFett wrote:If the pipeline for any single trade ship is defined by the form of "From Space Station A headed to Star Ship B" then I think it would be easier to manage. Obviously both underlined names can be changed to create any number of different pipelines between two vessels.

The process could be as simple as selecting the trade ship, clicking a pipeline button, then with two drop down lists the player is given which locations the trade ship will trade between. The items to be traded from one ship to another would also have to be defined on this screen.

that would bring up the whole issue again with managing every freighter individually, with my concepts pipelines and the freighters who serve them are independent from each other.
so you set up your transport routes and then assign freighters to it