Automate only lets you connect a chest to directly adjacent machines. Transport pipes would let you connect chests and machines from a distance.

Any updates? I NEED TO KNOW MORE ABOUT THIS. It's been 3y since I've started dreaming about this.

Unfortunately not. Most of my time goes towards maintaining SMAPI, Content Patcher, and a few dozen abandoned mods, so there's not much time left to work on major new features. Pull requests are welcome if anyone wants to contribute in the meantime though (feel free to ping me with @Pathoschild#0001 on the Stardew Valley Discord if you're interested!).

FarmAutomation.ItemCollector has a similar feature which links chests using paths. While that works to an extent, I think it can be unintuitive and limited in what you can do.

One possible approach is to link machines visually using an overlay view, similar to Gunpoint's crosslink mode. Each pipe could be two-way or one-way, or filter items by quality or type. Buildings would have special input/output pipes in the overlay to let you send items between buildings (e.g. send milk from your barn to the cheese presses in a shed).

52d3b3e adds a preliminary framework to support transport pipes in a future version. The rest of this comment explains how it works; feedback welcome. :)

Current system

We already have one component of transport pipes: IMachine abstracts machine logic into a state enum, input, and output. That makes the automation logic really easy; here's the actual core code:

foreach (MachineMetadata metadata in machines)
{
    IMachine machine = metadata.Machine;
    switch (machine.GetState())
    {
        case MachineState.Empty:
            machine.Pull(metadata.Connected);
            break;

        case MachineState.Done:
            metadata.Connected.TryPush(machine.GetOutput());
            break;
    }
}

However, the inputs and outputs are explicitly linked to item stacks and chests. That causes some important limitations for a transport pipe system:

• Consuming input from multiple sources is complicated.
  For example, let's say a machine needs 10 wood, and it has one connected chest with 8 and another with 5. The machine needs to collect the first stack and split the second, without actually removing anything from the chests until the item is consumed (e.g. the recipe might need a second ingredient which isn't present). Currently this is handled with a Requirement wrapper which manually syncs changes back to the underlying chests, which is messy and blocks higher-level features and only handles chests.
• Making machine output safe is complicated (and not currently done).
  If the connected chests don't have enough room to store the entire output, machines discard the remaining output due to the way output works (chests.TryPush returns true which triggers machine.Reset). If we fix that by not resetting if the push isn't complete, some machines will duplicate items instead (since machines don't necessarily return a 'held' stack, they might generate a stack on demand like bee houses).

Pipe abstraction

52d3b3e addresses the above problems by adding two components to complement IMachine:

• ITrackedStack manages a collection of items with item-stack-like semantics, while guaranteeing the safety of the underlying stack or stacks (which aren't accessible). You can call Reduce(int count) to deduct the specified number of items from the underlying stack(s), or Take(int count) to do the same while returning a new stack of the given size. The underlying stacks can come from anywhere — another machine, a chest, multiple chests, a custom source, etc.
• IPipe manages input & output between two endpoints. Pipes let machines request items and selectively consume those they need, or return output and safely deduct accepted items.

That mean machines now handle input/output with zero knowledge of what's behind a pipe. A given ingredient might come from a chest, multiple chests, or another machine (but the machine doesn't care). Machines now output tracked stacks, which means they can easily reduce/remove their internal stacks depending on how much output was accepted, without complicating their output logic.

Example pipe usage

Low-level APIs

Let's say a custom machine discards item qualities below a threshold (a commonly requested mod). Here's how it would get a matching item from a pipe and strip the quality:

ITrackedStack stack = pipe
   .OfType<SObject>() // ignore item types with no quality
   .FirstOrDefault(item => item.quality < SObject.highQuality);

if (stack != null)
{
   this.Machine.heldObject = stack.Take(stack.Count); // take all items from the underlying stack(s)
   this.Machine.heldObject.quality = 0;
}

And here's how it would safely return its output, without losing anything if the chests can only store part of the stack:

return new TrackedItem(
   this.Machine.heldObject,
   onReduced: i => this.Machine.heldObject.Stack = i.Stack,
   onEmpty: this.Reset
);

(Those examples just demonstrate the low-level APIs. The machine would probably want to handle a max number of items instead of the first stack it finds, which is easier with the higher-level APIs that are built on top of the low-level ones.)

Higher-level APIs

Most machines don't need to use the low-level APIs; they can simplify their code with pipe extensions instead. For example, here's how to handle a multi-ingredient recipe:

if (pipes.TryGetIngredient(SObject.coal, 1, out Requirement coal))
{
    if (pipes.TryConsume(SObject.iron, 5))
    {
        coal.Reduce();
        // start machine here
    }
}

Conclusion

In theory, we now have the underlying infrastructure to support transport pipes. Machines no longer know or care what their I/O is connected to, and the I/O pipes are precached ahead of time for the location. That means we can link arbitrary machines or chests together (e.g. based on the proposed overlay), and not change anything in the machine code.

This will be released as-is in Automate 1.2. If nothing comes up, we can continue building on it to eventually introduce a transport pipe feature for advanced players.

This sounds like an awesome idea! I'd love to help out with this one.

Here's a first mockup of how you'd use transport pipes.

1. First, you'd set up some machines.
   
2. Then you'd open the Automate overlay to set up advanced automation. Note how everything is faded except for the machines which can be automated.
   
3. You'd select the machines you want to group together. All machines within a group are automatically linked to the same inputs & outputs, and you can give each group a custom label.
   
4. And then you'd just pipe your inputs and outputs together. You could click those scrolls to set pipe options (e.g. only let certain qualities through) or view stats.
   

The overlay is meant for more advanced scenarios. The existing behaviour (chests auto-link to adjacent machines) would continue to work fine, and would show up as a group in the overlay automatically.

This is a very early mockup; feedback is welcome if you think it should work a different way. :)

Thank you for this fantastic mod! The in-out method you have created looks great. It would be interesting if you could somehow allow other mods to hook into that I/O system, so one could potentially create a milking mod which connects, or a tree-picking mod.

As for how to lay out a representation of this in the game, the select-and-connect method you are showing feels a bit more cheat-y than the rest of the mod. What if pipes were represented by physical items you had to lay down to connect chests to machines, and you could connect multiple machines together with pipes as well? You could potentially reuse an existing in-game item.

Yep, I plan to let mods create custom machines in a future version once the full framework is in place.

I'm not opposed to requiring physical connections instead of the overlay approach. It's a bit harder to implement and more limiting (e.g. you can't cross pipes), but it's not prohibitively harder. The main question would be what most players would prefer.

the gunpoint-like approach is quite interesting, I'd love to experiment with it but maybe making it more lore friendly in the future --having to unlock the linking device from the spirits or get it from the sorcerer, you probably know better than me-- could be cool, in the game you're a new farmer, not an former engineer and suddenly using pipes might not be quite lore friendly and harder for some people to deal with --as you said, pipes are less flexible than links-- in my modest opinion

I honestly prefer the simple in/out method, but I have to admit that part of my reason for doing so is that it's simpler to implement and I really really really want this done :). I don't think it feels too cheaty. I just think of it as putting some basic piping underground anyway. It would be a lot easier to work with than having to put down physical pipes to connect them.

Hello! First of all, thanks for this cool Modification - that clearly saves a lot of time for me...

I would like to jump into this discussion because I also had a few ideas in howto improve this cool mod. Unfortunately I am no C# programmer...

The idea:

I also prefer a simple in/out approach. The overlay idea is quite nice. But what about having parts of the machine on the Farm and other parts of it in the Basement of the house ? Or a crab cage in the river and the chest somewhere else ?

While I really like the automate mod, I otoh disliked that I ended up filling the Farm with tons of chests all over the place. I usually end up opening and closing them before I end the game, to ensure whether there are some remaining items left.

A simple approach would be this...

Use of "Antenas"

The principle of Antenas is similar as with chests... This means that the "Chest" behaviour must be kept as is... e.g. you can still arrange 8 items around a chest and the chest continues processing.

With Antenas you add a new feature.. You need to allow the game to craft an "Antena" (which looks sililar to the lightning rod). You can then place 8 items around the Antena (which does nothing so far)... Then you need to craft another Antena and there you need to store a chest beside it...

ooo
oIo -> Ic
ooo

o = Any item e.g. lightning rod, cheese maker, oil maker, ...
I = The new craftable antena item
c = Chest

Now you can right or left click on the Antena and assign a process number to it... e.g. process 1 .... and then the same with the other Antenna (process 1). Then the antenna will do the "catching" and forwarding and "catching" and returning from that said chest (which can be anywhere in the world).

That way you can extend a new XML object called "Antenna", having its own leaves inside the XML structure where you can assign the process numbers as well... So in case that if the player decides to stop using that mod, then the save game remains operating... The places where used to be the "Antennas" are then empty in the game and the chests remain where they have been placed (non operational).

So what I propose is two things... 1) keeping the "chest" behaviour as it is now... 2) extend it with the craftable "Antenna" item (the player needs to craft two of them) and have them assign a process number. If more Antennas are grouped together then all the machines interact each other. But one Antenna needs to remain at a chest (to make it operational) for input and output of goods like cheese, wine, oil etc. (haven't thought about that yet)...

Well this is just an concept idea... not thought through... But the simpler the better...

The usage of many "Antennas" as an example:

rI          rI         rI         rI         rI
                                                
                       Ic                      

r = rod
c = chest
l = antenna

The scenario shows 5 lightning rods somewhere on your farm. Each lightning rod has an "Antenna" besides it (or around it) and then there is one chest with an "Antenna" around it... All Antennas will be assigned a process number (say: 1)... Now they all produce batteries and sent the batteries to the chest which Antenna has the same process number...

You can change the process number of anything anytime... e.g. add more lightning rods or even a cheese machine to the same process number... Then the cheese machine will store its cheese in the same chest... and if there is milk inside... grab it and produce new cheese and then store the result in the chest... So batteries from the lightning rod and cheese will be stored in the chest (where you can also manually add milk, so the cheese machine will catch ist).

Another example:

mI                    ol
           Ic           
fl                    bl

m = mayonaise machine
o = oil machine
f = furnace
b = bait machine

c = chest
I = antenna

In this scenario we have 4 antennas next to the machines and 1 antenna to the chest... They all have process number (e.g.) 1 and now all interact with the chest... baits will be produced and sent to the chest (somewhere on your farm), oil is being made (if a truffle is placed in the oil maker or inside the chest). Same for furnace and mayonaise...

Another example (grouping)

mmm
mlm
mmm

Ic

fff
bIb
rhk

Here we have (grouped) 8 mayonaise machines around an Antenna... One chest with Antenna around it... 3 (F)Furnaces, 2 (B)Baitmakers, 1 Lightning (R)Rod, 1 (H)Honeymaker, 1 (K)Crystalarium.

The possibilities are enormous... That way you can sent batteries from long distance (via Antenna (call it a beam)) to your chest somewhere in a Barn (in the battery chest for example)...

Antennas can be placed and interact the same way like Automate does with chests... (e.g. the junimo hut), trash, etc... Regardless of what... There has always be one chest in the chain... A physical place to grab items out of it or inside it... No chest means... all Antennas with the same process number (and no chest) are not "transport piping" anything...

Here's my proposed idea for how the pipes should work. Theoretically it would work for a physical pipe or GUI system.

Some terms:

1. Let a chest or machine be known as an inventory.
2. Let a set of connected pipes/inventories be known as a network.
3. Let the manager of a network be known as a network controller.

Overall Idea

Basically, machines no longer are in charge of pushing/pulling items and pipes no longer have storage. Item movement is done instead by a network's controller, which directly moves items from one inventory to another. This solves problems related to gathering inputs from multiple locations and removing outputs.

1. The mod keeps track of all networks by storing a list of inventories connected by pipes for each network.
   • Each inventory in the network knows how far it is away from other inventories, based on pipe-manhattan distance.
   • This is updated whenever an object/building/terrain feature/tile is changed in the map.
   • Networks have inventories as their endpoints. For example,
     C---F---C
     Where C is chest and F is furnace. This consists of two networks, C---F and F---C.
     2. Every update tick consists of the controller first 1. filling ready-to-produce machines, and then 2. emptying finished machines.
     3. When a machine is empty, the controller searches all other inventories in the network for its inputs.
     • This search is done in an atomic manner - either all items for a recipe are found and placed in the machine, or none are.
     • This search prioritizes closer inventories based on pipe distance from the machine.
     • This search prioritizes other machine's outputs to look for inputs. That is, items will be first chosen from other machines than chests.
       4. When a machine is done, the controller searches for empty chests in the network for its outputs.
     • At this point, if the output is still exists, no other machine can use it, so the controller will look for a distance based chest to put it in.
       (Step 1 was filling empty machines. So if this machine still has items in it, no other machine could use it in a recipe, so it goes to a chest).
     • This search prioritizes closer inventories based on pipe distance from the machine.
     • This is done non-atomically - some of the output can be removed while some can remain. Until all output is removed, the machine stays in its done state.
       5. Pipes not connected to inventories only serve to define networks and direction
     • Direction pipe makes it so items can only flow in one direction through the pipe. This is enforced at network creation time when finding distances between machines.
       6. Pipes connected to inventories hold no storage and only serve to add the inventory to the network and define options
     • Input only pipes mark their connected inventory as only able to receive items.
     • Output only pipes mark their connected inventory as only able to send items.

Mod Compatibility

The machine interface would be changed to have methods for GetState() to retrieve the machine state, GetOutput() to retrieve the produced item, Push(Items) to put items into the machines, and GetRecipes() to get the inputs the machine takes. Push would only be called if the state of the machine is empty, and the items provided would be guaranteed to fulfill the inputs of some recipe. Custom machines would not care about pipes.

Considerations:

1. Item movement doesn't really transport through the pipes, it goes direct to inventories. I feel this is fine, unless we want animations of items actually travelling through pipes to reach their destination.
2. Pipes are by default bi-directional.
3. Theoretically compatible with a GUI system. The network would now contain groups, which contain lists of machines. Each group would have the input/output only property instead of at a machine level, but everything else should stay the same.
4. The network and distances would need to be recalculated every time the world changes.
5. Works with wireless pipes.

I definitely want to work it into the game lore if possible. The Junimos can already collect crops for you, maybe they're the power behind pipes? I'm not sure how I'll make pipes lore-friendly yet, but I'm open to any other suggestions.

If we go down the Junimos route, then the transport pipes could be an underground system of roots 😄.

On the other hand, everything else they seem to do is magic/spirit related... perhaps the "pipes" can be in the spirit realm.

[...]maybe making it more lore friendly in the future --having to unlock the linking device from the spirits or get it from the sorcerer, you probably know better than me-- could be cool, in the game you're a new farmer, not an former engineer[...]
—@HellDragger

I definitely want to work it into the game lore if possible. The Junimos can already collect crops for you, maybe they're the power behind pipes? I'm not sure how I'll make pipes lore-friendly yet, but I'm open to any other suggestions.

@Patronos

But what about having parts of the machine on the Farm and other parts of it in the Basement of the house ? Or a crab cage in the river and the chest somewhere else ?

My idea for that is letting you drag pipes to the edge of the map, which will poke out the other side. For example, drag a pipe in a barn to the edge of the building and it'll appear outside so you can connect it to things outside.

So what I propose is two things... 1) keeping the "chest" behaviour as it is now... 2) extend it with the craftable "Antenna" item (the player needs to craft two of them) and have them assign a process number. If more Antennas are grouped together then all the machines interact each other. But one Antenna needs to remain at a chest (to make it operational) for input and output of goods like cheese, wine, oil etc. (haven't thought about that yet)...

Thanks! Antennas are a really intriguing idea, but I'm not sure I'll go with that approach. Ideally I want the mod to be more like a gameplay mechanic than a cheat, and I feel like pipes are a better direction to meet that goal. Pipes let me tweak the mechanics to optionally introduce gameplay limitations (e.g. can't cross pipes?) and crafting constraints (e.g. need resources to craft each pipe segment?) to make large-scale automation more balanced.

I might consider antennas as a future enhancement in addition to pipes (e.g. if you want to transmit items long-distance), but I'd want to carefully consider the balance impact (e.g. maybe make antennas bulky or expensive?).

@danvolchek

Basically, machines no longer are in charge of pushing/pulling items and pipes no longer have storage. Item movement is done instead by a network's controller, which directly moves items from one inventory to another. This solves problems related to gathering inputs from multiple locations and removing outputs.

There isn't a distinct controller, but mostly the mod already works that way. Machines don't deal directly with item IO; instead they operate through an abstraction layer which includes IPipe (which handles IO) and ITrackedStack (which represents a stack of one item), with methods like pipes.TryPush(…) and pipes.TryGetIngredient(…) and stack.Consume() to defer the networking complexity to the Automate mod.

For example, TryGetIngredient might return four stacks from three chests/machines, and it makes no difference to the machine itself since the higher-level ITrackedStack interface doesn't change.

One limitation of your network controller approach is that machines sometimes need to peek at the available items. For example, the furnace needs both a coal and a stack of accepted input; only if both are present does it consume both as one atomic operation. With the networking controller approach, the furnace would need to consume the coal immediately even if no input was available, then push it back which would impact performance (unless the controller handled multi-item recipes as a top-level feature, which would increase complexity instead).

The current implementation tracks networks as flat connections, unlike your proposed approach. I think there's definitely value in sorting pipes by distance to improve predictability for players, and I might do that in the future, but I think it's better to have a flat list of connected pipes after the precalculation phase for simplicity.

When a machine is done, the controller searches for empty chests in the network for its outputs. [...] This is done non-atomically - some of the output can be removed while some can remain. Until all output is removed, the machine stays in its done state.

Yep, the mod already automates output non-atomically.

Pipes connected to inventories hold no storage and only serve to add the inventory to the network and define options

That's essentially how they work now. A pipe doesn't hold any storage, it's just an IO abstraction for things that do. For example, MachinePipe would let any machine provide/accept items for other machines by wrapping the IMachine interface. It's not needed yet (since all machines need to touch a chest), but it'll be useful when transport pipes are available to the player.

The machine interface would be changed to have methods for GetState() to retrieve the machine state, GetOutput() to retrieve the produced item, Push(Items) to put items into the machines, and GetRecipes() to get the inputs the machine takes. Push would only be called if the state of the machine is empty, and the items provided would be guaranteed to fulfill the inputs of some recipe. Custom machines would not care about pipes.

That's basically how machines work now, except for GetRecipes. Machines do receive the connected pipes in the current implementation, but they use abstractions for the pipe IO (they're not pushing items into each individual pipe, for example).

Item movement doesn't really transport through the pipes, it goes direct to inventories. I feel this is fine, unless we want animations of items actually travelling through pipes to reach their destination.

That's fine. That's how transport pipes currently work, and I have no current plans to add animations of visible items. We could add it later though, since each ITrackedStack tracks each item's source.

The upcoming Automate 1.8 adds two big features to support transport pipes:

1. In collaboration with @Camper6, the new overlay lets you visualise which machines are automated (green) or unconnected (red):

   

2. Chests now automate all connected machines, not only those immediately surrounding it:

   

Local storage indexes (#307) are the next step towards transport pipes, for those interested.

I just noticed that this issue is still open. I would argue that perhaps it's time to close it since transport pipes do in fact work. Although one of the expected features that it originally declared it would provide does not appear to be working: Connect a pipeline into a shed.

I'm close to testing if connecting a pipeline into a barn works. But I can report that connecting a pipeline into a large shed isn't currently working (well at least for me).

Would it be good to open a new issue about that or would you prefer to report it here and keep going on this issue?

The transport pipes described here are different from the connectors feature, although there's some overlap. The 'pipes' are the internal component to enable filtering and remote connections (including connections from building interiors to another location).

The ticket is pretty old so it might seem there's no movement, but it's gradually moving forward. For example, SMAPI 3.1 adds chest events which unblocks the local storage indexes mentioned in my previous comment.

Still I'll create an additional ticket for the issue I found. I couldn't find anything in the backlog that maps to the problem I'm having so if an issue does exist that provides a solution it would be good to link it. That would help people searching for a solution in the future.