Multi-Server Cluster

Running Multi-Server Cluster requires a Network File System that is accessible from every host in the PipelineWise cluster. (Amazon EFS, Google FileStore or similar)

Network File System is required because PipelineWise keeps runtime configuration files in a common place on the host machine at ${HOME}/.pipelinewise directory. If you run PipelineWise commands on multiple nodes that operate on the same project, then every node has to read/write into the same directory, doesn’t matter where the nodes are located. This is typically done by mounting ${HOME}/.pipelinewise on every node to a shared directory on NFS/EFS.