LuckPerms

LuckPerms

Mods
41.4k Downloads

Symbolic link for yaml-storage gives java.nio.file.FileAlreadyExistsException

TomLewis opened this issue · 16 comments

commented

Ahoy!

I have a few servers on the same machine with split storage

split-storage:
  enabled: true
  methods:
    user: MySQL
    group: yaml
    track: yaml
    uuid: MySQL
    log: MySQL

Its much, much easier to manage permissions using flatfiles while keeping user data and logs in MySQL, the only issue I have is with multiple servers having their own Groups in YAML it gets very messy when updating and changing plugins on one server, and having to copy them over to another, the simplest solution I could think of was a Symbolic link for the yaml-storage folder to the other server so then I could edit one set of permissions and set per-server or per-world in this one set of permissions!

Sadly when booting the second server, it threw a bunch of errors;

[16:43:17] [Server thread/INFO]: [LuckPerms] Loading configuration...
[16:43:18] [Server thread/INFO]: [LuckPerms] Loading storage provider... [SPLIT STORAGE]
[16:43:18] [Server thread/WARN]: java.nio.file.FileAlreadyExistsException: /home/user/folder/plugins/LuckPerms/yaml-storage
[16:43:18] [Server thread/WARN]: 	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94)
[16:43:18] [Server thread/WARN]: 	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
[16:43:18] [Server thread/WARN]: 	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
[16:43:18] [Server thread/WARN]: 	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)
[16:43:18] [Server thread/WARN]: 	at java.base/java.nio.file.Files.createDirectory(Files.java:689)
[16:43:18] [Server thread/WARN]: 	at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:796)
[16:43:18] [Server thread/WARN]: 	at java.base/java.nio.file.Files.createDirectories(Files.java:742)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.util.MoreFiles.createDirectoriesIfNotExists(MoreFiles.java:57)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.storage.implementation.file.AbstractConfigurateStorage.init(AbstractConfigurateStorage.java:159)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.storage.implementation.file.SeparatedConfigurateStorage.init(SeparatedConfigurateStorage.java:151)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.storage.implementation.split.SplitStorage.init(SplitStorage.java:86)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.storage.Storage.init(Storage.java:115)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.storage.StorageFactory.getInstance(StorageFactory.java:88)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.plugin.AbstractLuckPermsPlugin.enable(AbstractLuckPermsPlugin.java:146)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.bukkit.LPBukkitBootstrap.onEnable(LPBukkitBootstrap.java:161)
[16:43:18] [Server thread/WARN]: 	at org.bukkit.plugin.java.JavaPlugin.setEnabled(JavaPlugin.java:264)
[16:43:18] [Server thread/WARN]: 	at org.bukkit.plugin.java.JavaPluginLoader.enablePlugin(JavaPluginLoader.java:316)
[16:43:18] [Server thread/WARN]: 	at org.bukkit.plugin.SimplePluginManager.enablePlugin(SimplePluginManager.java:405)
[16:43:18] [Server thread/WARN]: 	at org.bukkit.craftbukkit.v1_12_R1.CraftServer.enablePlugin(CraftServer.java:395)
[16:43:18] [Server thread/WARN]: 	at org.bukkit.craftbukkit.v1_12_R1.CraftServer.enablePlugins(CraftServer.java:344)
[16:43:18] [Server thread/WARN]: 	at net.minecraft.server.v1_12_R1.DedicatedServer.init(DedicatedServer.java:223)
[16:43:18] [Server thread/WARN]: 	at net.minecraft.server.v1_12_R1.MinecraftServer.run(MinecraftServer.java:616)
[16:43:18] [Server thread/WARN]: 	at java.base/java.lang.Thread.run(Thread.java:834)
[16:43:18] [Server thread/INFO]: [me.lucko.luckperms.lib.hikari.HikariDataSource] luckperms-hikari - Starting...
[16:43:18] [Server thread/INFO]: [me.lucko.luckperms.lib.hikari.HikariDataSource] luckperms-hikari - Start completed.
[16:43:18] [Server thread/ERROR]: [LuckPerms] Failed to init storage implementation
[16:43:18] [Server thread/WARN]: java.lang.RuntimeException: One of the backings failed to init
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.storage.implementation.split.SplitStorage.init(SplitStorage.java:93)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.storage.Storage.init(Storage.java:115)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.storage.StorageFactory.getInstance(StorageFactory.java:88)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.plugin.AbstractLuckPermsPlugin.enable(AbstractLuckPermsPlugin.java:146)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.bukkit.LPBukkitBootstrap.onEnable(LPBukkitBootstrap.java:161)
[16:43:18] [Server thread/WARN]: 	at org.bukkit.plugin.java.JavaPlugin.setEnabled(JavaPlugin.java:264)
[16:43:18] [Server thread/WARN]: 	at org.bukkit.plugin.java.JavaPluginLoader.enablePlugin(JavaPluginLoader.java:316)
[16:43:18] [Server thread/WARN]: 	at org.bukkit.plugin.SimplePluginManager.enablePlugin(SimplePluginManager.java:405)
[16:43:18] [Server thread/WARN]: 	at org.bukkit.craftbukkit.v1_12_R1.CraftServer.enablePlugin(CraftServer.java:395)
[16:43:18] [Server thread/WARN]: 	at org.bukkit.craftbukkit.v1_12_R1.CraftServer.enablePlugins(CraftServer.java:344)
[16:43:18] [Server thread/WARN]: 	at net.minecraft.server.v1_12_R1.DedicatedServer.init(DedicatedServer.java:223)
[16:43:18] [Server thread/WARN]: 	at net.minecraft.server.v1_12_R1.MinecraftServer.run(MinecraftServer.java:616)
[16:43:18] [Server thread/WARN]: 	at java.base/java.lang.Thread.run(Thread.java:834)
[16:43:18] [Server thread/INFO]: [LuckPerms] Loading messaging service... [BUNGEE]
[16:43:18] [Server thread/INFO]: [LuckPerms] Loading internal permission managers...
[16:43:18] [Server thread/INFO]: [LuckPerms] Performing initial data load...
[16:43:18] [Server thread/WARN]: java.util.concurrent.CompletionException: java.lang.NullPointerException
[16:43:18] [Server thread/WARN]: 	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
[16:43:18] [Server thread/WARN]: 	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
[16:43:18] [Server thread/WARN]: 	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1739)
[16:43:18] [Server thread/WARN]: 	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1728)
[16:43:18] [Server thread/WARN]: 	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
[16:43:18] [Server thread/WARN]: 	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
[16:43:18] [Server thread/WARN]: 	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
[16:43:18] [Server thread/WARN]: 	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
[16:43:18] [Server thread/WARN]: 	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
[16:43:18] [Server thread/WARN]: Caused by: java.lang.NullPointerException
[16:43:18] [Server thread/WARN]: 	at java.base/java.nio.file.Files.provider(Files.java:100)
[16:43:18] [Server thread/WARN]: 	at java.base/java.nio.file.Files.newDirectoryStream(Files.java:471)
[16:43:18] [Server thread/WARN]: 	at java.base/java.nio.file.Files.list(Files.java:3698)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.storage.implementation.file.SeparatedConfigurateStorage.loadAllGroups(SeparatedConfigurateStorage.java:287)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.storage.implementation.split.SplitStorage.loadAllGroups(SplitStorage.java:173)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.storage.Storage.lambda$loadAllGroups$10(Storage.java:195)
[16:43:18] [Server thread/WARN]: 	at me.lucko.luckperms.common.storage.Storage.lambda$makeFuture$1(Storage.java:99)
[16:43:18] [Server thread/WARN]: 	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
[16:43:18] [Server thread/WARN]: 	... 6 more
[16:43:19] [Server thread/INFO]: [LuckPerms] Successfully enabled. (took 1177ms)

Searching for java.nio.file.FileAlreadyExistsException gives me this stack overflow which says that its when the same file is already open in another thread.
https://stackoverflow.com/a/43669557/381907

commented

I think it’s best for you to use the context feature of luckperms to have per world/server permission. https://github.com/lucko/LuckPerms/wiki/Context

commented

I think it’s best for you to use the context feature of luckperms to have per world/server permission. https://github.com/lucko/LuckPerms/wiki/Context

How would I use contexts with multiple servers? Just have 1 instance of LuckPerms on the bungee server and not on each server itself then setting contexts in the bungee?

At present I have it on the bungee with no perms, then installed LuckPerms on each server, each server has its own folder with its own yaml-storage folder for perms, and all user data is MySQL.

commented

You need to have luckperms of all server and sync them to a common sql database. Then, use context to set permissions/groups to a per server/world basis.

https://github.com/lucko/LuckPerms/wiki/Syncing-data-between-servers
https://github.com/lucko/LuckPerms/wiki/Context

commented

Yeah, I've had a shared database for users for years, I'm not looking to move permissions to mysql, they need to stay as flatfile, that's why I tried the symbolic link. I've found multiple other posts saying they did the same thing with success.

I already have got it on my bungee and all servers, all already linked to mysql for users (see my first post explaining with my data structures/
) I just want a shared tank permissions where I can set contexts for per server in the same config files, then I can share perms.

commented

Different workflows.
Why Flatfile beats MySQL for just permissions not users/logs, see first post I never doubted that have used MySQL for users/logs for years.

All 4 servers are on the same machine. I have a basic structure:

  • Minecraft Folder

    • Server_1/plugins/LuckPerms /yaml-storage/groups
    • Server_2/plugins/LuckPerms /yaml-storage/groups
    • Server_3/plugins/LuckPerms /yaml-storage/groups
    • Server_4/plugins/LuckPerms /yaml-storage/groups

  • I have 80 plugins, All of them have YAML configs, I have access to all of them from a local copy in VSCode using the VSCode workspace.

    • All 4 of my servers are accessible on the same structure easily from one window in VsCode, same single editor for all plugins, no need to separate this for 1 plugin.

  • I can edit and push a permission change without needing to Login to Minecraft/console not run any commands, not need to open a web browser, not do any of the extra 3-4 steps to use the web UI.

  • I have the full scope of VsCode or whatever edtior I need for the specific tasks at hand to quickly search and replace, organize, copy/paste, manage all my permissions locally, in a fast way using an editor, and offline.

My recommendation is an addition to the file-watcher service included with luckperms, you could easily have the file-watcher trigger a copy/run a script when it detects a change, for example, when it detects that default.yml has changed it could copy that file to 3 other locations, which on those servers will automatically update due to the file watcher.

A bit like rsync, alternatively, I could keep 4 separate copies and use rsync, if I could get a script to trigger on the event of file changes, if none of this happens, then I guess I will just have to setup rsync from a script and either put it on cron or write my own monitor for changes, all will be much simpler than having to, login to minecraft/console run a command to get a URL, somehow edit my hundreds of permissions in a web UI, I always am searching using regex, save the changes and then re-paste the URL to update it, all without the peolpe online bugging me to do XYZ where as I could do all this before wihtout needing to log into minecraft or hunt down a URl thats already passed me in my console by chat messages moving so fast.

Another alternative is to have the web UI allow server owners to have their own proper subdomain with auth, so there is no need to constantly re-login and re-create the temporary permissions each time, then its just a bookmark, this could also be done using DNS to prove that server owns the domain its editing the permissions on.

commented

they need to stay as flatfile

No they don’t. You’d be the first person ever to actually need to use YAML.
I mean you’re trying to do what using a database does with YAML. That’s a recipe for disaster.
Using symlinks is not a supported setup. Not because it’s impossible or even hard to do but because it means you have severe flaws in your storage concept. If you need to have YAML files synced you’re doing it wrong.

Now I’m giving you the benefit of the doubt. While I do think that the setup you’re attempting to use is fundamentally flawed I want to give you the opportunity to explain what you are trying to do and why you think that syncing YAML files between servers is the best way to go about things instead of just using a database.

  • 😄1
commented

While i do completely agree that using YAML files is a very very dumb idea, and knowing i will probably get flamed and/or worse, i can give you a hint (not a working solution) on how you can achieve this, assuming you are fully aware that you want to go that road and also me not taking any responsibility for any kind of damages.

Setup a version control of your choice (like gitlab) and put your files inside a repository. Then setup a CD job, that will update all the YAML files you need at any desired locations (and restart your servers if needed etc) , then hook your VScode to the version-control repo and voila. You change something - you push it and you have it.

As someone up said it, you have severe flaws in your storage concept BUT this way it might: 1) work 2) become a bit easier to debug problems 3) have at least a backup (of some sorts) of your old yml files and their contents. Again, this is a very stupid way of not-using-database, and a tricky workaround, but if setup properly it should work :D

commented

While i do completely agree that using YAML files is a very very dumb idea

You should read this workflow; #2525 (comment)

Just FYI, not a new server owner. 7+ years, just FYI I most likely understand the benefit of MySQL over 99.9% of most developers I see on spigot, but in this specific case, a YAML workflow for just permissions (NOT USERS & LOGS) is a better workflow for what ive already described as the pitfalls of the current web UI/MySQL system to manage permissions.
I have also given a few good improvements to solve the issues with the current bottlenecks in the web UI workflow you two are so keen on (I'm a web developer)

I will create my own script to do what I want until this is fixed.

commented

I have also given a few good improvements to solve the issues with the current bottlenecks in the web UI workflow you two are so keen on

have you?

commented

I have also given a few good improvements to solve the issues with the current bottlenecks in the web UI workflow you two are so keen on

have you?
#2525 (comment)

I will simplify them for you.

  1. Addition to the file-watcher service to run script/command, so we can have our own scripts do XYZ when it notices a file has been updated.
  2. Drastically improve the Web UI by allowing custom subdomains with auth, so there is no need to constantly re-login and re-create the temporary permissions each time, easily done using DNS to prove that server owns the domain its editing the permissions on.

Either would solve the current issues with the web UI "workflow".

commented

Until someone does a PR for it, it is very unlikely to be added to the plugin by Luck or myself.

commented

Or scrap all that and fix symbolic links which was my initial report, I use Symbolic links with DynMap with no issues.

  • 😕1
commented

Drastically improve the Web UI by allowing custom subdomains with auth, so there is no need to constantly re-login and re-create the temporary permissions each time, easily done using DNS to prove that server owns the domain its editing the permissions on.

Well it sounds like you aren't quite aware of how the web editor works:
To keep it simple, when opening a new session the data is uploaded to a central pastebin service hosted by Luck and for this plugin.
So a custom domain or whatever wouldn't solve the issue, as the editor never communicates directly with the server.

I've been toying with the idea of creating an extension for LP that allows the edtior to work live on the data. The main issue here being security.

Now onto the more intersting discussion: Your work flow.

From what I've gathered the main issue you face is that you are constantly updating the servers. Now to me that sounds like bad practice assuming you are not first testing everything on a test server. In case you don't that's what you should do.
Once you do this you can edit everything directly on the test server with YAML files and once you're done, export your permission changes, upload the export and a run a command. Nothing really changes about your workflow when configuring the plugins. The only thing that changes in your workflow is that you only need to upload everything once and it's done.
This also makes sure nothing bad happens during these times.

commented

Like I already said in this ticket, LuckPerms/LuckPermsWeb#300

If you have a web application running locally, why on earth would you choose to send data to a remote server to then pull the data into the web application when you have direct access via the database as you are using NodeJS installed locally which is fully capable of database queries.

This would no longer require the need for silly logging into the game to run a command to get a url to upload your permissions to a remote thrid party server, to then display it in a JS app just to edit it.

Ironic really, since I requested fixing Symbolic links to keep it simple, yet your workflow requires 100x more effort and steps and overly complicates simply editing a permissions file.

Symbolic links are just industry standard. They should be supported, end of.

commented

Symlinks were already somewhat supported - must be something unique about your setup which is causing that error to occur.. anyway, should be resolved by the above commit.

  • 👍1
commented

Symlinks were already somewhat supported - must be something unique about your setup which is causing that error to occur.. anyway, should be resolved by the above commit.

Thank you so much! So if it can be ignored it was nothing to be worried about in the first place? Just a warning from java?