Error: command not found

[medallyon]

Hi all, I was about to create a new issue on the repo, but thought to try here first to see if I'm doing anything wrong.

I'm trying to create a new Event that runs every day. This event runs a borg backup script I wrote and tested on my machine (sh ./daily-backup.sh). I copied the shell script contents into the Script Source, then manually ran the Event to test it. This results in the following error message:

Error (1): Script exited with code: 1: /opt/xyops/satellite/temp/xyops-script-temp-jmmiagzx24gt781h: line 19: borg: command not found

I also tried to run the same script as a Workflow instead of an Event, but the outcome is the same.

Am I doing something wrong or is this a bug? Many thanks =)

Ubuntu version: Ubuntu 24.04.3 LTS
xyOps version: 1.0.20 Docker (ghcr.io/pixlcore/xyops:latest)

P.S. Earlier I created a Workflow with another script just to test. The script is a one-liner and simply runs a command on my caddy docker container: docker exec -w /etc/caddy caddy caddy reload. This works without a hitch:

{"level":"info","ts":1772993116.456849,"msg":"using adjacent Caddyfile"}
{"level":"info","ts":1772993116.4622304,"msg":"adapted config to JSON","adapter":"caddyfile"}

[nickdollimount]

Off the top of my head, I can think of two possibilities. First is if your satellite is running inside the docker container and borg isn't installed inside the container and only exists on the parent host (where I assume you want it to run). Second is that you do have the satellite installed on the parent host but the path environment variable to borg isn't set up in the user context that the satellite is running as. Or third, I have no idea what I'm talking about and all of the above can be ignored 😆

[medallyon]

Thanks for the super quick reply!

I think you're spot on with the first hunch. I only started the xyops docker container using the compose file from the docks, which initializes the satellite on its own. I guess I'll try installing it as a systemd service instead!

[nickdollimount]

Sweet! In the xyOps UI, you can add a server and it will give you the command to run on the host to install the satellite service. You'll need to make sure that the docker container is configured so that the container an the host can reach each other via DNS hostname. That can be done with the hosts file, I assume, if you don't want to muck around with actual DNS.

[medallyon]

That worked! There was some friction during the Server install - I had to fiddle around with some steps, especially the weird hostname it tries to use (see screenshot), even though I've configured the Base App URL to point to the domain that I have the xyops Web GUI running on. Then I also had to deal with some random {"code":"satellite","description":"Access denied."} error responses and additionally had to adjust some options in config.json.

Now it's all installed, running, and I ran the test Event I mentioned in the OP. The borg command can be found!

However, I'm noticing some weird behaviour: The borg backup itself succeeded in around 1 minute, but the Event job is still running after 10 minutes. I sent an abort request (Abort job... button), however it's still hanging. Any tips for this situation?

[medallyon]

I restarted the xyops docker container to kill the hanging process. Now, the server that I previously installed is gone. Any idea what's going on here? =')

[nickdollimount]

I'm thinking you don't have any persistent volume configured for your container so it basically resets everything. I can't confirm because I'm just on my phone but I'm thinking you can set up a volume for /opt/xyops and that will cause the docker container to write out to a location on the host, keeping it intact during restarts. That's where the configuration is stored. Don't map it to the same location on the parent host though as you also have the satellite installed there and you don't want to possibly interfere with that.

[jhuckaby]

I had to fiddle around with some steps, especially the weird hostname it tries to use (see screenshot), even though I've configured the Base App URL to point to the domain that I have the xyops Web GUI running on.

The key here is that you actually have to set the hostname of the xyOps primary conductor server (or Docker container I am guessing in this case) to the hostname you want to advertise to other servers. It needs to be addressable on your network (DNS, hosts file, etc.). The Base App URL is not used in this case.

However, I'm noticing some weird behaviour: The borg backup itself succeeded in around 1 minute, but the Event job is still running after 10 minutes. I sent an abort request (Abort job... button), however it's still hanging. Any tips for this situation?

Please make sure you are running the latest xyOps (v1.0.20) and xySat (v1.0.12) versions. There was a bug that caused jobs to get stuck in a finishing state with some combinations of older versions (specifically xyOps v1.0.16 or older, and xySat v1.0.11 and newer).

I restarted the xyops docker container to kill the hanging process. Now, the server that I previously installed is gone. Any idea what's going on here? =')

Okay, so I think most of these things stem from the same core issue. Your xyOps primary conductor server needs to have a "real" hostname that resolves via DNS on your network. This means, if you are using Docker, you need to set the hostname of the container to something that resolves, so anything out on your network can reach it by name. This is the key to how xyOps manages a cluster of servers. It's all based on the hostname of the conductor server(s).

(The hostnames of the worker / satellite servers doesn't matter, but the conductor server very much does).

Ref: https://docs.xyops.io/#Docs/hosting

[medallyon]

Heya, quickly updating this post. It turns out most of my issues stemmed from an experiment I forgot about where I tried to move all of my docker services behind another network container.

The hostname in my compose file points to my domain, where xyops is already reverse-proxied via caddy. This meant that the New-Server URI now used the correct domain, although it still formats it as http://xyops.my-domain.com:5522/..., instead of simply https://xyops.my-domain.com/.... But after making those changes, curl successfully downloaded the satellite install script and also installed satellite without any problems.

However, I still had issues with the server disappearing after a restart of the conductor container. I made some changes to the /opt/xyops/satellite/config.json and restarted the xysat service:

• port: 5522 -> 443
• secure: false -> true

... which then successfully reconnects satellite to the conductor. However, this change wasn't persistent. After some investigation, it turns out that something was overwriting the above values in the Satellite after restarting the Conductor! After even more investigation, it turns out that the Conductor sends the instruction to overwrite the config values with those wrong values - maybe it shouldn't do that if the existing connection is working.

Finally, I noticed that the satellite.config object in the Conductor's config.json mirrors the Satellite's config.json. Changing these values to the correct ones means it's now also sending the correct config values to the Satellite, meaning that the Satellite server stays connected through the Conductor's restarts!

I was also planning on expanding the documentation with this use case, but noticed that the repo doesn't accept PRs, which is a shame. I'd love for you to include it so that others with a similar setup don't struggle as much setting up xyops: medallyon@257d6f8

Thanks for all the help!

[jhuckaby]

Very glad you were able to figure it out!

I'll look at improving the docs. I'm sorry for the confusion over the satellite.config object that gets distributed out to each satellite server on connect. I thought I documented that behavior, but possibly not well enough.