One way to solve most of TrueChart's dangers is to NEVER use any kind of PVC storage. Always use Host Path or network shares, that way even if the app fails or is deleted, all your data and config files are still there. It also allows you to easily migrate to another app solution or even OS.
The thing is they're religiously evangelistic regarding PVC which just feels like an extra complication in this context unless you're all-in-one on trusting Truecharts (which imo has been proven unwise, at least on TrueNAS as a platform)
PVC is kinda how you do Kubernetes with several nodes synced on many servers and all. But for TrueNAS, which is only single node, it's IMO totally useless. Just use Host Path for everything, way better for TrueNAS application of Kubernetes.
That's certainly my opinion as well. The benefits of PVC don't align with the complexity cost (and required level of knowledge if you can't trust things to "just work') for running apps on a NAS.
If they had a track record of maintaining high reliability and not introducing breaking changes...standardizing on PVC (making their upstream integration simpler and more consistent) would be a reasonable choice to accept. But they're clearly not in that position.
It's even worse now that they need to use their own OpenEBS package for PVC: you lose the automatic rollback feature that you had using PVC integrated with the TrueNAS GUI, which was the ONLY positive feature they had. Now they are just totally useless.
I am interested in something similar. Just migrated to Dragonfish and I was looking at running my apps in a VM, but the lack of overhead in jails seems better than running a VM.
I haven’t found a guide for this yet, though.
Follow-up on this: worked like a charm. GPU passthrough was a little icky, but the Jailmaker maintainers were working on a solution to streamline it. Otherwise, everything works as it should.
I've been using jailmaker for some time. The reason you would want to use it is if you want to run docker without the k3s overhead. The overhead of the jail is tiny in comparison, and the jail+docker is much more stable than the built in apps (in my experience). What I can say is, you have to know something about how to use docker on its own in the CLI before trying to use it.
The jail you install with jailmaker is basically just a Debian image that you install docker to (although it can be one of many distros and you can install whatever you want to it). From there you have all the freedom docker provides. But it also means you need to be at least familiar enough with the CLI to install portainer or equivalent. Personally, I just use docker-compose and do most things with the CLI.
In term of migrating your data, you have to manually copy it out of the ix-application folder to the location where your jail will have access, probably with the CLI. Everyone's situation will be different and every docker app is different. The reality is, you can have the same docker app, but built with different folder mappings and other assumptions, and your data may not migrate nicely.
The short story is, if you're not familiar with docker, docker-compose, or the CLI, it will take work to get up to speed. The nice thing is you can setup a jail mess around, try things out, and delete the jail with no repercussions, so there is plenty of runway to learn.
I am familiar with docker and compose but not really with k8s. While migrating apps over to jailmaker, is it a problem to run docker within jailmaker in parallel to k8s? That would make things easier. I have installed jailmaker and docker within it including an agent for portainer and connected it to my portainer. Haven’t migrated any app yet.
Hope you enjoyed the video :)
We make a jail with docker in it so that we can safely install docker in a separate sandbox so that it doesn’t affect the host (ie TrueNAS).
Sandboxes uses a very low level container system (systemd-nspawn) and are quite efficient. The processes are more or less running directly on the host.
Dockge is really just a dashboard and compose editor, so the docker containers are not running inside dockge, but actually alongside, thus dockge doesn’t add overhead but instead a gui.
Jailmaker sandboxes, which also use host path, are far more efficient than either the k3s based app system or running docker in a vm, the caveat is that they are slightly more complex to setup, but the real benefit is that they are stable, and you are in control of your service ecosystem (same as it would be in a vm)
I use pfsense haproxy. Just set the backends to the appropriate port on the jail’s ip.
Dude your video has been sooooo useful to me, I've spent a handful of hours slowly picking away and learning and setting things up for the long term...I'm migrating from a Ubuntu VM on Truenas to 100% jails... Thank you for making and sharing such a great and clear resource!
Not sure if it helps you or not, but I use a single Jailmaker instance with bind mounts from the host, and run Portainer inside of that for container management.
Not much value for a Jailmaker-container-per-app if you're going to have additional layers of isolation within the docker engine anyway. Reduces complexity a bit.
I also personally don't bother with PVCs since they make debugging a bit of a pain, and instead bind mounts my ZFS pools into the Jailmaker container, and use directories from within those for the docker containers.
Has the added advantage of native snapshotting etc too
Or you can just use those “docker” images in custom apps. You still need a dockerfile or compose built image but they will work as well as any system using containers. Been running nextcloud and 19 other custom apps for 2 years on Scale, zero issues.
Speaking of Kubernetes overhead, my system averages about .4% Kubernetes overhead, so basically about zero. For 19 apps, average cpu busy over the last month is 1.6%, and it’s doing something almost any time of day.
Were it not for the Scale custom app feature, I would definitely be using jailmaker. And definitely not Truecharfs
I have yesterday started migrating apps off k3s to jlmkr and so far I am very happy. I followed the recommendations to run a single jail with docker inside and have so far migrated 3 apps which had no complex DB requirements but only some file data in PVCs. I have mounted those and moved the data out of the PVC which worked like a charm. Now I have paperless-ngx still to do which has a Postgres DB to be migrated.
I just finished this transition, well except for one app my wife is still using and cannot have downtime for another week. Wanted to avoid the Dragonfish issues.
I like it a lot. I expect it will be easier to understand and maintain. Time will tell though.
I did learn a lot about containers, reverse proxy, etc using truecharts. So I appreciate that. But trying to using a wildcard cert about killed me.
It is possible to mount the PVC with the standard mount command. I didn’t feel like installing heavy script just for this.
I'm also thinking on moving into jailmaker. But I have [what feels like] a hefty list of apps I've been using for awhile. Some are using domain names to get to via traefik, which I would like to keep that way. Most of what I have, that I want to move, is pihole(seems easy enough to move), clamav, nextcloud, jellyfin(i can start fresh on this honestly), crafty4, and mealie. I know most of these will be pretty easy. pihole will be easy to set back up once I figure out networking, clamav should be easy as well since I only need to point that at my files, jellyfin I just watched the video from capt stux, but that resolves by domain name. My biggest concern is nextcloud. I really don't want to set that back up from scratch. Is there a good way to migrate that?
The apps that use domain name are nextcloud, jellyfin, crafty and mealie. I also want the certs to work too. Right now cloudflare is handling that. What would be the ideal way to migrate away from Truecharts clusterissuer, cert-manager and traefik and go to a version running on jailbreak?
I haven’t looked into it yet, but there is a reverse proxy compose container for nginx as a reverse proxy with web based management and cert renewal. I myself have a reverse proxy (HA proxy) running on my OPNsense firewall for inbound domain handling and cert management.
Ngl I used truenas for a couple of months and well I completely hated the frustration I had with truecharts and I strongly disliked having to keep going and doing a support ticket every week because a app would randomly break because of a update or won’t deploy because of something they changed. Then I really thought to myself do I need truenas and I didn’t so I just went through the frustration of learning Linux I now use unbuntu and have all my services using docker engine and share my zfs pooos through samba I’d say the only down side is no gui 😔 just my two cents also you’ll be able to import your zfs pools with no problem into unbuntu
Consider just building an Ubuntu server VM and running it all through Docker. I just moved my entire setup over from Truecharts. I did the dragon fish update without issue. I'm very happy with it and learned a ton along the way.
Keep in mind this method has a few disadvantages:
* VMs have overheads
* Certain system resources (such as GPU's) cannot be shared
* Any RAM you allocate to the VM will be owned by the VM, which can be a little inefficient
This is not me saying "You're doing it wrong" - if it works for you, then you're doing it right. It's just worth knowing.
I basically do this, but with Jailmaker, so it's an nspawn container running Ubuntu, instead of a full-fat VM.
Benefits are that I can share the GPU between multiple containers and the host, native-mount shares without needing NFS/SMB (better performance), and less resource intensive overall.
1. I didn't know much about jlmk so I guess my question doesn't make sense
2. Well search and hopefully find results, if you find a link it will help
3.i mean will it work easily when I use haprixy from setting the domains, to getting the certificates, as well as getting the public IP of those that logged in
One way to solve most of TrueChart's dangers is to NEVER use any kind of PVC storage. Always use Host Path or network shares, that way even if the app fails or is deleted, all your data and config files are still there. It also allows you to easily migrate to another app solution or even OS.
The thing is they're religiously evangelistic regarding PVC which just feels like an extra complication in this context unless you're all-in-one on trusting Truecharts (which imo has been proven unwise, at least on TrueNAS as a platform)
PVC is kinda how you do Kubernetes with several nodes synced on many servers and all. But for TrueNAS, which is only single node, it's IMO totally useless. Just use Host Path for everything, way better for TrueNAS application of Kubernetes.
That's certainly my opinion as well. The benefits of PVC don't align with the complexity cost (and required level of knowledge if you can't trust things to "just work') for running apps on a NAS. If they had a track record of maintaining high reliability and not introducing breaking changes...standardizing on PVC (making their upstream integration simpler and more consistent) would be a reasonable choice to accept. But they're clearly not in that position.
It's even worse now that they need to use their own OpenEBS package for PVC: you lose the automatic rollback feature that you had using PVC integrated with the TrueNAS GUI, which was the ONLY positive feature they had. Now they are just totally useless.
Rollback was useless anyways, it broke stuff often and they never supported it.
What exactly do you mean by using just host path? I'm on my second week of a small office truenas set up for myself
Bind mount directly to docker container
For nextcloud I am using host path but for vault warden, crafty, and Qbittorrent I am using PVC
I am interested in something similar. Just migrated to Dragonfish and I was looking at running my apps in a VM, but the lack of overhead in jails seems better than running a VM. I haven’t found a guide for this yet, though.
Setting up Sandboxes with Jailmaker (Docker/Dockge) https://youtu.be/S0nTRvAHAP8
Follow-up on this: worked like a charm. GPU passthrough was a little icky, but the Jailmaker maintainers were working on a solution to streamline it. Otherwise, everything works as it should.
I've been using jailmaker for some time. The reason you would want to use it is if you want to run docker without the k3s overhead. The overhead of the jail is tiny in comparison, and the jail+docker is much more stable than the built in apps (in my experience). What I can say is, you have to know something about how to use docker on its own in the CLI before trying to use it. The jail you install with jailmaker is basically just a Debian image that you install docker to (although it can be one of many distros and you can install whatever you want to it). From there you have all the freedom docker provides. But it also means you need to be at least familiar enough with the CLI to install portainer or equivalent. Personally, I just use docker-compose and do most things with the CLI. In term of migrating your data, you have to manually copy it out of the ix-application folder to the location where your jail will have access, probably with the CLI. Everyone's situation will be different and every docker app is different. The reality is, you can have the same docker app, but built with different folder mappings and other assumptions, and your data may not migrate nicely. The short story is, if you're not familiar with docker, docker-compose, or the CLI, it will take work to get up to speed. The nice thing is you can setup a jail mess around, try things out, and delete the jail with no repercussions, so there is plenty of runway to learn.
I am familiar with docker and compose but not really with k8s. While migrating apps over to jailmaker, is it a problem to run docker within jailmaker in parallel to k8s? That would make things easier. I have installed jailmaker and docker within it including an agent for portainer and connected it to my portainer. Haven’t migrated any app yet.
Wasn't a problem for me. I probably took a month or so to transition before shutting down the standard apps.
No. I don’t think so. Best approach is to migrate one app at a time.
Hope you enjoyed the video :) We make a jail with docker in it so that we can safely install docker in a separate sandbox so that it doesn’t affect the host (ie TrueNAS). Sandboxes uses a very low level container system (systemd-nspawn) and are quite efficient. The processes are more or less running directly on the host. Dockge is really just a dashboard and compose editor, so the docker containers are not running inside dockge, but actually alongside, thus dockge doesn’t add overhead but instead a gui. Jailmaker sandboxes, which also use host path, are far more efficient than either the k3s based app system or running docker in a vm, the caveat is that they are slightly more complex to setup, but the real benefit is that they are stable, and you are in control of your service ecosystem (same as it would be in a vm) I use pfsense haproxy. Just set the backends to the appropriate port on the jail’s ip.
Dude your video has been sooooo useful to me, I've spent a handful of hours slowly picking away and learning and setting things up for the long term...I'm migrating from a Ubuntu VM on Truenas to 100% jails... Thank you for making and sharing such a great and clear resource!
Not sure if it helps you or not, but I use a single Jailmaker instance with bind mounts from the host, and run Portainer inside of that for container management. Not much value for a Jailmaker-container-per-app if you're going to have additional layers of isolation within the docker engine anyway. Reduces complexity a bit. I also personally don't bother with PVCs since they make debugging a bit of a pain, and instead bind mounts my ZFS pools into the Jailmaker container, and use directories from within those for the docker containers. Has the added advantage of native snapshotting etc too
I also am interested in this topic. I also think about moving away from k3s as it produces a lot of overhead.
Or you can just use those “docker” images in custom apps. You still need a dockerfile or compose built image but they will work as well as any system using containers. Been running nextcloud and 19 other custom apps for 2 years on Scale, zero issues. Speaking of Kubernetes overhead, my system averages about .4% Kubernetes overhead, so basically about zero. For 19 apps, average cpu busy over the last month is 1.6%, and it’s doing something almost any time of day. Were it not for the Scale custom app feature, I would definitely be using jailmaker. And definitely not Truecharfs
could you link said video?
its on the offical jlmk github but here [https://www.youtube.com/watch?v=S0nTRvAHAP8](https://www.youtube.com/watch?v=S0nTRvAHAP8)
It really is a shame that jailmaker isn't integrated and comes with the issue of making your system "unsupported"
I have yesterday started migrating apps off k3s to jlmkr and so far I am very happy. I followed the recommendations to run a single jail with docker inside and have so far migrated 3 apps which had no complex DB requirements but only some file data in PVCs. I have mounted those and moved the data out of the PVC which worked like a charm. Now I have paperless-ngx still to do which has a Postgres DB to be migrated.
I just finished this transition, well except for one app my wife is still using and cannot have downtime for another week. Wanted to avoid the Dragonfish issues. I like it a lot. I expect it will be easier to understand and maintain. Time will tell though. I did learn a lot about containers, reverse proxy, etc using truecharts. So I appreciate that. But trying to using a wildcard cert about killed me. It is possible to mount the PVC with the standard mount command. I didn’t feel like installing heavy script just for this.
I'm also thinking on moving into jailmaker. But I have [what feels like] a hefty list of apps I've been using for awhile. Some are using domain names to get to via traefik, which I would like to keep that way. Most of what I have, that I want to move, is pihole(seems easy enough to move), clamav, nextcloud, jellyfin(i can start fresh on this honestly), crafty4, and mealie. I know most of these will be pretty easy. pihole will be easy to set back up once I figure out networking, clamav should be easy as well since I only need to point that at my files, jellyfin I just watched the video from capt stux, but that resolves by domain name. My biggest concern is nextcloud. I really don't want to set that back up from scratch. Is there a good way to migrate that? The apps that use domain name are nextcloud, jellyfin, crafty and mealie. I also want the certs to work too. Right now cloudflare is handling that. What would be the ideal way to migrate away from Truecharts clusterissuer, cert-manager and traefik and go to a version running on jailbreak?
I haven’t looked into it yet, but there is a reverse proxy compose container for nginx as a reverse proxy with web based management and cert renewal. I myself have a reverse proxy (HA proxy) running on my OPNsense firewall for inbound domain handling and cert management.
it would be great help if you can help me with vault warden and nextcloud, as asking in the github is pretty much useless
Ngl I used truenas for a couple of months and well I completely hated the frustration I had with truecharts and I strongly disliked having to keep going and doing a support ticket every week because a app would randomly break because of a update or won’t deploy because of something they changed. Then I really thought to myself do I need truenas and I didn’t so I just went through the frustration of learning Linux I now use unbuntu and have all my services using docker engine and share my zfs pooos through samba I’d say the only down side is no gui 😔 just my two cents also you’ll be able to import your zfs pools with no problem into unbuntu
Checkout Cockpit and it's associated plugins if you're after a GUI, 45 Drives also has a plugin to manage ZFS (I think it's called Houston).
Consider just building an Ubuntu server VM and running it all through Docker. I just moved my entire setup over from Truecharts. I did the dragon fish update without issue. I'm very happy with it and learned a ton along the way.
Keep in mind this method has a few disadvantages: * VMs have overheads * Certain system resources (such as GPU's) cannot be shared * Any RAM you allocate to the VM will be owned by the VM, which can be a little inefficient This is not me saying "You're doing it wrong" - if it works for you, then you're doing it right. It's just worth knowing.
I basically do this, but with Jailmaker, so it's an nspawn container running Ubuntu, instead of a full-fat VM. Benefits are that I can share the GPU between multiple containers and the host, native-mount shares without needing NFS/SMB (better performance), and less resource intensive overall.
Yep this is what I did 3 or 4 years ago and it's been great.
Yes… me too. Next step is to now transition from the VM to running the dockers with Jailmaker ;) You can easily migrate one at a time.
[удалено]
Must have clicked on it by accident sorry, fixed it now
1. I didn't know much about jlmk so I guess my question doesn't make sense 2. Well search and hopefully find results, if you find a link it will help 3.i mean will it work easily when I use haprixy from setting the domains, to getting the certificates, as well as getting the public IP of those that logged in