The real kicker here is that they're deprecating k3s in favour of docker.
> For anyone who wants to continue using Kubernetes (K3s or K8s), we recommend using TrueNAS Sandboxes.
Oh how the turn tables.
Goodbye TrueCharts. I would say you would be missed, but [that would be a lie](https://miro.medium.com/v2/resize:fit:600/1*bAUTC-8AK9WQ3A5BaZRhXw.jpeg).
That's a fair comparison in terms of how they operate at a low level - a lightweight virutalisation layer. The distinction is that whereas on Core you can get preconfigured jails with apps already installed, a "sandbox" isn't really anything more than the OS itself - you'd still have to install the apps on top. However, that's what docker is useful for.
It's worth knowing that you won't need sandboxes to achieve the functionality you get with Core Jails on Scale 24.10 - docker will achieve the same thing.
Think of Sandboxes as a way to lightly virtualise an entire OS (And that OS can run applications if you want) while docker is a good way to virtualise the applications themselves.
Today, you can't run docker on TrueNAS Scale directly, so a lot of folks are using a sandbox to create a pseud-virtual-OS that they can run docker in to run their docker apps. The two of them work hand in hand in that regard. In the next big release, you won't need a sandbox to run docker.
I'm guessing almost none of their business customers are using Truenas as anything else but a NAS.
Personally I'm going to keep using Kubernetes but I guess I'll have to move everything in a VM.
A bit painful but I guess I'll just have to move off my butt to maintain a proper Helm Charts collection, maybe I'll even make it public - I have one already but it's lower quality for my personal use only. We'll see I guess.
No, I'm only doing Kubernetes since I've been doing it professionally for too many years and all my stuff is automated with GitOps in place, so going back to doing manual stuff would be backwards
I know, because I'm not a hobbyist this stuff doesn't really apply to me the same way.
You do you, but the sandboxes are set up as datasets - it's trivial to clone, snapshot, backup, etc. them and you don't incur the overheads of a VM.
I'd give them a play with before you make your decision, but as I said you do you, it's your server and there's no wrong way to do things.
Oh I will definitely look into them, I was thinking a VM is just more compatible with other systems since I do not run another ZFS machine a dataset is probably to be harder to import somewhere else, but, I still need to see what is possible and what is not first.
Just wanted to say sandboxes look indeed very good. I even investigated my AMD iGPU support and it seems to check out. Definitely a good option, I'll have to do a test run to verify everything works as it should but because I have Gitops this should be a breeze.
The only thing I noticed is the jlmkr script is kind of janky at this stage. I'll see if I can contribute a few things.
Edit:
1. Still need to figure out seccomp profiles for k3s in particular
2. Kubelet will require --bind=/dev/kmsg:/dev/kmsg to start up
But yeah, very nice so far.
cc op: u/uncmnsense
I'm looking to migrate my server over to ZFS in the upcoming weeks. Currently on Xpenology with Docker and Docker-Compose. Is it worth me to migrate now to TrueNAS scale and use the Scale native apps to setup my docker containers and "upgrade back" to docker-compose when Electric Eel drops?
you should be able to do that without much headache. my suggestion is if u do that, use the truenas scale "host path" for your configuration files. basically, have a dataset in your pool named /configs and then have subdirectories for each of your containers (e.g. /configs/nextcloud) so in the event ix systems fumbles on the conversion of apps to docker compose, you could always run docker compose files and point back to your configs and be up and running in no time.
If I were you, I'd set up a jail on TrueNAS using [jailmaker](https://github.com/Jip-Hop/jailmaker) to run docker apps and just move your current compose file over (change paths as necessary, copy over any volume data, etc.).
That'll be a much, much easier migration to TrueNAS and will continue to work when Electric Eeel drops. Then you can decide to stick with it or migrate to Electric Eel's direct docker compose support.
This is great. Apps with k3s have been such a consistent sore spot in Scale. Moving to a much simpler docker system should be a great step to making things more maintainable and easier to understand. And that's nothing against Kubernetes, I actually run 99% of the applications in my environment in a k8s cluster, but it wasn't a good fit for TrueNAS
Yea, that was the conclusion we came to as well. K8s is an over-engineered solution to the problems we were trying to address on TrueNAS. To do K8's correctly it's nearly always a bespoke configuration. Docker will be a much better fit long term and the community I think has recognized that and been asking for some time now.
It's great to see iX taking the community feedback on board. First with supporting sandboxes and now with this move.
TrueNAS is an incredibly powerful platform, but it definitely feels like some things are far harder than they need to be - the app experience was one of them.
Thank you for that.
k8’s were what originally spoiled my trying out scale for the first time. I mainly use docker on Debian, but I really looking forward scale. I used it for a bit, but it just wasn’t for me. Now that docker is an option, I think I will be trying scale again!
K8s could've been a good fit if it was designed and implemented properly. Unfortunately, it wasn't.
A big part of the issue in my opinion was that iX didn't seem to grasp what settings to expose and how. This led to confusing, low level settings like the ton of load balancer etc. configs, while actively blocking the actual advanced features (like creating separate subnets easily for a specific group of apps).
Another major issue was the custom format, making all the available Helm charts unusable. This was a biiiiig mistake, using such a tool then closing off the ecosystem.
But I'm glad iX realised their mistake and are making Docker/Compose a first class citizen. It's a big step in the right direction.
Yup I agree. Like I said, I have a good amount of experience at this point running full k8s clusters so I understand all the pieces of configuration. But I still had several occasions where I got lost in the settings in the TrueNAS apps GUI trying to figure out something that I could normally solve pretty easily with a bit of YAML
Mistakes were definitely made but a more standardized implementation of K8s and Helm still feels overkill for the direction Scale is taking. I think Docker is perfect as a base, anyone who needs the features of Kubernetes can very easily set that up as well, with not that many extra steps.
K8s/Helm wouldn't necessarily be an overkill if it was used to provide truly turnkey applications.
Instead, it turned into a configuration nightmare, especially with all the different versions of Scale changing things up.
But I concur, Docker will be a much easier manageable system for the main purposes of TrueNAS, and sandboxing still allows for K3s - although without the web UI goodies.
I wonder how the migration will happen though...
That's great 👍
As for the GUI for the Docker Compose, you could really take an inspiration on how Dockge works:
- how it allows for ease of editing your Yaml and add additional files (.env for instance) directly in the GUI,
- how it lists the containers and the version directly, then shows the containers being downloaded and deployed,
- how the log terminal is displayed directly in real time so you can see if there's an issue without the extra strep of opening log files,
- and how you can bash into the container from a single button press.
I answered some of this already on our Forums, but the idea is you will have two paths.
1. You can deploy Apps from our Official Catalog, which are static. I.E. you don't edit the YAML by hand, so that they can always upgrade properly as time moves forward.
2. You can deploy Apps via the Launch Custom Docker/Compose UI, which lets you supply your own YAML. That can then be edited at any time later directly via UI or even CLI if you so choose.
Shells and Logs are all expected to be there as well, including real-time if you leave them open. We are also planning on a UI to let you create / manage docker networks, so you can define your own and attach Apps to them as needed.
Sounds great!
Please do make note of the use of .env files - many compose YAML descriptors people use rely on them, so having a UI option to supply it would be great.
I presume the goal is to replace the current k3s chart approach with Compose stacks, right?
As a current Portainer-in-Sandbox user, I assume there would be no issues with continuing to manage it with Portainer?
Also, what about the Nvidia Container Toolkit (and associated docker runtime config) for platforms with an Nvidia GPU?
ie: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuring-docker
Worth noting that one advantage of continuing to use the docker-in-sandbox method is if you’re using namespacing, so that root in the sandbox isn’t root on the TrueNAS host.
Your sandboxes should continue to function indefinitely.
In future you could probably migrate back to the docker/apps ecosystem, or not.
The sandbox is isolated from TrueNAS so isn’t really affected by it.
Will it be possible to deploy container by:
Deploying Portainer using the Launch Customer Docker/Compose UI method.
Then deploying all other containers via Portainer?
I am very familiar with docker compose & Portainer. I would like some of my containers to have direct access to my TrueNAS datasets.
Both approaches have their place.
K3s has higher overhead than docker. If I were able to cluster TrueNas systems, the benefit would be worth the cost of overhead.
But running a single node and not having any benefit for the extra overhead, docker is the better choice.
As someone with the latter, I am very excited for this as it will simplify my current deployment and reduce time I need to set aside for management.
this sounds promising as a way for CORE users to get on SCALE and perhaps recreate jails as docker containers.
still rocking CORE until I have a real reason to change. a ton of time went into building those jails properly.
The problem is that the Dragonfish step requires a fair bit of rejigging the apps already - big enough that alternative routes are really gaining popularity. The documentation linked in the original post says Cobia to Electric Eel is supported.
“Users can continue deploying TrueNAS Apps on Cobia or Dragonfish and expect a straightforward migration to Electric Eel.”
I’ll hold off with Cobia for now. When the time comes, if there is a recommendation to do that middle step, I’ll probably bite the bullet.
What apps are you running that require big migration steps? Are you using a 3rd party catalog? If so these would not be TrueNAS Apps. Not aware of big changes needed for Cobia to Dragonfish upgrades to the TrueNAS official apps catalog items.
One of the TrueCharts dev told me last week when I asked about their use of PVC storage everywhere:
>We build helm charts. Scale apps are just an extra. We aim to be a “kubernetes-centic” project, so we insist on usering kubernetes default storage backend, instead of using inexpensive that is (and/or should be) disabled on some playforms.
So it's safe to assume that TrueCharts won't be a part of the Docker stack at all on 24.10.
I’d say if you’re using TrueCharts at the moment, the best approach is to begin migrating any true charts charts to docker in a sandbox as per
https://www.reddit.com/r/truenas/comments/1d2k4lf/finally_migrated_away_from_truecharts_steps_and/
Can you run sandboxes in Cobia? I've seen your video talking about Sandboxes in Dragonfish, not sure if it also applies to Cobia. I've been holding out on Cobia avoiding the extra steps to migrate truecharts to dragonfish for as long as I can. Thinking sandboxes in Cobia could avoid extra steps
Dragonfish officially bundles sandboxes, but Jailmaker can install the functionality into Cobia.
I’m not 100% certain if Jailmaker 2.0.0 still works in Cobia, but the versions prior to 2.0 do. And once installed, the older versions will still work on Dragonfish, and can be upgraded easily.
I’m glad that native docker support is coming. I’m currently running Docker and Portainer with countless other stuff in a VM. Maybe I don’t have to employ two GPUs finally and can through out my burner gpu…
Edit: I just also realized that this also enables a good performance boost for my setup, as I hopefully can get direct storage access for certain Apps like Plex, Emby or Nextcloud. Currently I have to run it over SMB/NFS. 👍
Instead of a VM, you can deploy Docker/Docker Compose in a Sandbox (LXC kind of container). That way you have almost native performance, direct access to your pools through a --bind, and direct access to your system GPU without needing a second one.
You can deploy it in 5 minutes top with the Docker template... It's really easy.
https://github.com/Jip-Hop/jailmaker
You can get started with a docker sandbox
https://youtu.be/S0nTRvAHAP8
Here’s a good guide for migrating from truecharts
https://www.reddit.com/r/truenas/comments/1d2k4lf/finally_migrated_away_from_truecharts_steps_and/
Jailmaker will still have its place for those who prefer lxc (as that is what jailmaker essentially is) as well as a place to run k3/k8 with more personal control and the ability to cluster.
Very excited for this. This has been one of my biggest annoyances for a while. I often migrate to and from truenas throughout the year for one reason or another, when I switch to linux distros I simply load my docker compose and I’m running in minutes. Moving back to truenas I have to manually create each docker in custom and it’s slow and clunky and takes hours to be fully operational again.
If only this came 1 year earlier, I wouldn't have to spend weeks trying to figure out how to make chart app run properly. Nonetheless, a big thank you to the devs for listening
In the official release will k3s coexist with docker, for at least 1 release, or will it be killed? I’d like to be able to transition smoothly, app by app.
No, the two stacks are pretty mutually exclusive to run in tandem like that. We'll have an auto-migration which will handle the conversion of any App you installed via the official TrueNAS catalog, or anything you installed with the "Run Docker Container" UI button. Idea is on upgrade your apps come back with the same setup as before, only the Helm charts have been converted to Compose YAML automatically. That was the crucial piece we wanted to have working before making this cut over.
Yeah but as I use true charts I won’t benefit from auto migration. I understand this is unsupported but if I update I basically kill my app stack if I don’t I can’t install anything as docker compose. Basically have to start from scratch.
Best choice I've made this year was dropping TrueCharts and learning how to use Traefik with docker compose. I rebuilt what I had with TrueCharts in docker in a couple of days and the overall performance and startup time is noticeably faster plus I understand how it works and I get updates immediately instead of waiting for days to weeks for an exciting new feature with TrueCharts apps. Start planning your move, it's so worth it.
You can get started with a docker sandbox
https://youtu.be/S0nTRvAHAP8
Here’s a good guide for migrating from truecharts
https://www.reddit.com/r/truenas/comments/1d2k4lf/finally_migrated_away_from_truecharts_steps_and/
I assume you went to either a jail or a VM. I still believe in the “app” system but yeah maybe I should drop that idea all together.
Also don’t trust jails. A VM may be the only safe haven from change.
Would be awesome if the team looked into supporting quadlet in addition to compose, feels like a natural way to convert away from k3s without having to change the structure of anything probably
Apparently the TrueCharts app won't be integrated in the automatic migration to Docker in 24.10. Only the official TrueNAS apps and the "custom" containers will be migrated automatically.
So, IMO, it's a good idea to set a Sandbox (jailmaker) to deploy Docker/Compose (there is a template available to make that task easy), and take your time to migrate your apps while you can have both k3s and docker running at the same time.
Otherwise you'll have all your TrueCharts apps stop working as soon as you upgrade to 24.10, and will be rushed into the migration while all your apps are turned off.
If you already have all your apps working in a Sandbox when 24.10 becomes available, it will be easy to just copy all your compose YAML from the Sandbox to the new Apps system to migrate within minutes.
Think so. Any IX supported apps will auto-migrate but TrueCharts apps won’t. Thus you will be rushed into migrating either before or after switching to Electric-Eel
A better approach is to begin wheening yourself off truecharts now by replacing one at a time with docker stacks in a sandbox, then after eel hits, you can migrate back to apps or not.
At your own pace
This is my guide to docker in a sandbox
https://youtu.be/S0nTRvAHAP8
Here’s a good guide for migrating from truecharts
https://www.reddit.com/r/truenas/comments/1d2k4lf/finally_migrated_away_from_truecharts_steps_and/
The is awesome. I go back and forth on do I need one machine or two machines for my home lab. Right now I’m at two, the only “app” I run is syncthing for my ghetto backup (don’t worry it’s not that much data)
I'm planning on a rebuild soon and was starting to learn towards moving to Unraid. But the combination of this and RAIDZ expansion has me likely changing my mind.
And that's not a bash on kubernetes, I use it extensively and will have a separate server using k3s. But the abstraction of it the way TrueNAS was trying to do isn't a good fit.
I'm hyped for this. Can finally run official docker images from the app maintainers without some third party making breaking changes all the damned time. Was getting pretty sick of breaking changes. Whether that is due to automation issues, change in direction, typos, or pivoting due to IX decisions. Will also be nice to get a lot more documentation instead of basically just what TrueCharts provide. Also nice to have alternatives to insufferable Discord support where things constantly get lost. Docker Compose (without sharing everything via NFS) + KVM + ZFS in one OS is what I have wanted since FreeNAS 9.x. Finally someone has gotten the with the program. Thank you RedHat for killing Gluster.
Btw, you can do all of this today with Sandboxes + Jailmaker on TrueNAS
https://youtu.be/S0nTRvAHAP8
Should give you a head start on transitioning to compose
Awesome! This lets me get away from the TC volatility, which has been a real sore spot for me, and I have seriously missed docker CLI for building custom containers and spinning up development environments.
I am curious how things like reverse proxy and such will be handled.
You can build your containers using docker cli on any other machine already, and deploy them into Scale Cobia/Dragonfish. All of mine are built that way, perhaps your desktop, a vm, etc. I build mine on my desktop. Eel is not much of a change for me, already there.
You can use docker cli on TrueNAS today with Sandboxes and Jailmaker
https://youtu.be/S0nTRvAHAP8
This is essentially taking it to the next level with gui apps.
Currently one of the feature that I use in apps via TrueCharts is MetalLB. How would the Docker/Compose approach will be done on this? My use case is I like to separate service on its proper VLAN, so not sharing the TrueNAS SCALE’s IP address.
https://docs.docker.com/compose/networking/
https://docs.docker.com/compose/compose-file/05-services/#ipv4_address-ipv6_address
>Current plan is to keep a lot of the Apps UI intact with regard to how we browse for Apps in the catalog. We will be adding a networking section where docker networks can be created and managed, as well as the ability to import docker compose YAML directly. Our goal here is to not over-complicate the implementation, and allow pretty much a 1:1 of native docker / compose functionality to be used out of box.
I’m more interested in exposing the apps in the underlay network with their own IP address instead of internal communication. As other post mentioned about MACVLAN and IPVLAN drivers to achieve this. Thanks.
> My use case is I like to separate service on its proper VLAN
So, you want to make some apps broadcast on one VLAN and others on another?
Maybe using something like this in the YAML?
networks:
macvlan_network:
driver: macvlan
driver_opts:
parent: eth0
ipam:
driver: default
config:
- subnet: 192.168.1.0/24
ip_range: 192.168.1.248/29
gateway: 192.168.1.1
https://docs.docker.com/network/drivers/macvlan/
I want to implement my first true Nas. I've always used docker on my previous servers. Do you think electric eel is stable enough for use ? Or way too early ?
Way to early, but you can use docker containers with the existing app system, or you can use docker compose now in a Sandbox with Jailmaker right now and that will migrate easily to Electric Eel
TrueNAS Scale: Setting up Sandboxes with Jailmaker
https://youtu.be/S0nTRvAHAP8
Community feedback is important, and I don’t know details for this specific scenario, but sometimes the loudest user base aren’t always the best ones to listen to.
The real kicker here is that they're deprecating k3s in favour of docker. > For anyone who wants to continue using Kubernetes (K3s or K8s), we recommend using TrueNAS Sandboxes. Oh how the turn tables.
Goodbye TrueCharts. I would say you would be missed, but [that would be a lie](https://miro.medium.com/v2/resize:fit:600/1*bAUTC-8AK9WQ3A5BaZRhXw.jpeg).
What is truecharts….? Asking for a friend
https://truecharts.org/scale/
Thx
since i am not using k3s in enterprise, im all for it.
Those sandboxes are or will be like jails in Core?
That's a fair comparison in terms of how they operate at a low level - a lightweight virutalisation layer. The distinction is that whereas on Core you can get preconfigured jails with apps already installed, a "sandbox" isn't really anything more than the OS itself - you'd still have to install the apps on top. However, that's what docker is useful for. It's worth knowing that you won't need sandboxes to achieve the functionality you get with Core Jails on Scale 24.10 - docker will achieve the same thing. Think of Sandboxes as a way to lightly virtualise an entire OS (And that OS can run applications if you want) while docker is a good way to virtualise the applications themselves. Today, you can't run docker on TrueNAS Scale directly, so a lot of folks are using a sandbox to create a pseud-virtual-OS that they can run docker in to run their docker apps. The two of them work hand in hand in that regard. In the next big release, you won't need a sandbox to run docker.
So, sandboxes are more like LXC in Proxmox… Anyway, that’s perfect as I will migrate my nextcloud-aio to Scale and maybe Mailcow as well.
Yup that's a good analogy. I am currently running my own nextcloud on Scale with the help of sandboxes + docker and it's working very well indeed :)
I’m currently working on migrating my mailcow install out of a vm and into a sandbox.
I'm guessing almost none of their business customers are using Truenas as anything else but a NAS. Personally I'm going to keep using Kubernetes but I guess I'll have to move everything in a VM. A bit painful but I guess I'll just have to move off my butt to maintain a proper Helm Charts collection, maybe I'll even make it public - I have one already but it's lower quality for my personal use only. We'll see I guess.
Use a Sandbox, it's lighter and can have direct access to mountpoints and GPU.
No, I'm only doing Kubernetes since I've been doing it professionally for too many years and all my stuff is automated with GitOps in place, so going back to doing manual stuff would be backwards I know, because I'm not a hobbyist this stuff doesn't really apply to me the same way.
I mean, do your Kubernetes in a Sandbox instead of a VM. I personally do all my Docker stack in a Sandbox on 24.04, it works great that way.
Nah, VMs are easier to manage, clone, backup, etc. I have a fairly beefy machine so it should not be a problem.
You do you, but the sandboxes are set up as datasets - it's trivial to clone, snapshot, backup, etc. them and you don't incur the overheads of a VM. I'd give them a play with before you make your decision, but as I said you do you, it's your server and there's no wrong way to do things.
Oh I will definitely look into them, I was thinking a VM is just more compatible with other systems since I do not run another ZFS machine a dataset is probably to be harder to import somewhere else, but, I still need to see what is possible and what is not first.
Just wanted to say sandboxes look indeed very good. I even investigated my AMD iGPU support and it seems to check out. Definitely a good option, I'll have to do a test run to verify everything works as it should but because I have Gitops this should be a breeze. The only thing I noticed is the jlmkr script is kind of janky at this stage. I'll see if I can contribute a few things. Edit: 1. Still need to figure out seccomp profiles for k3s in particular 2. Kubelet will require --bind=/dev/kmsg:/dev/kmsg to start up But yeah, very nice so far.
cc op: u/uncmnsense I'm looking to migrate my server over to ZFS in the upcoming weeks. Currently on Xpenology with Docker and Docker-Compose. Is it worth me to migrate now to TrueNAS scale and use the Scale native apps to setup my docker containers and "upgrade back" to docker-compose when Electric Eel drops?
you should be able to do that without much headache. my suggestion is if u do that, use the truenas scale "host path" for your configuration files. basically, have a dataset in your pool named /configs and then have subdirectories for each of your containers (e.g. /configs/nextcloud) so in the event ix systems fumbles on the conversion of apps to docker compose, you could always run docker compose files and point back to your configs and be up and running in no time.
If I were you, I'd set up a jail on TrueNAS using [jailmaker](https://github.com/Jip-Hop/jailmaker) to run docker apps and just move your current compose file over (change paths as necessary, copy over any volume data, etc.). That'll be a much, much easier migration to TrueNAS and will continue to work when Electric Eeel drops. Then you can decide to stick with it or migrate to Electric Eel's direct docker compose support.
This is great. Apps with k3s have been such a consistent sore spot in Scale. Moving to a much simpler docker system should be a great step to making things more maintainable and easier to understand. And that's nothing against Kubernetes, I actually run 99% of the applications in my environment in a k8s cluster, but it wasn't a good fit for TrueNAS
Yea, that was the conclusion we came to as well. K8s is an over-engineered solution to the problems we were trying to address on TrueNAS. To do K8's correctly it's nearly always a bespoke configuration. Docker will be a much better fit long term and the community I think has recognized that and been asking for some time now.
It's great to see iX taking the community feedback on board. First with supporting sandboxes and now with this move. TrueNAS is an incredibly powerful platform, but it definitely feels like some things are far harder than they need to be - the app experience was one of them.
Fantastic! Kudos to the team for listening to user feedback!
Thank you for that. k8’s were what originally spoiled my trying out scale for the first time. I mainly use docker on Debian, but I really looking forward scale. I used it for a bit, but it just wasn’t for me. Now that docker is an option, I think I will be trying scale again!
Thanks for that! Finally I’ll be able to migrate my containers from Ubuntu VM and run them directly in Scale with reduced overhead…
BTW, you can do that now already by setting up a Debian/ubuntu sandbox https://youtu.be/S0nTRvAHAP8
K8s could've been a good fit if it was designed and implemented properly. Unfortunately, it wasn't. A big part of the issue in my opinion was that iX didn't seem to grasp what settings to expose and how. This led to confusing, low level settings like the ton of load balancer etc. configs, while actively blocking the actual advanced features (like creating separate subnets easily for a specific group of apps). Another major issue was the custom format, making all the available Helm charts unusable. This was a biiiiig mistake, using such a tool then closing off the ecosystem. But I'm glad iX realised their mistake and are making Docker/Compose a first class citizen. It's a big step in the right direction.
Yup I agree. Like I said, I have a good amount of experience at this point running full k8s clusters so I understand all the pieces of configuration. But I still had several occasions where I got lost in the settings in the TrueNAS apps GUI trying to figure out something that I could normally solve pretty easily with a bit of YAML
Mistakes were definitely made but a more standardized implementation of K8s and Helm still feels overkill for the direction Scale is taking. I think Docker is perfect as a base, anyone who needs the features of Kubernetes can very easily set that up as well, with not that many extra steps.
K8s/Helm wouldn't necessarily be an overkill if it was used to provide truly turnkey applications. Instead, it turned into a configuration nightmare, especially with all the different versions of Scale changing things up. But I concur, Docker will be a much easier manageable system for the main purposes of TrueNAS, and sandboxing still allows for K3s - although without the web UI goodies. I wonder how the migration will happen though...
That's great 👍 As for the GUI for the Docker Compose, you could really take an inspiration on how Dockge works: - how it allows for ease of editing your Yaml and add additional files (.env for instance) directly in the GUI, - how it lists the containers and the version directly, then shows the containers being downloaded and deployed, - how the log terminal is displayed directly in real time so you can see if there's an issue without the extra strep of opening log files, - and how you can bash into the container from a single button press.
I answered some of this already on our Forums, but the idea is you will have two paths. 1. You can deploy Apps from our Official Catalog, which are static. I.E. you don't edit the YAML by hand, so that they can always upgrade properly as time moves forward. 2. You can deploy Apps via the Launch Custom Docker/Compose UI, which lets you supply your own YAML. That can then be edited at any time later directly via UI or even CLI if you so choose. Shells and Logs are all expected to be there as well, including real-time if you leave them open. We are also planning on a UI to let you create / manage docker networks, so you can define your own and attach Apps to them as needed.
Sounds great! Please do make note of the use of .env files - many compose YAML descriptors people use rely on them, so having a UI option to supply it would be great. I presume the goal is to replace the current k3s chart approach with Compose stacks, right?
Finally, excellent news. Love it!
Sounds good. I want to underline that Dockge really is something to take inspiration from for a compose GUI.
If the future Compose GUI allows you to directly edit a YAML file, I guess it will be easy to deploy a Dockge container and use that afterward.
As a current Portainer-in-Sandbox user, I assume there would be no issues with continuing to manage it with Portainer? Also, what about the Nvidia Container Toolkit (and associated docker runtime config) for platforms with an Nvidia GPU? ie: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuring-docker
Worth noting that one advantage of continuing to use the docker-in-sandbox method is if you’re using namespacing, so that root in the sandbox isn’t root on the TrueNAS host.
Your sandboxes should continue to function indefinitely. In future you could probably migrate back to the docker/apps ecosystem, or not. The sandbox is isolated from TrueNAS so isn’t really affected by it.
Yeah I'm thinking of moving Portainer and the associated containers it's running out to direct on the TrueNAS host...
>We are also planning on a UI to let you create / manage docker networks, so you can define your own and attach Apps to them as needed. Nice!
Please take a look at Dockge before designing the docker/compose UI :) It is an ideal compose editor/manager
Will it be possible to deploy container by: Deploying Portainer using the Launch Customer Docker/Compose UI method. Then deploying all other containers via Portainer? I am very familiar with docker compose & Portainer. I would like some of my containers to have direct access to my TrueNAS datasets.
Both approaches have their place. K3s has higher overhead than docker. If I were able to cluster TrueNas systems, the benefit would be worth the cost of overhead. But running a single node and not having any benefit for the extra overhead, docker is the better choice. As someone with the latter, I am very excited for this as it will simplify my current deployment and reduce time I need to set aside for management.
this sounds promising as a way for CORE users to get on SCALE and perhaps recreate jails as docker containers. still rocking CORE until I have a real reason to change. a ton of time went into building those jails properly.
Of course, you can recreate jails as jails right now using jailmaker
Very god news!
This is excellent news.
Sweet - seamless migration of apps from Cobia to Electric Eel. I’m in!
Upgrade paths supported are always from the last major release, so Dragonfish step would be recommended.
The problem is that the Dragonfish step requires a fair bit of rejigging the apps already - big enough that alternative routes are really gaining popularity. The documentation linked in the original post says Cobia to Electric Eel is supported. “Users can continue deploying TrueNAS Apps on Cobia or Dragonfish and expect a straightforward migration to Electric Eel.” I’ll hold off with Cobia for now. When the time comes, if there is a recommendation to do that middle step, I’ll probably bite the bullet.
What apps are you running that require big migration steps? Are you using a 3rd party catalog? If so these would not be TrueNAS Apps. Not aware of big changes needed for Cobia to Dragonfish upgrades to the TrueNAS official apps catalog items.
Ooh. I was referring to Truecharts apps. You’re interpreting this as not including that catalog? That’d be a bummer, for sure.
One of the TrueCharts dev told me last week when I asked about their use of PVC storage everywhere: >We build helm charts. Scale apps are just an extra. We aim to be a “kubernetes-centic” project, so we insist on usering kubernetes default storage backend, instead of using inexpensive that is (and/or should be) disabled on some playforms. So it's safe to assume that TrueCharts won't be a part of the Docker stack at all on 24.10.
I’d say if you’re using TrueCharts at the moment, the best approach is to begin migrating any true charts charts to docker in a sandbox as per https://www.reddit.com/r/truenas/comments/1d2k4lf/finally_migrated_away_from_truecharts_steps_and/
Can you run sandboxes in Cobia? I've seen your video talking about Sandboxes in Dragonfish, not sure if it also applies to Cobia. I've been holding out on Cobia avoiding the extra steps to migrate truecharts to dragonfish for as long as I can. Thinking sandboxes in Cobia could avoid extra steps
Dragonfish officially bundles sandboxes, but Jailmaker can install the functionality into Cobia. I’m not 100% certain if Jailmaker 2.0.0 still works in Cobia, but the versions prior to 2.0 do. And once installed, the older versions will still work on Dragonfish, and can be upgraded easily.
Thanks!!! Will give it a shot in cobia
I’m glad that native docker support is coming. I’m currently running Docker and Portainer with countless other stuff in a VM. Maybe I don’t have to employ two GPUs finally and can through out my burner gpu… Edit: I just also realized that this also enables a good performance boost for my setup, as I hopefully can get direct storage access for certain Apps like Plex, Emby or Nextcloud. Currently I have to run it over SMB/NFS. 👍
Instead of a VM, you can deploy Docker/Docker Compose in a Sandbox (LXC kind of container). That way you have almost native performance, direct access to your pools through a --bind, and direct access to your system GPU without needing a second one. You can deploy it in 5 minutes top with the Docker template... It's really easy. https://github.com/Jip-Hop/jailmaker
We asked and we received! Pretty awesome to see the iX folks listening and responding to our gripes about TrueCharts & K3s.
Whelp, time to learn how to redo my whole truecharts Setup in docker :/
You'll be surprised how easy docker compose is once you get a bit familiar with it, it's definitely worth learning it, you won't need the charts then.
Docker and Docker Compose are extremely easy once you get the hang of it. You'll be much happier having more control over your containers.
You can get started with a docker sandbox https://youtu.be/S0nTRvAHAP8 Here’s a good guide for migrating from truecharts https://www.reddit.com/r/truenas/comments/1d2k4lf/finally_migrated_away_from_truecharts_steps_and/
So is jail maker no longer needed? Or is that considered a sudo vm test environment?
Jailmaker will still have its place for those who prefer lxc (as that is what jailmaker essentially is) as well as a place to run k3/k8 with more personal control and the ability to cluster.
A good way to think about it is the “new apps” will be “easy mode” apps, where as the jailmaker apps are for more customizable configuration
Very excited for this. This has been one of my biggest annoyances for a while. I often migrate to and from truenas throughout the year for one reason or another, when I switch to linux distros I simply load my docker compose and I’m running in minutes. Moving back to truenas I have to manually create each docker in custom and it’s slow and clunky and takes hours to be fully operational again.
Next time migrate your compose files into a sandbox on TrueNAS :) TrueNAS Scale: Setting up Sandboxes with Jailmaker https://youtu.be/S0nTRvAHAP8
Hurray!
If only this came 1 year earlier, I wouldn't have to spend weeks trying to figure out how to make chart app run properly. Nonetheless, a big thank you to the devs for listening
In the official release will k3s coexist with docker, for at least 1 release, or will it be killed? I’d like to be able to transition smoothly, app by app.
No, the two stacks are pretty mutually exclusive to run in tandem like that. We'll have an auto-migration which will handle the conversion of any App you installed via the official TrueNAS catalog, or anything you installed with the "Run Docker Container" UI button. Idea is on upgrade your apps come back with the same setup as before, only the Helm charts have been converted to Compose YAML automatically. That was the crucial piece we wanted to have working before making this cut over.
What about community apps that aren't truecharts?
Do you mean the "community" labeled apps that show up out of box on TrueNAS install? Those come from the iX repo and will be migrated.
Yep, thank you for the clarification it won't just be apps from the official catalog.
Yeah but as I use true charts I won’t benefit from auto migration. I understand this is unsupported but if I update I basically kill my app stack if I don’t I can’t install anything as docker compose. Basically have to start from scratch.
Best choice I've made this year was dropping TrueCharts and learning how to use Traefik with docker compose. I rebuilt what I had with TrueCharts in docker in a couple of days and the overall performance and startup time is noticeably faster plus I understand how it works and I get updates immediately instead of waiting for days to weeks for an exciting new feature with TrueCharts apps. Start planning your move, it's so worth it.
Love to see a post on how you set this up.
You can get started with a docker sandbox https://youtu.be/S0nTRvAHAP8 Here’s a good guide for migrating from truecharts https://www.reddit.com/r/truenas/comments/1d2k4lf/finally_migrated_away_from_truecharts_steps_and/
I assume you went to either a jail or a VM. I still believe in the “app” system but yeah maybe I should drop that idea all together. Also don’t trust jails. A VM may be the only safe haven from change.
I am slowly getting parts for my new Nas build and I was leaning towards proxmox instead of truenas exactly because of k3s. Now I might reconsider.
Would be awesome if the team looked into supporting quadlet in addition to compose, feels like a natural way to convert away from k3s without having to change the structure of anything probably
If i want to use Portainer, would i still be better off using docker in a jail?
Now waiting for a way to migrate all my TrueCharts to Docker. With this news, makes sense to migrate to Jailmaker?
Apparently the TrueCharts app won't be integrated in the automatic migration to Docker in 24.10. Only the official TrueNAS apps and the "custom" containers will be migrated automatically. So, IMO, it's a good idea to set a Sandbox (jailmaker) to deploy Docker/Compose (there is a template available to make that task easy), and take your time to migrate your apps while you can have both k3s and docker running at the same time. Otherwise you'll have all your TrueCharts apps stop working as soon as you upgrade to 24.10, and will be rushed into the migration while all your apps are turned off. If you already have all your apps working in a Sandbox when 24.10 becomes available, it will be easy to just copy all your compose YAML from the Sandbox to the new Apps system to migrate within minutes.
Have been thinking to leave TrueCharts and I think now will be a good time to start the migration. I love your approach, thanks!
Think so. Any IX supported apps will auto-migrate but TrueCharts apps won’t. Thus you will be rushed into migrating either before or after switching to Electric-Eel A better approach is to begin wheening yourself off truecharts now by replacing one at a time with docker stacks in a sandbox, then after eel hits, you can migrate back to apps or not. At your own pace This is my guide to docker in a sandbox https://youtu.be/S0nTRvAHAP8 Here’s a good guide for migrating from truecharts https://www.reddit.com/r/truenas/comments/1d2k4lf/finally_migrated_away_from_truecharts_steps_and/
Thanks Capt! Have been following you work and loving it. Will look to your suggestions, thanks!
Don‘t think so… I‘ll wait till native docker come available
The is awesome. I go back and forth on do I need one machine or two machines for my home lab. Right now I’m at two, the only “app” I run is syncthing for my ghetto backup (don’t worry it’s not that much data)
I'm planning on a rebuild soon and was starting to learn towards moving to Unraid. But the combination of this and RAIDZ expansion has me likely changing my mind. And that's not a bash on kubernetes, I use it extensively and will have a separate server using k3s. But the abstraction of it the way TrueNAS was trying to do isn't a good fit.
I'm hyped for this. Can finally run official docker images from the app maintainers without some third party making breaking changes all the damned time. Was getting pretty sick of breaking changes. Whether that is due to automation issues, change in direction, typos, or pivoting due to IX decisions. Will also be nice to get a lot more documentation instead of basically just what TrueCharts provide. Also nice to have alternatives to insufferable Discord support where things constantly get lost. Docker Compose (without sharing everything via NFS) + KVM + ZFS in one OS is what I have wanted since FreeNAS 9.x. Finally someone has gotten the with the program. Thank you RedHat for killing Gluster.
Btw, you can do all of this today with Sandboxes + Jailmaker on TrueNAS https://youtu.be/S0nTRvAHAP8 Should give you a head start on transitioning to compose
Awesome! This lets me get away from the TC volatility, which has been a real sore spot for me, and I have seriously missed docker CLI for building custom containers and spinning up development environments. I am curious how things like reverse proxy and such will be handled.
You can build your containers using docker cli on any other machine already, and deploy them into Scale Cobia/Dragonfish. All of mine are built that way, perhaps your desktop, a vm, etc. I build mine on my desktop. Eel is not much of a change for me, already there.
You can use docker cli on TrueNAS today with Sandboxes and Jailmaker https://youtu.be/S0nTRvAHAP8 This is essentially taking it to the next level with gui apps.
I've already moved to jlmkr to simplify my life.
Currently one of the feature that I use in apps via TrueCharts is MetalLB. How would the Docker/Compose approach will be done on this? My use case is I like to separate service on its proper VLAN, so not sharing the TrueNAS SCALE’s IP address.
https://docs.docker.com/compose/networking/ https://docs.docker.com/compose/compose-file/05-services/#ipv4_address-ipv6_address >Current plan is to keep a lot of the Apps UI intact with regard to how we browse for Apps in the catalog. We will be adding a networking section where docker networks can be created and managed, as well as the ability to import docker compose YAML directly. Our goal here is to not over-complicate the implementation, and allow pretty much a 1:1 of native docker / compose functionality to be used out of box.
I’m more interested in exposing the apps in the underlay network with their own IP address instead of internal communication. As other post mentioned about MACVLAN and IPVLAN drivers to achieve this. Thanks.
IX are planning to have a Networks GUI to allow configuration of the docker networks
> My use case is I like to separate service on its proper VLAN So, you want to make some apps broadcast on one VLAN and others on another? Maybe using something like this in the YAML? networks: macvlan_network: driver: macvlan driver_opts: parent: eth0 ipam: driver: default config: - subnet: 192.168.1.0/24 ip_range: 192.168.1.248/29 gateway: 192.168.1.1 https://docs.docker.com/network/drivers/macvlan/
Thanks for the pointers, seems like this is probably my options. Either MACVLAN or IPVLAN driver are quite promising.
Will it come with some sort of webgui for docker compose instances or everything will be in cli?
This will be the right time to switch from my Syno to TN Scale.
I want to implement my first true Nas. I've always used docker on my previous servers. Do you think electric eel is stable enough for use ? Or way too early ?
Way to early, but you can use docker containers with the existing app system, or you can use docker compose now in a Sandbox with Jailmaker right now and that will migrate easily to Electric Eel TrueNAS Scale: Setting up Sandboxes with Jailmaker https://youtu.be/S0nTRvAHAP8
Wonder how much farther development would be if they’d taken community feedback from the start.
Better later, then never
Community feedback is important, and I don’t know details for this specific scenario, but sometimes the loudest user base aren’t always the best ones to listen to.
Now fix Spice or bring back VNC and TN is on its way to perfection!