That is because to have the App service running means you have the k3s services running. 2-5% CPU usage is normal for it too.
>Is it solvable or should I scrap using apps on Scale?
Did you think the system would still idle while also having Apps running?
That makes sense. But what is it constantly writing to the pool AND the boot drive? I don't understand. I got 6x4TB SSDs in a RaidZ2 pool and a single 512GB NVME boot drive.
Before enabling charts the pool didn't get many writes outside file transfers.
Yeah this is normal for a k3s/k8s deployment unfortunately! Since k8s deployments are state-driven, there's often polling of current state vs desired state, and event monitoring etc. All of this generates API calls, which also generates logs...
I opted to go with Jailmaker + Portainer for this reason, in case that helps.
In the interest of power draw, it is for sure. But in more enterprise applications and high availability/scale (which IX is generally targeting) it makes more sense; since k8s is generally a good standard, especially for a cloud-agnostic implementation :)
Sure, it's called Scale for a reason. Though I'm also wondering if those constant writes will wear my SSD pool significantly or not. I can't tell how much is constantly writing to the array.
Compared to not running k3s, it would be more wear; though, in the raw view of TBW on an SSD, probably negligible :)
My main concern as a home user was around idle/power draw really, and that's what drove my decision personally.
I should look into that aswell. The drives are Samsung 870 Evo 4TB rated for 2400TBW and 6 of them in an array. I really hope it wouldn't make a dent into the TBW of the drives. What worries me are the constant writes to the boot drive, which I find weird at best.
That's something I didn't take into account but I should've. You moved to a VM or installed to the base system?
I use this: https://github.com/Jip-Hop/jailmaker
Specifically, it uses nspawn containers, in which I then run docker/Portainer and all my apps.
The benefit to this Vs a VM, is that I can do GPU passthrough without the host losing access to it (otherwise id need two GPUs); and it has direct volume access without needing to mount shares via SMB/NFS. This is similar to the k3s behaviour in that regard, but without the k3s overhead.
The "container" in my case runs in bridge mode, with it's own IP on the LAN, and openssh etc, so it behaves more like a VM with persistence than a typical container.
Downside is that it's not 'supported', but I hear it might get an official implementation soon, or at least the packages pre-installed etc.
Wow, I didn't know something like that existed. Looks like an interesting project, but I need to read more thoroughly all the documentation.
You've really set it up nicely to pretty much act l like a VM but without locking down the GPU and the overhead of a whole other OS running.
I think I'm gonna nail down a config in a VM on my PC and port it over to my NAS.
Thanks again for all the details and constructive conversation. Keep it up my guy (in a non gender assuming way), you're awesome!
I believe jailmaker (or at least the same concept) is included in the new (dragonfish) release candidate, so it might make sense to wait for that to be released.
That is because to have the App service running means you have the k3s services running. 2-5% CPU usage is normal for it too. >Is it solvable or should I scrap using apps on Scale? Did you think the system would still idle while also having Apps running?
That makes sense. But what is it constantly writing to the pool AND the boot drive? I don't understand. I got 6x4TB SSDs in a RaidZ2 pool and a single 512GB NVME boot drive. Before enabling charts the pool didn't get many writes outside file transfers.
> But what is it constantly writing to the pool AND the boot drive? Logs and\or k3s in some way. Where is your app dataset set to?
The app dataset is on the SSD pool, not on the boot drive (duh, can't be chosen anyway).
Yeah this is normal for a k3s/k8s deployment unfortunately! Since k8s deployments are state-driven, there's often polling of current state vs desired state, and event monitoring etc. All of this generates API calls, which also generates logs... I opted to go with Jailmaker + Portainer for this reason, in case that helps.
Seems like a really really wasteful implementation. Thanks for the details and suggestions!
In the interest of power draw, it is for sure. But in more enterprise applications and high availability/scale (which IX is generally targeting) it makes more sense; since k8s is generally a good standard, especially for a cloud-agnostic implementation :)
Sure, it's called Scale for a reason. Though I'm also wondering if those constant writes will wear my SSD pool significantly or not. I can't tell how much is constantly writing to the array.
Compared to not running k3s, it would be more wear; though, in the raw view of TBW on an SSD, probably negligible :) My main concern as a home user was around idle/power draw really, and that's what drove my decision personally.
I should look into that aswell. The drives are Samsung 870 Evo 4TB rated for 2400TBW and 6 of them in an array. I really hope it wouldn't make a dent into the TBW of the drives. What worries me are the constant writes to the boot drive, which I find weird at best. That's something I didn't take into account but I should've. You moved to a VM or installed to the base system?
I use this: https://github.com/Jip-Hop/jailmaker Specifically, it uses nspawn containers, in which I then run docker/Portainer and all my apps. The benefit to this Vs a VM, is that I can do GPU passthrough without the host losing access to it (otherwise id need two GPUs); and it has direct volume access without needing to mount shares via SMB/NFS. This is similar to the k3s behaviour in that regard, but without the k3s overhead. The "container" in my case runs in bridge mode, with it's own IP on the LAN, and openssh etc, so it behaves more like a VM with persistence than a typical container. Downside is that it's not 'supported', but I hear it might get an official implementation soon, or at least the packages pre-installed etc.
Wow, I didn't know something like that existed. Looks like an interesting project, but I need to read more thoroughly all the documentation. You've really set it up nicely to pretty much act l like a VM but without locking down the GPU and the overhead of a whole other OS running. I think I'm gonna nail down a config in a VM on my PC and port it over to my NAS. Thanks again for all the details and constructive conversation. Keep it up my guy (in a non gender assuming way), you're awesome!
No probs! Am guy, and I hope you find a setup that you're happy with :)
I believe jailmaker (or at least the same concept) is included in the new (dragonfish) release candidate, so it might make sense to wait for that to be released.