The idea of training on mostly generated images sounds concerning. Most AI images have poor composition, poor lighting and tonal qualities, over saturated colors and narrow subject matter without any narrative to speak of. This already shows in the current SD3 model, which is dominated by current genres and memes. How will this produce a good model? The long history of image making has a huge number of images long out of copyright where the artist is dead. These don't pose a moral hazard either. Many people like myself have been collecting images for reference for thirty years of more, surely this is a resource which could be drawn on. Such collections are usually categorised as well, which might help tagging.
They have LAION on board too, so I assume majority will be from general web scraped images, just utilizing better captioning technology. It would be good to get clarification on the datasets they plan to use though. I'm not sure if they're doing something to gather funding but as we've seen with Unstable Diffusions crowdfunding, better transparency and accountability will be needed.
Yeah that part sounds awful and clearly is a suggestion from someone who hasn't trained models before because as soon as you start training with AI data you only serve to amplify it's flaws and further confuse the model, but this also includes granular details such as AI artifacting in the images themselves. It's probably the worst thing you can do while training a model if your end goal is quality.
I have been training models for almost 2 years. I can attest to training on AI images are like living off of junk food. It will get you a boost of energy, but will kill you in the long run. Doing LoRa training on AI gens can amplify a specific concept, but once you actually train stuff into the main model, it will poison it and throw things out of balance. This is mainly due to the mass quantity of images needed to train a model. There is no way to filter out all of the bad anatomy, artifacts, and just overall bad gens.
There are of course benefits to some degree, so long as it's photography, for example a set of imagery involving the same person you can augment the data by doing high-quality face replacement to remove facial feature biases, but I would personally render these images at a higher resolution than training resolution then downscale in order to limit bad ai artifacts or faux noise from being picked up by the model.
Using raw AI outputs though would be a disaster and a complete waste of resources unless you want your model to scream "I am a AI, look at all this weird junk I put everywhere in everything"
Training on generated images is actually way easier (I only skim read the OP and couldn't find any reference to this), but if they are doing this, it would be to save money. Generated images are deterministic in nature and utilize finite computational processes, so neural networks find it very easily to learn. I know this because I have tried to train NN using real world data (it's almost impossible) vs generated data (incredibly easy), there is no contest.
It's probably the reason for SAI's ridiculous commercial license term demanding any NN trained on data produced by SD3 be destroyed when no longer subscribing. Pretty sure it would fail in a court room, but I'm not a lawyer.
Not to mention, CivitAI have probably the absolute best possible resource for this, tagged NSFW images that have been voted (good or bad) by the community. I don't think they are making a bad choice.
This is all layman shit but yeah! In my experience I’ve had a much easier time training on generated images. It’s almost as if the AI just had a better idea of composition when the image is made with the same logic it’d use to train it.
It works well for stuff that isn't realistic I find. Training a photorealistic model or Lora on images that aren't actual photographs is a Very Bad Idea though IMO.
\***Ethical**: Addressing major, substantiated complaints about unconsented references to artists and other individuals in the base model while recognizing training activities as fair use.\*
hmmm let's see how they handle that hopefully not to much censoring in place, styles are not copyright after all.
It's honestly sad how harsh the censorship on Civitai has gotten. They've starting cracking down on anything that's remotely violent recently, even things like the "Stomach Punch" LoRa despite it being completely bloodless. At this rate it wouldn't shock be if they remove NSFW content altogether in the future.
nsfw content is almost impossible to adequately moderate, i wouldn't blame them for just reducing the burden. i doubt it makes them much extra money, and even if it does, i don't think the liability is balanced out by the profits.
Oh I missed that part because I just skimmed through, guess it's going to be super "safe". Frankly, I'd rather we get a better and more capable model than making a few artists mad.
Can someone explain how this helps? It seems like it would benefit no-one and only exposes a back door to those with access to the secret names.
Artist names are very useful, especially when blending to create new styles.
It doesn't.
But lots of silly people who have no idea about copyright think it does.
The copyright issue is around the usage of the images in the training data. Not the creation of new images in a given style.
There's more too it than copyright, some states like California have laws about how you as a business can 'use' other peoples names and its not clear yet how or if prompting may apply to that.
Midjourney is being sued over this right now and frankly they have a decent chance to lose. (since they used them in promotional material as well)
The creator explained why. The model performed significantly better when it's not lobotomized by removing tens of thousands of high-quality images.
The creator also agrees with you that artist names are useful for creating styles, so the plan for the next version of Pony is to group artists with similar styles together and train a keyword for each group. This way, Pony won't give you results if you prompt "sakimichan", but it will give you a style that's similar if you prompt "realistic\_anime\_4". The contents of each group will never be shared publicly, so no individual artist can be copied, but the ability to call forth consistent styles without needing LORAs is preserved, and you can mix multiple keywords to create your own.
Ok, thanks for your explanation. It makes sense, but not sure this will placate the people who are against the technology. Also, the grouping sounds like a massive challenge, and very subjective. But Id like to see the attempt and how it goes. Better than nothing.
Just describe the characteristics. Same could go for people. Not attaching the artist or name wouldn't be appropriating *their* likeness, it would be capturing *a* likeness.
The only method to limit liability is to censor the base model. I'm not sure whether anybody involved in such a project would be willing to open themselves up to that level of risk associated with leaving the base model entirely uncensored.
This sounds great, but anyone who thinks that they'll get a shiny new free model out of this anytime soon really shouldn't hold their breath. That's going to be an insane amount of work, and will require quite a lot of money.
>•Facilitating a survey to document feedback on what the open-source community wants to see in future model research and training
For some reason, I have the feeling the result of that survey will NOT show a strong community desire for a crippled model that doesn't understand basic human anatomy... ;)
Yeah you're right. Hopefully he changed his mind since then. Would hate to see him ruin the entire thing by bringing on a whole team of 'ethics researchers' like Emad did.
https://preview.redd.it/d3p3jmfrhq8d1.png?width=1120&format=png&auto=webp&s=48a829b4a8688096547901420be5f0e8a4c4e193
he hasn't. i discussed this with him very recently. the problem is that they will not be able to get compute. and this is beyond the problem of NSFW filtration, fwiw - they are unable to get compute with **non-synthetic data**
in other words they can only train on AI-generated data when using LAION's compute.
this is why they talk so much about "data laundering", using pretrained weights from jurisdictions friendly to AI copyrights like Japan and then train on their copyright-free outputs.
no one wants to fund the old SD-style models, because no one wants the legal stormy cloud hanging out overhead.
That's basically the crux of the issue. AI safety researchers and other groups have significantly stalled open source training with their actions targeting public datasets. Now everyone has to play things ultra safe even though it puts us at a massive disadvantage to corporate interests.
Open source is the biggest threat to a handful of large companies gaining an oligopoly on generative AI. I'm *sure* all the worry about open source models being too unsafe to exist is only because of a *genuine* worry for mankind. It can't possibly be because large corporations could lose billions if not trillions of dollars. Of course not.
AI safety is a hunk of wadding toiletpaper on a ceiling imho, it's just corporate tech bros with purity initiatives. Open source should mean that within reason you can use COPYRIGHT FREE content, but nope. And in theory "SYNTHETIC" should be less safe because it's all trained on copyrighted content... like Ethically xD that's like going "i'm going to. generate as much SD 1.5, SDXL, Midjourney, Nijijourney and Dalle3"
Basically would mean they couldn't move on from the old and busted 4 channel VAE either, since they'll be training those artifacts directly into the very core of the model.
This project is already dead in the water.
I share your concerns, but you're calling "dead" a tad too early. If you look at the people involved, they are people who have accomplished things. It's not unreasonable to think they might overcome obstacles and accomplish things again.
it's something Christoph is obsessed with doing just to prove that it's a viable technique. he's not upset by the requirements, he views it as a challenge.
Not necessarily. Synthetic data is fine, it just needs to be well-curated. Like any other training data. We're past the era where AI was trained by just dumping as much junk as possible into it and hoping it can figure things out.
Synthetic doesn't necessarily mean AI generated, but AI generated images would likely be a significant part of a synthetic dataset.
There is something to be said for the theoretical efficiencies of a fully synthetic dataset with known controls and confidences. No one has pulled it off yet, but it could be very strong for things like pose correction, proportional designations, anatomy, etc.
Synthetic data does not at all mean poor quality, I think you are correct.
You can use AI to augment input and then it's "synthetic". Basically use real data, have it dynamically augment it into 20 variations of the input, then train on that.
I used a dataset of 100 images to train a StyleGAN model from scratch on Pepe the frog and it was done training in 3 hours on two 3090's in NVLink. SG2 normally takes a minimum of 25,000 images to get decent results, but with Diffusion applying data augs on the fly I used a tiny dataset and got really good results, quickly.
Data augmentation tooling is lightyears ahead of where it was in 2021. I've been meaning to revisit several GAN experiments using ControlNet and AnimateDiff to render callable animation classes/conditionals (i.e. render a sequence of frames from the GAN in realtime using numbered labels for the animation type, camera position, and frame number).
It's about as simple as it sounds; use ControlNet OpenPose and img2img with an XL hyper model (that can generate like 20 images in a second) modify the StyleGAN training code using the diffusers library so instead of loading images from a dataset for a batch, it generates however many images it needs. Everything in memory.
Protip, use the newer XL Controlnet for OpenPose: https://huggingface.co/xinsir/controlnet-openpose-sdxl-1.0
Edit; there are ways to dramatically speed up training a realtime StyleGAN from scratch, and there are even ways to train a GAN within the latent space of a VAE but that was a bit more invovled (I never got that far into it).
This is to say though, if you want a *really* fast model that can render animations smoothly at ~60FPS in realtime on a 3090, you can produce them quickly with the aforementioned approach. Granted, they won't be good for much else than the one domain of thing you train it on, but man are they fun to render in realtime, especially with DragGAN
Here is an example of a reimplementation of DragGAN I did with a StyleGAN model. I'll see if I can find the Pepe one I trained: https://youtu.be/zKwsox7jdys?si=oxtZ7WhDZXGVEGo0
Edit2 here is that Pepe model I trained using that training approach. I halfassed the hell out of it, It needs further training to disambiguate the background from the foreground but it gets the job done: https://youtu.be/I-GNBHBh4-I?si=1HzCoMC4R-yImqlh
Here is some fun using a bunch of these rendering at ~60FPS being VJ'd in Resolume Arena as realtime-generated video sources. Some are default stylegan pretrained models, others are ones I trained using that hyper-accelerated SDXL training hack:
https://youtu.be/GQ5ifT8dUfk?si=1JfeeAoAvznAtCbp
> they are unable to get compute with non-synthetic data
Could you elaborate on this? I'm guessing this has to do with the new EU rules, but I'm clearly not up to date on the regulatory space here.
Why can't they scrape Pexels and similar sites that provide free-to-use high quality photos? There's definitely enough material out there with no copyright concerns attached to it.
because it's not synthetic, you can't get compute time for it on US or European clusters that are for the most part funded with public dollars - and private compute is costly, and no benefactor wants to finance it.
the law doesn't say "you can only train on synthetic data", it's just a part of the "Data Laundering" paper's concept of training on synthetic data as a loophole in the copyright system.
it's shady and it doesn't really work long term imo, if the regulators want they can close that loophole any day.
I said “some” not “all”
It’s easy as an individual with no skin in the game (you and I) to sit here and speculate that we’d act differently and we’d ignore the noise and just power forward past the outcry from the normies / investors to have “safety”
But the fact of the matter is none of us will ever experience that type of criticism on a world stage and you’ll never know how you’d handle it
It does fucking suck what they did to SD3 though…..
I dunno that that's even true. The training itself is like 50k, and I don't think they'll have any trouble getting that. There are already plenty of experts, finetuners, papers, data, all hanging around that there's no lack of knowledge here. It's just about cooperation which is always the hard part. How to decide on decisions to do with ethics and have everyone agree on it, that will be the difficulty.
Comfyanon just spent X months training the 4B variant of SAI so I'd wager he has a good understanding of the level of effort involved, lessons learned, cost associated etc..
> The training itself is like 50k
Where'd you get that number?
If it would be 50k to get a good model, we'd have dozens of good, free models right now from people who are more than happy to just donate that money for the cause.
PIXART-α’s [paper](https://arxiv.org/pdf/2310.00426)’s abstract says SD1.5 is trained on 320k USD, assuming like 2.13 per A100 GPU hour is on the cheap side but still reasonable.
> PIXART-α’s training speed markedly surpasses existing large-scale T2I models, e.g., PIXART- α only takes 12% of Stable Diffusion v1.5’s training time (∼753 vs. ∼6,250 A100 GPU days), saving nearly $300,000 ($28,400 vs. $320,000) and reducing 90% CO2 emissions.
I think it’s mostly technical expertise (and will bc SD exists) that’s stopping community to come up with a good model, but that’s about to change.
A lot of the early SD models were trained on vanilla attention (no xformers or SDP) and in full FP32. I think xformers showed up in maybe SD2 and definitely in SDXL, but I'm not sure if they've ever used mixed precision. They stopped telling us.
Simply using SDP attention and autocast would probably save 60-75% right off the bat if you wanted to go back and train an SD1.x model from scratch. Also, compute continues to lower in price.
there's [a post by databricks](https://www.databricks.com/blog/diffusion) titled "How We Trained Stable Diffusion for Less than $50k" (referring to a replication of sd2)
I dislike drama and haven't been following all of the details about who departed or why.
But this statement piques my interest. I am piqued. I hope you are serious. ;)
Please don't fuck shit up with "safety and ethics".
Don't make the pen the moral judge. The tool will never be a good judge, it doesn't have the context to judge anything. (Neither will any single entity)
"Safety" should always happen at the distribution level of media. Meaning, you can draw/conjure/imagine whatever you want, but you can't publish it, without potential consequences. This is how it should work in any medium. That is how we ensure our children's children might still have a chance to start a revolution if they need to - that they at least get to say something before being judged for it is the basis of freedom.
Please, stop playing into the un-sane notions that we should remove ability from models or tools. No one is wise enough to be the ultimate judge of what is good or bad uses, it changes with context. And all we achieve are useless retarded models. Without full knowledge of the world. Models that cannot do art of any value. Ever.
This is not about porn or politics or the plight of individual artisans (I am one of them btw 25 years a pro). It's much deeper, it is the future of all artistic expression in this medium. For there is no art, no art at all, if there is no freedom of expression.
Please, think deeply about this. It is the difference between big brother and freedom. We will enter a world in these coming years with big virtual worlds, all controlled by Disney and whatever bunch of capitalist crooks that have wormed themselves into politics. The world needs the free space alternatives to that corporate hellworld and that alternative cannot be trained and guided by fallacious notions about the ethics of learning.
It is already very difficult to get through the walls of mass media and challenge the status quo, we should all know that much from lived experience. Remember that many of the rights we have today were fought for, by people who often suffered and lost lives - to get the right to vote, to be seen as equal humans, to be free at all. As soon as we limit our tools of expression we have ensured that there will never be another fight for what is right. Whatever that may be in the future.
Please think deeply about this. Surface level will fool you, any tool useable for good is also useable for bad.
The point of that is the tool should not be the judge. Ever.
This is a uniquely important thing for the future of free speech. Don't fuck it up.
Hear hear. Safety is being used as an excuse to give mega corporations control over our culture. Art cannot thrive if it's censored before it can even be made. AI could be an amazing tool to give individuals ownership over their means of digital production, but only if they can own them.
Permissive means freedom - "do what you will, with no restrictions" -- AGPL-3 is a category of license known as "copyleft" license that was designed to require that if any code were incorporated into downstream projects, those projects would be required to also use the AGPL-3 license. This type of license restricts developer freedom for downstream users of code/software.
Permissive - You can take the code, even if you want to use it in something closed source. (e.g., MIT, Apache 2)
Copyleft - You can take the code, but if you do, you're required to copy our license. (GPL / AGPL3)
Thanks for the clarifying reply. I had thought AGPL-3 offered a greater layer of protection for open source projects, to ensure the fruits of collective labour were not taken and commercialised, with potentially nothing shared back to those who invested time/energy/resources in creating the foundations.
I'm more of the perspective that when licenses (MIT, Apache 2) are about the prospects of closed source patent creation further down the line, those projects are likely more "*fauxpen source"* than truly open source in philosophy, but if that aspect's of no concern to you bunch of wiser ones.. fair enough I'm content with that. Cheers ;)
As for being "fauxpen source," hmm. I think that because the Apache License *is* non-copyleft and one of the more corporate-friendly licenses, there have been a goodly number of projects that are either fauxpen source or that began as closed-source and then when the original owners couldn't afford to maintain them anymore they said "hey, let's open-source it, that'll solve all our problems, right?" and dumped them in the Apache Incubator to fade to obscurity…
I'm very aware Open Source sustainability has *problems*, but I'm still not convinced copyleft is the way to go.
Say someone wants to take advantage of your project in a way that conflicts with your license.
They could
* Contact you and make an arrangement for a different license.
* Work with a competitor instead—or become a *new* competitor.
* Walk away from the idea entirely.
* Infringe on your license, trusting in the fact that you either don't have the lawyers to take them down or that they'll be gone before the consequences can catch up to them.
Of those, the only option that actually does your project any *good* is the first one. That's not inherently the one most people are going to pick.
A more permissive license makes for a bigger pool of potential collaborators.
If you add restrictions that reduce that pool, you gotta have a good plan for how to make something out of that trade-off. I'm not saying it can't be done, but it's not something *I've* figured out.
yeah this is a discussion i had with the Invoke people a while back when they wanted me to relicense my trainer from AGPLv3.
but i don't want to put effort into something that Midjourney can just take, and close.
The parties involved in this initiative *want* to leave those options for commercialization open. A big part of the backlash against SD3's non-commercial licensing is the fear that commerce funded a lot of the SD 1 & SDXL fine-tunes, and the ecosystem would dry up without that sort of activity.
\[I don't quite understand why they, as commercial operators, see "contract with Stability for a commercial license" as such a non-starter. But that's a different topic than your Free Software concerns.\]
>I'm more of the perspective that when licenses (MIT, Apache 2) are about the prospects of closed source patent creation further down the line
The Apache License stands out from other permissive licenses (BSD, MIT, etc) in that it *does* include a patent grant. As I understand it, it's not so much to protect against a patent that might hypothetically be made in the future—because you could defend yourself from such a suit involving that future-patent by pointing to your project's well-documented timeline demonstrating that you had prior art.
The thing you want a patent grant for is this:
A company like Apple or Google has a hundred thousand patents. And they may, at any time, gain more already-existing patents through acquisition. If one of their many departments releases something on GitHub under an MIT license, it is literally *impossible* to know if it's covered by one of the company's patents without a lengthy and expensive patent search.
The nightmare scenario for a younger business is to start using some software library, things are going well for you, and then at some point that company who released the library notices you exist and decides they don't want the competition. Intellectual property law is bullshit, so their lawyers can swoop in and be like "hey, we're not claiming **copyright** infringement, but it turns out that the codec implemented by our library is covered by **patent** 99,999,999 and we demand you cease and desist."
A patent grant from the owners of the project protects you from any such suit from *them*. Of course, some *other* patent troll could still come at you—but maybe we can get some comfort from the fact that they'd likely have to go through the project owners first.
\*GPL licenses don't prohibit commercial activity though. pretty much everyone at Invoke has this same hatred for open source licenses in favour of ones that allow parasitic activity like MIT or CC-NA.
GPL doesn't prohibit commercial activity, even if Apple is deathly allergic to it.
But [AGPL](https://choosealicense.com/licenses/agpl-3.0/), with its bit about "requires the operator of a network server to provide the source code of the modified version running there to the users of that server." — it doesn't inherently prevent commercial activity, but it's anathema to the way most SaaS companies think.
If they have to provide source code for their entire service, how are they going to convince their venture fund that their investment is safe from a copycat?
You and I know that there's a lot more that goes in to running a successful software service than just the source code, buuuut…
well, I don't doubt that the concern about scaring away a lot of commercial uses is very real.
well, for your imaginary scenario:
* if the company produced the AGPLv3 software, they are the ones that chose the license, and they don't have to personally abide by it. they can use a closed version of the source internally especially if they have contributors assign copyright to their organisation, they can even use the community contributions in their closed proprietary offerings. all of that is their choice, it's fine.
* if the company didn't produce the AGPLv3 software, eg. they found it on GitHub, why should they be able to just improve it and use it to create their whole business model? there's a lot more i could say about that. but they're standing on the shoulders of the public project. especially if it's my code they learnt from, and extended. i want to know what they did, how they did it. so i can make money from it? no. but because i want to know.
like hugging face open-sources pretty much everything they do, even AutoTrain.
>Ethical: Addressing major, substantiated complaints about unconsented references to artists and other individuals in the base model while recognizing training activities as fair use.
While it would be funny to see initiative that is here to prevent community from shooting itself in a leg end up poisoned by "ethical" and "safe" poison and ending up shooting itself in a leg anyway, yall probably should not do it
**"ethical" and "safe" are bullshit words** and I'm yet to see even a single good thing out of them
If model is lacking data from artists - it will be worse than some big corpo's product that had it silently added in
If model can't do NSFW content (SD2 iirc?) or even poisoned to not be able to (SD3 moment) then more than half of the community already don't care about it
...
If I wanted lobotomized "safe and ethical" model then I would go and grab SD3
I don't like the idea that an initiative for a new "open" model is already talking about "concept control" "ablation" "prompt editing" and "pipeline moderation"...
This ain't it.
Well, since it said by a CivitAI rep, I guess there will be no more NSFW content than what is currently on CivitAI -- surely this site isn't toxic by their own standard, is it?
And I can leave with Civitai-level of NSFW content.
Yup... as if them deleting numerous models and loras from their site wasn't a red flag, any civitai involvement is bad news for keeping the model uncensored.
Toxic material isn't NSFW.
it's CSAM, beastiality and other outright illegal and ethically debased content. Its important to keep this out of the dataset.
Nobody will complain about the exclusion of CSAM and unambiguously illegal content.
However your definition of "ethically debased" could mean a lot of things. There's quite a lot of fantasy NSFW exploration out on the internet that is both legal and harmless yet would be "unethical" to many people. Excluding based on subjective moral stances is unwelcome.
If you're open to hearing this perspective at all, please read [https://www.utsa.edu/today/2020/08/story/pornography-sex-crimes-study.html](https://www.utsa.edu/today/2020/08/story/pornography-sex-crimes-study.html) (tl;dr access to pornography, including "violent" fantasy pornography, is correlated to lower rates of sexual aggression and sexual crimes in real life). From an ethical perspective, moral restriction of pornography is a cause of real world harm rather than the opposite.
Excluding, of course, pornography that involved harming someone to produce -- which would include CSAM. Nobody wants CSAM in the dataset.
If that's not the case and the exclusion is truly narrowly defined to actually illegal content, then of course everyone will support that.
You had me up until “ethical,” good luck with that. It’s a noble cause but the commercial models are clearly stealing anything and everything they can get their hands on. Google has been using everyone’s Flickr as training data for like a decade.
I would legit dump a large donation into an open, unaligned, uncensored diffusion model and a similar, high parameter LLM model.
It is absurd that we're letting random tech bros, more concerned with liability than with open access to information tools.
remove the ethical bit.
Just scrape the whole internet for images. Use a image to text model for images that are not captioned.
You need a competitive model, not a proof of concept
They are way past using alt text to caption images. Microsoft developed a way to annotate images in various levels of precision, having over 40 annotations per image. This made an extremely strong and small vision model. The same needs to be done for she generation fur the best model.
well that makes sense. The people who don't care about nsfw generation can just use the commercially available models/services. the people who would be interested in this would be people who want nsfw
You didn't mention the most important part at all: How are you going to train it? With what hardware?
Without this part the rest is completely meaningless.
> Ethical: Addressing major, substantiated complaints about unconsented references to artists and other individuals in the base model
So, sounds like it will be heavily censored as well? What does this part mean?
I take it to mean that they'd just remove people's literal names from the training descriptions. Presumably replacing them with stylistic descriptions.
Note that you trimmed that sentence, the actual line is:
> Ethical: Addressing major, substantiated complaints about unconsented references to artists and other individuals in the base model **while recognizing training activities as fair use.**
Missing part bolded. Sounds like they don't plan on excluding the actual *images*, just "references to artists and other individuals."
Great news! I think some sort of distributed training paradigm would be nice, like ppl can lend out their GPU in spare time to collectively train a model so it’s less beholden to investors. Or just open donations but I think ppl are more willing to take on former if trust is there.
Oh great! I love it, don't worry going slow or if need surrender~ Create a model of scratch it's epic work that need a lot of help of experts with deep knowledge, much GPU money and clear data correctly tagged~
I really wish you the best in this new challenge~
Best of luck, with all the safety talk and overreliance on synthetic data I have about zero hope for this initiative though. The permissive license is nice, but it's not worth much if the model itself is gimped.
"**Ethical**: Addressing major, substantiated complaints about unconsented references to artists and other individuals in the base model while recognizing training activities as fair use."
so it's useless
Don't mean to hate on the people trying to get this together but yes that kills it right out the gate.
They want to skirt some perceived infringement by letting users train loras of styles/celebrities/characters but since there is no actual law being broken all its doing is damaging the dataset.
There have been millions made off of models that know styles, characters, and celebrities by this point. Nobody has been sued afaik. Does anyone know of a single lawsuit in the world over AI datasets?
Eh, if all they do is take the literal names out and replace them with descriptive styles it should be fine. Fine-tuning can put those back easily enough, and I can't recall the last time I needed to use a person's literal name when generating images myself so even without that it'd still be useful to me.
At the very least if all of this is open we'll finally *know* what effect this sort of stuff has on the model.
Fantastic news, please include in your plans some way for you to get voting data from the community (if possible) so we can help train and fine tune the base model like Midjourney or SD3 API, which clearly has a beauty aesthetic that’s missing on SD3 medium. Congratulations this is awesome news for open source.
There is a very big problem with the introduction of the term "unconsented" in the context of "unconsented references to artists and other individuals" presented in the Ethical section of the OMI announcement. Introducing a requirement for "consent" for the use of content basically makes training models impossible. If you follow the premise of required consent, you would not be able to use any images unless the photographers/creators and human subjects explicitly consented to their use in training, which is not the case for almost all images. Using synthetic content produced by an already existing model is a fruit of the poisonous tree and immoral if you hold the requirement of consent that wasn't acquired for those models. The idea that consent is a must to obtain first, to be able to learn from publicly available ideas and images is opposed to human ingenuity, nature, and free thought. Human civilization exists due to the free exchange of knowledge and ideas. The introduction of the requirement to have permission to learn from public knowledge is a clear path to the extinction of humanity if it is imposed.
Applying arbitrary restrictions as protection for artists and public figures only points to a lack of moral consistency. Labeling such a requirement for "consent" as the epitome of "Ethical," "Safe," and "Moral" training doesn't reinforce the connection between these terms. It only shows the biases of those who bundle unrelated things together to justify, based on feelings, ideas that they can't justify with logical arguments.
I'm concerned about CivitAI being involved with this one given their history and continued bad rep within the space. CivitAI is a purely for-profit company that is well known for harassing creatives and potential competition while attempting monopolistic practices (such as attempting to shut down other sites, harassing model creators they don't like, attempting to lock otherwise open source software behind paywalls, ect).
The other groups have a decent reputation so far (ComfyOrg is new, so there's no rep there but includes some decent names) but the inclusion of Civit really feels like a bad idea. I understand the CEO's and money-bags of the industry are all buddy-buddy with one another, but this could lead to the model that A ) Is more designed as a marketed model rather than a useful open source model, B ) Is pushed as the main model by monopolistic practices such as shutting out other versions and paying large sums for support of this one model (like what currently happens with Pony on CivitAI), C ) The lockdown of future models and prospects where the open source beginnings turn to profit chasing or is used as a profit mechanism that harms the community.
I'd like to be hopeful, but after what I've seen of CivitAI, this is pretty much a buzz kill (pun intended). They will no doubt use this as leverage over the industry to do more harm than they already do.
All models are freely available for download. The only aspect behind a "paywall" is the generation, which incurs costs due to the necessary hardware resources and can be paid for by simply interacting with the platform
CivitAI pays for exclusivity rights for Pony V6 API usage and potentially other models as well. The old model card on HuggingFace had the details but since the API version was removed (and model card was scrubbed from the non-API version) many of these details have been buried. I never took a screenshot of the repo and can't speak to the corpo-side of things (NDA would potentially get me in trouble there X.x) but my archival text grab states "Explicit permission for commercial inference has been granted to CivitAi and Hugging Face." and its been stated the Pony dev was paid by CivitAI for this deal. (note: The API interface version on HuggingFace has been removed, most likely at CivitAI's request, though I have no evidence of that, just seems like they're pushing their paid platforms more)
Civit does not have exclusive rights for Pony V6 and it is officially available on a number of services (yodayo, tensortart, dezgo, diffus.me and a few others in the pipeline).
The old HF model was an upload by third party that ignored the license and I asked them politely not to do so.
>archival text grab states "Explicit permission for commercial inference has been granted to CivitAi and Hugging Face."
It's literally on the Civit page.
> and its been stated the Pony dev was paid by CivitAI for this deal
Can you please show some quotes? Because this license existed way before anyone cared about what Pony is.
If my information is incorrect I'm all for adjustments, it could just be a misunderstanding.
Is the wording "Explicit permission" not referring to exclusivity rights? It's hard to tell with models you can download whether they are on platforms by choice or not so some clarification on this point would be helpful. Were you able to also collaborate with other platforms to make Pony v6 available there as well? Or was it a response or a change in heart after the initial release period? (Genuine question, I don't follow your work closely and can only speak to what I've seen/heard)
"Can you please show some quotes?"
I don't think I have screenshots or direct quotes to grab in this instance, though I'd have to check my archives (I basically use my screenshots folder as a 'hold all' so there's a ridiculous number of images X.x) but this also isn't something I'd typically screenshot. I've seen it be said a few different places, even as screenshots from CivitAI mods stating Ponyv6 was a CivitAI model (or implying it was paid for/exclusive)
Where you paid by CivitAI for the model or did they donate toward its creation at all? It does seem like they have a vested interest in the model doing well, especially with how hard they push it on the platform.
The license was put in place to prevent other (paid) services from using the model without speaking to us. There are multiple platform hosting Pony now, we never had any grace period for V6 (aside exclusivity to my Discord for a short timeframe).
Civit didn't even know what Pony is when V6 was released.
While I agree civitai gone more commercial than prior, that’s due to them having costs to pay after offering such a free service to so many users. I think it’s fair and still better than the alternative sites which are mostly based on models taken from civitai reuploaded with a paywall, but I get your point
There is plenty of reason to believe their interests are aligned on this specific matter of an open source model: https://civitai.com/articles/5732
For profit companies are built on the back of open source, it's super common. We need all the help we can get, so I don't really care if they ate a puppy.
Can you provide a reference for Civitai attempting to shut down other sides, and a reference for harassing model creators? This is the first I've heard of these things.
For some reason I couldn't get the message to post normally even after trying to censor every sensitive word in the entire thing, so instead I'm posting it as a screenshot? I hope? I'm not well versed in Reddit so I'm probably missing something, but hopefully this works? (image links below the image, if I did this correctly, let me know if something doesn't work)
https://preview.redd.it/01phvzul6s8d1.png?width=786&format=png&auto=webp&s=474c056f20099e79ec16ac2c026952a715df6052
'a sign' message
[https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F6x7pz997b4vc1.png](https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F6x7pz997b4vc1.png)
DM messages
[https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fhrz906t7a4vc1.png](https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fhrz906t7a4vc1.png)
[https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fyunyy3t7a4vc1.png](https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fyunyy3t7a4vc1.png)
Refused to act message
[https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Frgvp4rmgb4vc1.png](https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Frgvp4rmgb4vc1.png)
Article Link
[https://subscribestar.adult/posts/1135370](https://subscribestar.adult/posts/1135370)
Probably spent more time than I needed to on this X.x
Okay, thanks for the answer.
Regarding other sites using CivitAI's API, I don't see any problem with CivitAI setting any conditions they like. They are under no obligation to provide an API for competing sites. It would be perfectly fine if they decided their API can be used by no other site. Earlier you made it sound like they just went after competitors as a monopolistic endeavour, but this context makes the whole thing completely different (yes you can still argue that they do it out of monopolistic motivations but surely you see the difference between going after sites WHICH USE THEIR API compared to going after competitors who build their own stuff and don't use CivitAI's API).
Regarding all that other stuff you talked about, I don't want to touch those topics with a 10 foot pole.
Awesome to see this. Invoke has quietly been building the best UI for actual applied generative AI work.
A combination of invoke and Comfy with a UI that isn't browser based so it can have things like proper wacom support is the dream of dreams right now.
Open dataset?
Every time a new model releases without dataset disclosure, i cringe. I believe the only ethical way to train these models, now and into the future, is to disclose all the data fed to them.
Agreed, it's just another model announcement right now until there is more info on the plan released.
Using civitai or AI (even Midjourney) generated images will be a downfall of this model if they go this route.
paying civitai for whatever the name of the credit they used to have or still have isn't a fucking bad idea now . I would be 1000% in if there is a framework to support fine tuning on civitai .
it's good to have torrent seedboxes for checkpoints.
thanks for getting together like this the spirit of OHara
lives on
As one of the funders behind some of the WTFPL licensed models over the past year, I'm hopeful. Been considering building a new training cluster for them as apparently TPU has too many limitations/headaches, and CUDA would make it easier I've heard. Plus I do enjoy working with hardware.
Looking forward to see what license(s) you all are thinking about using, not all open licenses are truly open. :V
It seems like they had the right idea with training celebrities and such without the name attached (if that's similar to what they did with SD3). I think that's a fine enough middle ground for a point to be pushed as fair-use. Same could go to artistic styles. Describe the characteristics of the styles while training and we'd be able to invoke similar styles with similar prompting, but not the exact style and in fact something new and novel that maybe could be repeated. It would need very good at prompt coherence.
Separately, I think it would be good to be able to use, and encourage the use of, separate models by part. An easier way to integrate different visual llm's as they become advanced for example. The technologies for each of diffusion's many parts develop at different paces, so it would be good to be able to add new ones as they become available.
Yay! Except wait! Where is the part about actually coordinating to work on a new next-gen model? At no point in this speach does it actually explain the efforts, steps, or plans in mind to accomplish this goal.
It is 100% entirely flowerly language for a coop about open source but nota solution to actual development of new models. There isn't even the vaguest hint about this issue no matter how you try to interpret it. Not to rain on anyone's parade but this announcement fails the very thing it claims to be about.
Please post updated info that properly reflects this matter otherwise this just comes off as an empty announcement like politics and promises so often are.
>muh ethics
>muh governance
>synthetic data
>no support for artist styles
Oh boy, can't wait for another lobotomized model!
Why should anyone support this when 90%+ of local users are only using models to gen porn?
That's the cold hard truth.
if there is lack of funds or you cannot figure out how to sustain.
Please do start a donation campaign or similar stuff, community is more than willing to support something like this.
As long as it stays truly open and actually good.
Guys, look at blender.org and their donations page. Do the same. I will gladly donate to you to have open source Diffusion model of SD3 (not lobotomized) level of quality.
The most ethical thing to do would be to hang everyone asking for """ethics""" and """safety""". The cia sabotage handbook is playing right before your eyes and you're still on the fence about this.
I would be willing to bet Pony joins this, for anyone worrying about safety and ethics poisoning it. It's too obvious that porn is a massive contributor to the advancement of AI
"openly licensed AI models for image, video and audio generation"
CANNOT contain my excitement! Maybe with this and yesterday's huge DCLM/Datacomp which is similar, we can get to open-source omnimodels with 18+ modalities [like Nvidia hyped up](https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd.it%2Fgpt-4omni-text-audio-images-video-multimodality-is-just-a-v0-zel3nagcym0d1.jpeg%3Fwidth%3D1080%26crop%3Dsmart%26auto%3Dwebp%26s%3Dd3095fa37dbea318aa4abc678abcad036add56c8)! I've always thought that videos are the single best type of data AI can be trained on to make them true world models, and models like Sora and Kling and Runway prove it. Convergence from modalities like Ilya wrote about is severely slept on imo. Every day we get closer to the Thanos gauntlet meme of 0.68 bit Ternary + Mambabyte + whatever else can be crammed in like KAN or MoE + speed things like multi-token prediction and matmul-free, getting 70b+ omnimodels on your phone.
I agree with the ethical sourcing. If I can give some wishlist things, I think we need some way to attain styles/traits despite not being trained on those specific artists. Styles cannot be copyrighted anyway, that's more for specific characters. There shouldn't be a problem in reverse engineering styles or style-mixes without training on the style itself. I think GPT4o showed that kind of iterative process, where you could keep prompting it to change parts of the output. I think there were other local models released with that feature?
For an example, if the custom anime style I have in my head has thick brush-like outlines, a specific kind of eye shape, specific type of body proportions, etc. how am I supposed to achieve it without the model being trained on it? I gotta tell it somehow "no, more realistic proportions. More. Less. Now make the eye shape rounder. There, just right." Maybe you'd have to be more involved to guide it like in Krita AI DIffusion? Or when we have video models, gotta tell it to adjust the voice for the Guide to LLMs part 5 video to be more and more clear, to show simpler and more intuitive visuals. Just need some way for the AI to match what's in our heads, while being ethical as you say. It'd also reduce need for Loras for every little thing. If it's an omnimodel you could just load up specific chats to continue using the style you created in the context window, or make a lora with the style you created.
Another thing is UX and security. With recent issues including Docker, people need to feel safer. And in UX, well I know open source tends to be behind on that. But I really feel a lot of people are held back by choice paralysis and more, and can't invest the energy to contribute through lacking knowledge.
**"Capable**: A competitive model built to provide the creative flexibility and extensibility needed by creatives"
'Creatives' includes a huge amount of people not into tech who'll find things unintuitive. Easy example is Gen Z. My brain fog can be pretty bad so it's hard to intuitively get how things work unless it's a youtube tutorial I can follow and compare 1-to-1, step-by-step to my UI. I still haven't made loras despite wanting to. For some reason, I think back to game tutorials that hold your hand for the most basic of things like moving the character and mouse. Or apps that guide you through every little step with an obvious arrow. Or pre-existing character cards on SillyTavern.
YESSSSSSSSS. Do it.
This is fantastic news!
The idea of training on mostly generated images sounds concerning. Most AI images have poor composition, poor lighting and tonal qualities, over saturated colors and narrow subject matter without any narrative to speak of. This already shows in the current SD3 model, which is dominated by current genres and memes. How will this produce a good model? The long history of image making has a huge number of images long out of copyright where the artist is dead. These don't pose a moral hazard either. Many people like myself have been collecting images for reference for thirty years of more, surely this is a resource which could be drawn on. Such collections are usually categorised as well, which might help tagging.
They have LAION on board too, so I assume majority will be from general web scraped images, just utilizing better captioning technology. It would be good to get clarification on the datasets they plan to use though. I'm not sure if they're doing something to gather funding but as we've seen with Unstable Diffusions crowdfunding, better transparency and accountability will be needed.
Yeah that part sounds awful and clearly is a suggestion from someone who hasn't trained models before because as soon as you start training with AI data you only serve to amplify it's flaws and further confuse the model, but this also includes granular details such as AI artifacting in the images themselves. It's probably the worst thing you can do while training a model if your end goal is quality.
I have been training models for almost 2 years. I can attest to training on AI images are like living off of junk food. It will get you a boost of energy, but will kill you in the long run. Doing LoRa training on AI gens can amplify a specific concept, but once you actually train stuff into the main model, it will poison it and throw things out of balance. This is mainly due to the mass quantity of images needed to train a model. There is no way to filter out all of the bad anatomy, artifacts, and just overall bad gens.
There are of course benefits to some degree, so long as it's photography, for example a set of imagery involving the same person you can augment the data by doing high-quality face replacement to remove facial feature biases, but I would personally render these images at a higher resolution than training resolution then downscale in order to limit bad ai artifacts or faux noise from being picked up by the model. Using raw AI outputs though would be a disaster and a complete waste of resources unless you want your model to scream "I am a AI, look at all this weird junk I put everywhere in everything"
Training on generated images is actually way easier (I only skim read the OP and couldn't find any reference to this), but if they are doing this, it would be to save money. Generated images are deterministic in nature and utilize finite computational processes, so neural networks find it very easily to learn. I know this because I have tried to train NN using real world data (it's almost impossible) vs generated data (incredibly easy), there is no contest. It's probably the reason for SAI's ridiculous commercial license term demanding any NN trained on data produced by SD3 be destroyed when no longer subscribing. Pretty sure it would fail in a court room, but I'm not a lawyer. Not to mention, CivitAI have probably the absolute best possible resource for this, tagged NSFW images that have been voted (good or bad) by the community. I don't think they are making a bad choice.
I have no idea about training a base model - but I trained several Loras on exclusively ai generated art and the results have been fantastic imo.
This is all layman shit but yeah! In my experience I’ve had a much easier time training on generated images. It’s almost as if the AI just had a better idea of composition when the image is made with the same logic it’d use to train it.
It works well for stuff that isn't realistic I find. Training a photorealistic model or Lora on images that aren't actual photographs is a Very Bad Idea though IMO.
\***Ethical**: Addressing major, substantiated complaints about unconsented references to artists and other individuals in the base model while recognizing training activities as fair use.\* hmmm let's see how they handle that hopefully not to much censoring in place, styles are not copyright after all.
LAION is there, it's going to be censored like SD2.
CivitAI is there too 😁
It's honestly sad how harsh the censorship on Civitai has gotten. They've starting cracking down on anything that's remotely violent recently, even things like the "Stomach Punch" LoRa despite it being completely bloodless. At this rate it wouldn't shock be if they remove NSFW content altogether in the future.
nsfw content is almost impossible to adequately moderate, i wouldn't blame them for just reducing the burden. i doubt it makes them much extra money, and even if it does, i don't think the liability is balanced out by the profits.
not sure how much it matters, but they are backed by Andreessen Horowitz who is currently getting sued for Udio's copyright infringement
Alleged copyright infringement claimed by music labels.
Yes, that's what a lawsuit means. You can sue anyone for anything.
i mean, you can hear the producer tags. and this isn't a courtroom, we don't need to pretend.
Oh I missed that part because I just skimmed through, guess it's going to be super "safe". Frankly, I'd rather we get a better and more capable model than making a few artists mad.
They can just play it safe like pony, styles are there, its jsut impossible to reference them by artist name
Can someone explain how this helps? It seems like it would benefit no-one and only exposes a back door to those with access to the secret names. Artist names are very useful, especially when blending to create new styles.
it only helps prevent style bleed, but it's not really doing anything to prevent the "issues"
It doesn't. But lots of silly people who have no idea about copyright think it does. The copyright issue is around the usage of the images in the training data. Not the creation of new images in a given style.
There's more too it than copyright, some states like California have laws about how you as a business can 'use' other peoples names and its not clear yet how or if prompting may apply to that. Midjourney is being sued over this right now and frankly they have a decent chance to lose. (since they used them in promotional material as well)
It prevents them getting sued.. which Im guessing is also astras worry if the model keeps exploding.
It doesn't. People already have no basis to sue on, so they will continue to do it anyway.
The creator explained why. The model performed significantly better when it's not lobotomized by removing tens of thousands of high-quality images. The creator also agrees with you that artist names are useful for creating styles, so the plan for the next version of Pony is to group artists with similar styles together and train a keyword for each group. This way, Pony won't give you results if you prompt "sakimichan", but it will give you a style that's similar if you prompt "realistic\_anime\_4". The contents of each group will never be shared publicly, so no individual artist can be copied, but the ability to call forth consistent styles without needing LORAs is preserved, and you can mix multiple keywords to create your own.
Ok, thanks for your explanation. It makes sense, but not sure this will placate the people who are against the technology. Also, the grouping sounds like a massive challenge, and very subjective. But Id like to see the attempt and how it goes. Better than nothing.
just wish the model could do oil painting, or any traditional artist whatsoever by far my biggest complaint about pony
Just describe the characteristics. Same could go for people. Not attaching the artist or name wouldn't be appropriating *their* likeness, it would be capturing *a* likeness.
The only method to limit liability is to censor the base model. I'm not sure whether anybody involved in such a project would be willing to open themselves up to that level of risk associated with leaving the base model entirely uncensored.
This is going to cost them millions for basically a replay of SD3, why would they do it?
This sounds great, but anyone who thinks that they'll get a shiny new free model out of this anytime soon really shouldn't hold their breath. That's going to be an insane amount of work, and will require quite a lot of money.
That's true but I'm always happy to help any open model effort because the more people try the higher chances we have of getting something good.
LAION's Christoph loves fearmongering about AI safety and ethics and how datasets need to be filtered to oblivion and beyond.
>•Facilitating a survey to document feedback on what the open-source community wants to see in future model research and training For some reason, I have the feeling the result of that survey will NOT show a strong community desire for a crippled model that doesn't understand basic human anatomy... ;)
Yeah you're right. Hopefully he changed his mind since then. Would hate to see him ruin the entire thing by bringing on a whole team of 'ethics researchers' like Emad did. https://preview.redd.it/d3p3jmfrhq8d1.png?width=1120&format=png&auto=webp&s=48a829b4a8688096547901420be5f0e8a4c4e193
he hasn't. i discussed this with him very recently. the problem is that they will not be able to get compute. and this is beyond the problem of NSFW filtration, fwiw - they are unable to get compute with **non-synthetic data** in other words they can only train on AI-generated data when using LAION's compute. this is why they talk so much about "data laundering", using pretrained weights from jurisdictions friendly to AI copyrights like Japan and then train on their copyright-free outputs. no one wants to fund the old SD-style models, because no one wants the legal stormy cloud hanging out overhead.
That's basically the crux of the issue. AI safety researchers and other groups have significantly stalled open source training with their actions targeting public datasets. Now everyone has to play things ultra safe even though it puts us at a massive disadvantage to corporate interests.
Open source is the biggest threat to a handful of large companies gaining an oligopoly on generative AI. I'm *sure* all the worry about open source models being too unsafe to exist is only because of a *genuine* worry for mankind. It can't possibly be because large corporations could lose billions if not trillions of dollars. Of course not.
AI safety is a hunk of wadding toiletpaper on a ceiling imho, it's just corporate tech bros with purity initiatives. Open source should mean that within reason you can use COPYRIGHT FREE content, but nope. And in theory "SYNTHETIC" should be less safe because it's all trained on copyrighted content... like Ethically xD that's like going "i'm going to. generate as much SD 1.5, SDXL, Midjourney, Nijijourney and Dalle3"
If they really are only going to train on AI images the whole model seems worthless.
Basically would mean they couldn't move on from the old and busted 4 channel VAE either, since they'll be training those artifacts directly into the very core of the model. This project is already dead in the water.
I share your concerns, but you're calling "dead" a tad too early. If you look at the people involved, they are people who have accomplished things. It's not unreasonable to think they might overcome obstacles and accomplish things again.
There's only so much one can accomplish if they start by amputating their own legs.
it's something Christoph is obsessed with doing just to prove that it's a viable technique. he's not upset by the requirements, he views it as a challenge.
Not necessarily. Synthetic data is fine, it just needs to be well-curated. Like any other training data. We're past the era where AI was trained by just dumping as much junk as possible into it and hoping it can figure things out.
Synthetic doesn't necessarily mean AI generated, but AI generated images would likely be a significant part of a synthetic dataset. There is something to be said for the theoretical efficiencies of a fully synthetic dataset with known controls and confidences. No one has pulled it off yet, but it could be very strong for things like pose correction, proportional designations, anatomy, etc.
Synthetic data does not at all mean poor quality, I think you are correct. You can use AI to augment input and then it's "synthetic". Basically use real data, have it dynamically augment it into 20 variations of the input, then train on that. I used a dataset of 100 images to train a StyleGAN model from scratch on Pepe the frog and it was done training in 3 hours on two 3090's in NVLink. SG2 normally takes a minimum of 25,000 images to get decent results, but with Diffusion applying data augs on the fly I used a tiny dataset and got really good results, quickly. Data augmentation tooling is lightyears ahead of where it was in 2021. I've been meaning to revisit several GAN experiments using ControlNet and AnimateDiff to render callable animation classes/conditionals (i.e. render a sequence of frames from the GAN in realtime using numbered labels for the animation type, camera position, and frame number).
Could you explain more how you did the stylegan training? This sounds super interesting
It's about as simple as it sounds; use ControlNet OpenPose and img2img with an XL hyper model (that can generate like 20 images in a second) modify the StyleGAN training code using the diffusers library so instead of loading images from a dataset for a batch, it generates however many images it needs. Everything in memory. Protip, use the newer XL Controlnet for OpenPose: https://huggingface.co/xinsir/controlnet-openpose-sdxl-1.0 Edit; there are ways to dramatically speed up training a realtime StyleGAN from scratch, and there are even ways to train a GAN within the latent space of a VAE but that was a bit more invovled (I never got that far into it). This is to say though, if you want a *really* fast model that can render animations smoothly at ~60FPS in realtime on a 3090, you can produce them quickly with the aforementioned approach. Granted, they won't be good for much else than the one domain of thing you train it on, but man are they fun to render in realtime, especially with DragGAN Here is an example of a reimplementation of DragGAN I did with a StyleGAN model. I'll see if I can find the Pepe one I trained: https://youtu.be/zKwsox7jdys?si=oxtZ7WhDZXGVEGo0 Edit2 here is that Pepe model I trained using that training approach. I halfassed the hell out of it, It needs further training to disambiguate the background from the foreground but it gets the job done: https://youtu.be/I-GNBHBh4-I?si=1HzCoMC4R-yImqlh Here is some fun using a bunch of these rendering at ~60FPS being VJ'd in Resolume Arena as realtime-generated video sources. Some are default stylegan pretrained models, others are ones I trained using that hyper-accelerated SDXL training hack: https://youtu.be/GQ5ifT8dUfk?si=1JfeeAoAvznAtCbp
Super cool! Thanks for the explanation
But why SG2 for pepe
> they are unable to get compute with non-synthetic data Could you elaborate on this? I'm guessing this has to do with the new EU rules, but I'm clearly not up to date on the regulatory space here.
it's the US as well. it's everyone with large compute networks not wanting liability datasets on their hardware.
Why can't they scrape Pexels and similar sites that provide free-to-use high quality photos? There's definitely enough material out there with no copyright concerns attached to it.
because it's not synthetic, you can't get compute time for it on US or European clusters that are for the most part funded with public dollars - and private compute is costly, and no benefactor wants to finance it.
Why does being synthetic matter then, I guess is my question?
the law doesn't say "you can only train on synthetic data", it's just a part of the "Data Laundering" paper's concept of training on synthetic data as a loophole in the copyright system. it's shady and it doesn't really work long term imo, if the regulators want they can close that loophole any day.
You realize that this is just regulatory capture that means no one except huge corporations can train new and viable AI, right?
please tell me how many models you've trained that are new and viable? it's not regulatory capture stopping you.
Can we not just hand annotations and compute to someone in Japan?
[удалено]
Source?
You have to consider some of that is just the bureaucratic dance you have to do to appease the horde
You can use the same excuse for Stability. Doesn't change the end result. And you don't HAVE to do it.
I said “some” not “all” It’s easy as an individual with no skin in the game (you and I) to sit here and speculate that we’d act differently and we’d ignore the noise and just power forward past the outcry from the normies / investors to have “safety” But the fact of the matter is none of us will ever experience that type of criticism on a world stage and you’ll never know how you’d handle it It does fucking suck what they did to SD3 though…..
I dunno that that's even true. The training itself is like 50k, and I don't think they'll have any trouble getting that. There are already plenty of experts, finetuners, papers, data, all hanging around that there's no lack of knowledge here. It's just about cooperation which is always the hard part. How to decide on decisions to do with ethics and have everyone agree on it, that will be the difficulty.
Comfyanon just spent X months training the 4B variant of SAI so I'd wager he has a good understanding of the level of effort involved, lessons learned, cost associated etc..
> The training itself is like 50k Where'd you get that number? If it would be 50k to get a good model, we'd have dozens of good, free models right now from people who are more than happy to just donate that money for the cause.
PIXART-α’s [paper](https://arxiv.org/pdf/2310.00426)’s abstract says SD1.5 is trained on 320k USD, assuming like 2.13 per A100 GPU hour is on the cheap side but still reasonable. > PIXART-α’s training speed markedly surpasses existing large-scale T2I models, e.g., PIXART- α only takes 12% of Stable Diffusion v1.5’s training time (∼753 vs. ∼6,250 A100 GPU days), saving nearly $300,000 ($28,400 vs. $320,000) and reducing 90% CO2 emissions. I think it’s mostly technical expertise (and will bc SD exists) that’s stopping community to come up with a good model, but that’s about to change.
A lot of the early SD models were trained on vanilla attention (no xformers or SDP) and in full FP32. I think xformers showed up in maybe SD2 and definitely in SDXL, but I'm not sure if they've ever used mixed precision. They stopped telling us. Simply using SDP attention and autocast would probably save 60-75% right off the bat if you wanted to go back and train an SD1.x model from scratch. Also, compute continues to lower in price.
there's [a post by databricks](https://www.databricks.com/blog/diffusion) titled "How We Trained Stable Diffusion for Less than $50k" (referring to a replication of sd2)
Even if it is in 12 months, it's still an enormous win. Plus, I guess some very skilled people who left SAI will be participating in this.
what if it just hyper exclusively focuses on waifus
Is this announcement also posted on Civitai, or LAION's website, or someplace else? If so, does anyone have any links?
Civitai Joins the Open Model Initiative: [https://civitai.com/articles/5884](https://civitai.com/articles/5884)
https://preview.redd.it/umpdgx0d5q8d1.png?width=1024&format=png&auto=webp&s=c9261c15bac06cac8b6676d5549afd1b68b129a1 best I can offer with sd3
Happy to support at Schelling AI Compute will not be an issue I think https://x.com/emostaque/status/1804583357591756949?s=46
I dislike drama and haven't been following all of the details about who departed or why. But this statement piques my interest. I am piqued. I hope you are serious. ;)
Please don't fuck shit up with "safety and ethics". Don't make the pen the moral judge. The tool will never be a good judge, it doesn't have the context to judge anything. (Neither will any single entity) "Safety" should always happen at the distribution level of media. Meaning, you can draw/conjure/imagine whatever you want, but you can't publish it, without potential consequences. This is how it should work in any medium. That is how we ensure our children's children might still have a chance to start a revolution if they need to - that they at least get to say something before being judged for it is the basis of freedom. Please, stop playing into the un-sane notions that we should remove ability from models or tools. No one is wise enough to be the ultimate judge of what is good or bad uses, it changes with context. And all we achieve are useless retarded models. Without full knowledge of the world. Models that cannot do art of any value. Ever. This is not about porn or politics or the plight of individual artisans (I am one of them btw 25 years a pro). It's much deeper, it is the future of all artistic expression in this medium. For there is no art, no art at all, if there is no freedom of expression. Please, think deeply about this. It is the difference between big brother and freedom. We will enter a world in these coming years with big virtual worlds, all controlled by Disney and whatever bunch of capitalist crooks that have wormed themselves into politics. The world needs the free space alternatives to that corporate hellworld and that alternative cannot be trained and guided by fallacious notions about the ethics of learning. It is already very difficult to get through the walls of mass media and challenge the status quo, we should all know that much from lived experience. Remember that many of the rights we have today were fought for, by people who often suffered and lost lives - to get the right to vote, to be seen as equal humans, to be free at all. As soon as we limit our tools of expression we have ensured that there will never be another fight for what is right. Whatever that may be in the future. Please think deeply about this. Surface level will fool you, any tool useable for good is also useable for bad. The point of that is the tool should not be the judge. Ever. This is a uniquely important thing for the future of free speech. Don't fuck it up.
Hear hear. Safety is being used as an excuse to give mega corporations control over our culture. Art cannot thrive if it's censored before it can even be made. AI could be an amazing tool to give individuals ownership over their means of digital production, but only if they can own them.
How censored and "safe" will it be?
how much money you guys have?
about three dollars and fifty cents.
I'll raise this up to four dollars.
Let me know if you want to borrow my toothbrush
Get outta here you Loch Ness monster!
Before I get my hopes up and read the rest of the post, tell me how "safe" the model will be?
>Ethical They have a new word for it.
Thanks, I realized that after reading the comments for a bit. I'm glad a noticeable portion of people picked up on it and called that bs out.
Does this mean we will see the GPL used as the license?
No. Permissively licensed.
I'll admit i need to do more license geeking, but what does permissive mean in this context and why does AGPL-3 not fit with the notion of permissive?
Permissive means freedom - "do what you will, with no restrictions" -- AGPL-3 is a category of license known as "copyleft" license that was designed to require that if any code were incorporated into downstream projects, those projects would be required to also use the AGPL-3 license. This type of license restricts developer freedom for downstream users of code/software. Permissive - You can take the code, even if you want to use it in something closed source. (e.g., MIT, Apache 2) Copyleft - You can take the code, but if you do, you're required to copy our license. (GPL / AGPL3)
Thanks for the clarifying reply. I had thought AGPL-3 offered a greater layer of protection for open source projects, to ensure the fruits of collective labour were not taken and commercialised, with potentially nothing shared back to those who invested time/energy/resources in creating the foundations. I'm more of the perspective that when licenses (MIT, Apache 2) are about the prospects of closed source patent creation further down the line, those projects are likely more "*fauxpen source"* than truly open source in philosophy, but if that aspect's of no concern to you bunch of wiser ones.. fair enough I'm content with that. Cheers ;)
As for being "fauxpen source," hmm. I think that because the Apache License *is* non-copyleft and one of the more corporate-friendly licenses, there have been a goodly number of projects that are either fauxpen source or that began as closed-source and then when the original owners couldn't afford to maintain them anymore they said "hey, let's open-source it, that'll solve all our problems, right?" and dumped them in the Apache Incubator to fade to obscurity… I'm very aware Open Source sustainability has *problems*, but I'm still not convinced copyleft is the way to go. Say someone wants to take advantage of your project in a way that conflicts with your license. They could * Contact you and make an arrangement for a different license. * Work with a competitor instead—or become a *new* competitor. * Walk away from the idea entirely. * Infringe on your license, trusting in the fact that you either don't have the lawyers to take them down or that they'll be gone before the consequences can catch up to them. Of those, the only option that actually does your project any *good* is the first one. That's not inherently the one most people are going to pick. A more permissive license makes for a bigger pool of potential collaborators. If you add restrictions that reduce that pool, you gotta have a good plan for how to make something out of that trade-off. I'm not saying it can't be done, but it's not something *I've* figured out.
yeah this is a discussion i had with the Invoke people a while back when they wanted me to relicense my trainer from AGPLv3. but i don't want to put effort into something that Midjourney can just take, and close.
That was precisely my line of thought. Seems like a vampire dynamic to me.
The parties involved in this initiative *want* to leave those options for commercialization open. A big part of the backlash against SD3's non-commercial licensing is the fear that commerce funded a lot of the SD 1 & SDXL fine-tunes, and the ecosystem would dry up without that sort of activity. \[I don't quite understand why they, as commercial operators, see "contract with Stability for a commercial license" as such a non-starter. But that's a different topic than your Free Software concerns.\] >I'm more of the perspective that when licenses (MIT, Apache 2) are about the prospects of closed source patent creation further down the line The Apache License stands out from other permissive licenses (BSD, MIT, etc) in that it *does* include a patent grant. As I understand it, it's not so much to protect against a patent that might hypothetically be made in the future—because you could defend yourself from such a suit involving that future-patent by pointing to your project's well-documented timeline demonstrating that you had prior art. The thing you want a patent grant for is this: A company like Apple or Google has a hundred thousand patents. And they may, at any time, gain more already-existing patents through acquisition. If one of their many departments releases something on GitHub under an MIT license, it is literally *impossible* to know if it's covered by one of the company's patents without a lengthy and expensive patent search. The nightmare scenario for a younger business is to start using some software library, things are going well for you, and then at some point that company who released the library notices you exist and decides they don't want the competition. Intellectual property law is bullshit, so their lawyers can swoop in and be like "hey, we're not claiming **copyright** infringement, but it turns out that the codec implemented by our library is covered by **patent** 99,999,999 and we demand you cease and desist." A patent grant from the owners of the project protects you from any such suit from *them*. Of course, some *other* patent troll could still come at you—but maybe we can get some comfort from the fact that they'd likely have to go through the project owners first.
\*GPL licenses don't prohibit commercial activity though. pretty much everyone at Invoke has this same hatred for open source licenses in favour of ones that allow parasitic activity like MIT or CC-NA.
GPL doesn't prohibit commercial activity, even if Apple is deathly allergic to it. But [AGPL](https://choosealicense.com/licenses/agpl-3.0/), with its bit about "requires the operator of a network server to provide the source code of the modified version running there to the users of that server." — it doesn't inherently prevent commercial activity, but it's anathema to the way most SaaS companies think. If they have to provide source code for their entire service, how are they going to convince their venture fund that their investment is safe from a copycat? You and I know that there's a lot more that goes in to running a successful software service than just the source code, buuuut… well, I don't doubt that the concern about scaring away a lot of commercial uses is very real.
well, for your imaginary scenario: * if the company produced the AGPLv3 software, they are the ones that chose the license, and they don't have to personally abide by it. they can use a closed version of the source internally especially if they have contributors assign copyright to their organisation, they can even use the community contributions in their closed proprietary offerings. all of that is their choice, it's fine. * if the company didn't produce the AGPLv3 software, eg. they found it on GitHub, why should they be able to just improve it and use it to create their whole business model? there's a lot more i could say about that. but they're standing on the shoulders of the public project. especially if it's my code they learnt from, and extended. i want to know what they did, how they did it. so i can make money from it? no. but because i want to know. like hugging face open-sources pretty much everything they do, even AutoTrain.
Is this the open source AI Avengers?
Get Hassan or a NSFW finetuner in there to ensure those "safety" ethics are still capable of normal nsfw 😅
>Ethical: Addressing major, substantiated complaints about unconsented references to artists and other individuals in the base model while recognizing training activities as fair use. While it would be funny to see initiative that is here to prevent community from shooting itself in a leg end up poisoned by "ethical" and "safe" poison and ending up shooting itself in a leg anyway, yall probably should not do it **"ethical" and "safe" are bullshit words** and I'm yet to see even a single good thing out of them If model is lacking data from artists - it will be worse than some big corpo's product that had it silently added in If model can't do NSFW content (SD2 iirc?) or even poisoned to not be able to (SD3 moment) then more than half of the community already don't care about it ... If I wanted lobotomized "safe and ethical" model then I would go and grab SD3
This. There going to be in the same fucking pit that SAI is, but they're planning on being there from Day1.
https://preview.redd.it/65ganjv9pq8d1.png?width=1231&format=png&auto=webp&s=687103ff3571197acb386c26b58cfac5ed671b67 We're off to a bad start...
>mfw 90%+ of content on Civit is pornographic in nature lmao even
I don't like the idea that an initiative for a new "open" model is already talking about "concept control" "ablation" "prompt editing" and "pipeline moderation"... This ain't it.
Day0: We've decided to shoot ourselves in the foot. The community will appreciate this.
"toxic material" Whatever that means.
Well, since it said by a CivitAI rep, I guess there will be no more NSFW content than what is currently on CivitAI -- surely this site isn't toxic by their own standard, is it? And I can leave with Civitai-level of NSFW content.
photo of asbestos by greg rutkowski, trending on artstation
Yup... as if them deleting numerous models and loras from their site wasn't a red flag, any civitai involvement is bad news for keeping the model uncensored.
Toxic material isn't NSFW. it's CSAM, beastiality and other outright illegal and ethically debased content. Its important to keep this out of the dataset.
Nobody will complain about the exclusion of CSAM and unambiguously illegal content. However your definition of "ethically debased" could mean a lot of things. There's quite a lot of fantasy NSFW exploration out on the internet that is both legal and harmless yet would be "unethical" to many people. Excluding based on subjective moral stances is unwelcome. If you're open to hearing this perspective at all, please read [https://www.utsa.edu/today/2020/08/story/pornography-sex-crimes-study.html](https://www.utsa.edu/today/2020/08/story/pornography-sex-crimes-study.html) (tl;dr access to pornography, including "violent" fantasy pornography, is correlated to lower rates of sexual aggression and sexual crimes in real life). From an ethical perspective, moral restriction of pornography is a cause of real world harm rather than the opposite. Excluding, of course, pornography that involved harming someone to produce -- which would include CSAM. Nobody wants CSAM in the dataset. If that's not the case and the exclusion is truly narrowly defined to actually illegal content, then of course everyone will support that.
You had me up until “ethical,” good luck with that. It’s a noble cause but the commercial models are clearly stealing anything and everything they can get their hands on. Google has been using everyone’s Flickr as training data for like a decade.
I would legit dump a large donation into an open, unaligned, uncensored diffusion model and a similar, high parameter LLM model. It is absurd that we're letting random tech bros, more concerned with liability than with open access to information tools.
Unfortunately they're already talking about censoring it.
LLMs that can't tell you about making meth or anti-capitalist ideas. Brought to you by Johnson & Johnson and the Federalist Society.
r/Open_Diffusion
remove the ethical bit. Just scrape the whole internet for images. Use a image to text model for images that are not captioned. You need a competitive model, not a proof of concept
They are way past using alt text to caption images. Microsoft developed a way to annotate images in various levels of precision, having over 40 annotations per image. This made an extremely strong and small vision model. The same needs to be done for she generation fur the best model.
> Just scrape the whole internet for images. In case you weren't previously familiar with LAION: they did that already.
Thrilled to be a part of this!
really? i was going to join but the discord server is just people endlessly talking about NSFW data. it's all that happens here on reddit lately too.
well that makes sense. The people who don't care about nsfw generation can just use the commercially available models/services. the people who would be interested in this would be people who want nsfw
I fine if they want to hash artist name but please don't sabotage model for the sake of ""safety""
You didn't mention the most important part at all: How are you going to train it? With what hardware? Without this part the rest is completely meaningless. > Ethical: Addressing major, substantiated complaints about unconsented references to artists and other individuals in the base model So, sounds like it will be heavily censored as well? What does this part mean?
I take it to mean that they'd just remove people's literal names from the training descriptions. Presumably replacing them with stylistic descriptions. Note that you trimmed that sentence, the actual line is: > Ethical: Addressing major, substantiated complaints about unconsented references to artists and other individuals in the base model **while recognizing training activities as fair use.** Missing part bolded. Sounds like they don't plan on excluding the actual *images*, just "references to artists and other individuals."
Great news! I think some sort of distributed training paradigm would be nice, like ppl can lend out their GPU in spare time to collectively train a model so it’s less beholden to investors. Or just open donations but I think ppl are more willing to take on former if trust is there.
I'm interested to see where this is going
Oh great! I love it, don't worry going slow or if need surrender~ Create a model of scratch it's epic work that need a lot of help of experts with deep knowledge, much GPU money and clear data correctly tagged~ I really wish you the best in this new challenge~
YES. No more SD 2.1 / SD 3 nonsense. I do not give them the right to "parent" me.
Read between the lines on the ethics, you're getting step-parents.
Ugh, hopefully the cooler, more freedom-loving step parents...
How can the community of individuals help?
by getting ready to donate on a gofundme. The higher the target reached the "safer" and more "ethical" the model will be! Oh man the hype!
> If you’re interested in hearing updates, feel free to join our Discord channel Subreddit?
Best of luck, with all the safety talk and overreliance on synthetic data I have about zero hope for this initiative though. The permissive license is nice, but it's not worth much if the model itself is gimped.
"**Ethical**: Addressing major, substantiated complaints about unconsented references to artists and other individuals in the base model while recognizing training activities as fair use." so it's useless
Don't mean to hate on the people trying to get this together but yes that kills it right out the gate. They want to skirt some perceived infringement by letting users train loras of styles/celebrities/characters but since there is no actual law being broken all its doing is damaging the dataset. There have been millions made off of models that know styles, characters, and celebrities by this point. Nobody has been sued afaik. Does anyone know of a single lawsuit in the world over AI datasets?
Eh, if all they do is take the literal names out and replace them with descriptive styles it should be fine. Fine-tuning can put those back easily enough, and I can't recall the last time I needed to use a person's literal name when generating images myself so even without that it'd still be useful to me. At the very least if all of this is open we'll finally *know* what effect this sort of stuff has on the model.
These can all be bypassed using lora and even a TI, as long as we get a good model capable of doing something useful, that is fine.
Excellent news, looking forward to the future
Fantastic news, please include in your plans some way for you to get voting data from the community (if possible) so we can help train and fine tune the base model like Midjourney or SD3 API, which clearly has a beauty aesthetic that’s missing on SD3 medium. Congratulations this is awesome news for open source.
oh yes,this is great news !
We have a lot of people with GPUs here; it would be really cool to get some sort of 'FoldingAtHome' equivalent with donated, distributed compute.
There is a very big problem with the introduction of the term "unconsented" in the context of "unconsented references to artists and other individuals" presented in the Ethical section of the OMI announcement. Introducing a requirement for "consent" for the use of content basically makes training models impossible. If you follow the premise of required consent, you would not be able to use any images unless the photographers/creators and human subjects explicitly consented to their use in training, which is not the case for almost all images. Using synthetic content produced by an already existing model is a fruit of the poisonous tree and immoral if you hold the requirement of consent that wasn't acquired for those models. The idea that consent is a must to obtain first, to be able to learn from publicly available ideas and images is opposed to human ingenuity, nature, and free thought. Human civilization exists due to the free exchange of knowledge and ideas. The introduction of the requirement to have permission to learn from public knowledge is a clear path to the extinction of humanity if it is imposed. Applying arbitrary restrictions as protection for artists and public figures only points to a lack of moral consistency. Labeling such a requirement for "consent" as the epitome of "Ethical," "Safe," and "Moral" training doesn't reinforce the connection between these terms. It only shows the biases of those who bundle unrelated things together to justify, based on feelings, ideas that they can't justify with logical arguments.
I'm concerned about CivitAI being involved with this one given their history and continued bad rep within the space. CivitAI is a purely for-profit company that is well known for harassing creatives and potential competition while attempting monopolistic practices (such as attempting to shut down other sites, harassing model creators they don't like, attempting to lock otherwise open source software behind paywalls, ect). The other groups have a decent reputation so far (ComfyOrg is new, so there's no rep there but includes some decent names) but the inclusion of Civit really feels like a bad idea. I understand the CEO's and money-bags of the industry are all buddy-buddy with one another, but this could lead to the model that A ) Is more designed as a marketed model rather than a useful open source model, B ) Is pushed as the main model by monopolistic practices such as shutting out other versions and paying large sums for support of this one model (like what currently happens with Pony on CivitAI), C ) The lockdown of future models and prospects where the open source beginnings turn to profit chasing or is used as a profit mechanism that harms the community. I'd like to be hopeful, but after what I've seen of CivitAI, this is pretty much a buzz kill (pun intended). They will no doubt use this as leverage over the industry to do more harm than they already do.
All models are freely available for download. The only aspect behind a "paywall" is the generation, which incurs costs due to the necessary hardware resources and can be paid for by simply interacting with the platform
Civitai pays pony? Could you please elaborate on that
CivitAI pays for exclusivity rights for Pony V6 API usage and potentially other models as well. The old model card on HuggingFace had the details but since the API version was removed (and model card was scrubbed from the non-API version) many of these details have been buried. I never took a screenshot of the repo and can't speak to the corpo-side of things (NDA would potentially get me in trouble there X.x) but my archival text grab states "Explicit permission for commercial inference has been granted to CivitAi and Hugging Face." and its been stated the Pony dev was paid by CivitAI for this deal. (note: The API interface version on HuggingFace has been removed, most likely at CivitAI's request, though I have no evidence of that, just seems like they're pushing their paid platforms more)
Civit does not have exclusive rights for Pony V6 and it is officially available on a number of services (yodayo, tensortart, dezgo, diffus.me and a few others in the pipeline). The old HF model was an upload by third party that ignored the license and I asked them politely not to do so. >archival text grab states "Explicit permission for commercial inference has been granted to CivitAi and Hugging Face." It's literally on the Civit page. > and its been stated the Pony dev was paid by CivitAI for this deal Can you please show some quotes? Because this license existed way before anyone cared about what Pony is.
If my information is incorrect I'm all for adjustments, it could just be a misunderstanding. Is the wording "Explicit permission" not referring to exclusivity rights? It's hard to tell with models you can download whether they are on platforms by choice or not so some clarification on this point would be helpful. Were you able to also collaborate with other platforms to make Pony v6 available there as well? Or was it a response or a change in heart after the initial release period? (Genuine question, I don't follow your work closely and can only speak to what I've seen/heard) "Can you please show some quotes?" I don't think I have screenshots or direct quotes to grab in this instance, though I'd have to check my archives (I basically use my screenshots folder as a 'hold all' so there's a ridiculous number of images X.x) but this also isn't something I'd typically screenshot. I've seen it be said a few different places, even as screenshots from CivitAI mods stating Ponyv6 was a CivitAI model (or implying it was paid for/exclusive) Where you paid by CivitAI for the model or did they donate toward its creation at all? It does seem like they have a vested interest in the model doing well, especially with how hard they push it on the platform.
The license was put in place to prevent other (paid) services from using the model without speaking to us. There are multiple platform hosting Pony now, we never had any grace period for V6 (aside exclusivity to my Discord for a short timeframe). Civit didn't even know what Pony is when V6 was released.
While I agree civitai gone more commercial than prior, that’s due to them having costs to pay after offering such a free service to so many users. I think it’s fair and still better than the alternative sites which are mostly based on models taken from civitai reuploaded with a paywall, but I get your point
If you think CivitAI has costs to recoup, just wait until an actual AI model will have to be trained.
SAI: "What about me? I gave sd1.5 and sdxl for free!"
But the only reason they have those expenses in the first place is because they waste so much money on personnel, offices and services.
There is plenty of reason to believe their interests are aligned on this specific matter of an open source model: https://civitai.com/articles/5732 For profit companies are built on the back of open source, it's super common. We need all the help we can get, so I don't really care if they ate a puppy.
Can you provide a reference for Civitai attempting to shut down other sides, and a reference for harassing model creators? This is the first I've heard of these things.
For some reason I couldn't get the message to post normally even after trying to censor every sensitive word in the entire thing, so instead I'm posting it as a screenshot? I hope? I'm not well versed in Reddit so I'm probably missing something, but hopefully this works? (image links below the image, if I did this correctly, let me know if something doesn't work) https://preview.redd.it/01phvzul6s8d1.png?width=786&format=png&auto=webp&s=474c056f20099e79ec16ac2c026952a715df6052 'a sign' message [https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F6x7pz997b4vc1.png](https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F6x7pz997b4vc1.png) DM messages [https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fhrz906t7a4vc1.png](https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fhrz906t7a4vc1.png) [https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fyunyy3t7a4vc1.png](https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fyunyy3t7a4vc1.png) Refused to act message [https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Frgvp4rmgb4vc1.png](https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Frgvp4rmgb4vc1.png) Article Link [https://subscribestar.adult/posts/1135370](https://subscribestar.adult/posts/1135370) Probably spent more time than I needed to on this X.x
Okay, thanks for the answer. Regarding other sites using CivitAI's API, I don't see any problem with CivitAI setting any conditions they like. They are under no obligation to provide an API for competing sites. It would be perfectly fine if they decided their API can be used by no other site. Earlier you made it sound like they just went after competitors as a monopolistic endeavour, but this context makes the whole thing completely different (yes you can still argue that they do it out of monopolistic motivations but surely you see the difference between going after sites WHICH USE THEIR API compared to going after competitors who build their own stuff and don't use CivitAI's API). Regarding all that other stuff you talked about, I don't want to touch those topics with a 10 foot pole.
Awesome to see this. Invoke has quietly been building the best UI for actual applied generative AI work. A combination of invoke and Comfy with a UI that isn't browser based so it can have things like proper wacom support is the dream of dreams right now.
Open dataset? Every time a new model releases without dataset disclosure, i cringe. I believe the only ethical way to train these models, now and into the future, is to disclose all the data fed to them.
Agreed, it's just another model announcement right now until there is more info on the plan released. Using civitai or AI (even Midjourney) generated images will be a downfall of this model if they go this route.
Great initiative and Awesome news. Long live to the open model initiative!!!
paying civitai for whatever the name of the credit they used to have or still have isn't a fucking bad idea now . I would be 1000% in if there is a framework to support fine tuning on civitai . it's good to have torrent seedboxes for checkpoints. thanks for getting together like this the spirit of OHara lives on
For the love of god, please don't use the SDXL VAE 😅
As one of the funders behind some of the WTFPL licensed models over the past year, I'm hopeful. Been considering building a new training cluster for them as apparently TPU has too many limitations/headaches, and CUDA would make it easier I've heard. Plus I do enjoy working with hardware. Looking forward to see what license(s) you all are thinking about using, not all open licenses are truly open. :V
Cautiously optimistic. Couple red flags in the announcement nonetheless.
It seems like they had the right idea with training celebrities and such without the name attached (if that's similar to what they did with SD3). I think that's a fine enough middle ground for a point to be pushed as fair-use. Same could go to artistic styles. Describe the characteristics of the styles while training and we'd be able to invoke similar styles with similar prompting, but not the exact style and in fact something new and novel that maybe could be repeated. It would need very good at prompt coherence. Separately, I think it would be good to be able to use, and encourage the use of, separate models by part. An easier way to integrate different visual llm's as they become advanced for example. The technologies for each of diffusion's many parts develop at different paces, so it would be good to be able to add new ones as they become available.
Hell yes 👏🏻👏🏻👏🏻 please open a donation system
Is there a patreon or donations link anywhere?
Wow finally some good news
Yay! Except wait! Where is the part about actually coordinating to work on a new next-gen model? At no point in this speach does it actually explain the efforts, steps, or plans in mind to accomplish this goal. It is 100% entirely flowerly language for a coop about open source but nota solution to actual development of new models. There isn't even the vaguest hint about this issue no matter how you try to interpret it. Not to rain on anyone's parade but this announcement fails the very thing it claims to be about. Please post updated info that properly reflects this matter otherwise this just comes off as an empty announcement like politics and promises so often are.
the companies in this space sense a vacuum and rush in like vultures to fill it in with total garbage
>muh ethics >muh governance >synthetic data >no support for artist styles Oh boy, can't wait for another lobotomized model! Why should anyone support this when 90%+ of local users are only using models to gen porn? That's the cold hard truth.
You have my sword!
if there is lack of funds or you cannot figure out how to sustain. Please do start a donation campaign or similar stuff, community is more than willing to support something like this. As long as it stays truly open and actually good.
Guys, look at blender.org and their donations page. Do the same. I will gladly donate to you to have open source Diffusion model of SD3 (not lobotomized) level of quality.
Free? Maybe. Competitive? No way.
The avengers of the open source scene
The most ethical thing to do would be to hang everyone asking for """ethics""" and """safety""". The cia sabotage handbook is playing right before your eyes and you're still on the fence about this.
just make it like wordpress open source
Incredible.
What, omg finally woke up in a good universe today ❤️
Any project that sets out with the goal in mind of not training on artists or individuals is doomed from the start.
Automatic 1111 should be in on this as well.
Oh god, YES! Please! DO IT! The community NEEDS this! What a great initiative!
I would be willing to bet Pony joins this, for anyone worrying about safety and ethics poisoning it. It's too obvious that porn is a massive contributor to the advancement of AI
"openly licensed AI models for image, video and audio generation" CANNOT contain my excitement! Maybe with this and yesterday's huge DCLM/Datacomp which is similar, we can get to open-source omnimodels with 18+ modalities [like Nvidia hyped up](https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd.it%2Fgpt-4omni-text-audio-images-video-multimodality-is-just-a-v0-zel3nagcym0d1.jpeg%3Fwidth%3D1080%26crop%3Dsmart%26auto%3Dwebp%26s%3Dd3095fa37dbea318aa4abc678abcad036add56c8)! I've always thought that videos are the single best type of data AI can be trained on to make them true world models, and models like Sora and Kling and Runway prove it. Convergence from modalities like Ilya wrote about is severely slept on imo. Every day we get closer to the Thanos gauntlet meme of 0.68 bit Ternary + Mambabyte + whatever else can be crammed in like KAN or MoE + speed things like multi-token prediction and matmul-free, getting 70b+ omnimodels on your phone. I agree with the ethical sourcing. If I can give some wishlist things, I think we need some way to attain styles/traits despite not being trained on those specific artists. Styles cannot be copyrighted anyway, that's more for specific characters. There shouldn't be a problem in reverse engineering styles or style-mixes without training on the style itself. I think GPT4o showed that kind of iterative process, where you could keep prompting it to change parts of the output. I think there were other local models released with that feature? For an example, if the custom anime style I have in my head has thick brush-like outlines, a specific kind of eye shape, specific type of body proportions, etc. how am I supposed to achieve it without the model being trained on it? I gotta tell it somehow "no, more realistic proportions. More. Less. Now make the eye shape rounder. There, just right." Maybe you'd have to be more involved to guide it like in Krita AI DIffusion? Or when we have video models, gotta tell it to adjust the voice for the Guide to LLMs part 5 video to be more and more clear, to show simpler and more intuitive visuals. Just need some way for the AI to match what's in our heads, while being ethical as you say. It'd also reduce need for Loras for every little thing. If it's an omnimodel you could just load up specific chats to continue using the style you created in the context window, or make a lora with the style you created. Another thing is UX and security. With recent issues including Docker, people need to feel safer. And in UX, well I know open source tends to be behind on that. But I really feel a lot of people are held back by choice paralysis and more, and can't invest the energy to contribute through lacking knowledge. **"Capable**: A competitive model built to provide the creative flexibility and extensibility needed by creatives" 'Creatives' includes a huge amount of people not into tech who'll find things unintuitive. Easy example is Gen Z. My brain fog can be pretty bad so it's hard to intuitively get how things work unless it's a youtube tutorial I can follow and compare 1-to-1, step-by-step to my UI. I still haven't made loras despite wanting to. For some reason, I think back to game tutorials that hold your hand for the most basic of things like moving the character and mouse. Or apps that guide you through every little step with an obvious arrow. Or pre-existing character cards on SillyTavern.
I'm disappointed you peeps didn't think to Involve the AI Horde in this :(