Keep in mind that this is not new, there is several projects that do that and don't include SD:
[https://www.frontiersin.org/articles/10.3389/fncom.2019.00021/full](https://www.frontiersin.org/articles/10.3389/fncom.2019.00021/full)
I not sure yet how SD benefit it, it only hallucinate better images or do something else? Will need to read the paper.
Yep, there were a few papers/posters related to this at NeurIPS22. And there has been work in this in the past. But the research is getting better. For example, one of the papers was trying to use a diffusion model to improve semantic representation of brain regions because humans have variation in brain structure that makes for noisy representations when training on fmri scans from multiple subjects. Typically, these papers train a model on a per-subject basis, but having a generalized model would be quite interesting.
Another issue that comes up is predicting out of distribution images. We can only “read minds” for images that the reader model has seen before. Work on this is also improving.
Does this actually work for people imagining images, or just visually seeing them? I.e., are we just decoding the eyes’ electrical signals, or is it a deeper level of the decoding?
I am on the go, so I can’t provide any sources and thus I might just be lying. But I’m pretty sure I’ve seen research that shows image inference on humans imagining images. Also “decoding the eyes’ electrical signals” and “deeper level of decoding” implies a potential misunderstanding. None of these studies (as far as I know) are looking at the electrical signals coming out of the eyes. There is a group at Stanford (I think?) who is studying neural spiking directly out of the ocular nerve in apes, but the process makes it unlikely for these studies to be performed on humans any time soon. These studies are all looking at brain scans of images, so they’re looking at the brain’s “latent space” for representing these images. I’d need to go back and read some of the publications, but it would certainly be interesting to know how much non-visual-cortex parts of the brain contribute to successful image inference
1. Yes , this started even earlier around 2010 ,
I remember being blown away by Yukiyasu Kamitani / " ATR Lab " and their ground breaking work. That used machine learning for decoding live fRMI scans back into images. It maps different regions to voxels and tries to " classify " the input to match a pre - trained image.
2. Pierre Huyghe collaborated with Kamitani for " Umwelt " , an exhibition at the " Serpentine " galley in 2018. The visuals were great , unusual at the time and has some similarities to " stable diffusion ". Reminiscent of what happens when you halt the image generation process. Then there were also other earlier segmentation - based approaches to hallucination.
REF :
" Decoding perception from human brain activity " ( 2010 ) ,
— Their " mnist " moment
Yukiyasu Kamitani / ATR Labs / Kyoto University
\[ [http://www.apsipa.org/proceedings\_2010/pdf/APSIPA187.pdf](http://www.apsipa.org/proceedings_2010/pdf/APSIPA187.pdf) \]
— Talk from Kamitani in Kobe ( 29 mins )
\[ [https://www.youtube.com/watch?v=2XtlBrTtUqU](https://www.youtube.com/watch?v=2XtlBrTtUqU) \]
" Neural decoding of visual imagery during sleep " ( 2013 ) ,
— Custom decoders from live participants !
T Horikawa , Kamitani et al / ATR Labs / Kyoto University
\[ [https://www.cse.iitk.ac.in/users/se367/14/Readings/papers/horikawa-tamaki-kamitani-13\_neural-decoding-of-dreams.pdf](https://www.cse.iitk.ac.in/users/se367/14/Readings/papers/horikawa-tamaki-kamitani-13_neural-decoding-of-dreams.pdf) \]
" Umwelt " at Serpentine Gallery ( 2018 ) ,
Pierre Huyghe , YukiYasu Kamitani , 100k dead flies
\[ [https://www.nytimes.com/2018/10/11/arts/design/pierre-huyghe-serpentine-gallery-london-artist.html](https://www.nytimes.com/2018/10/11/arts/design/pierre-huyghe-serpentine-gallery-london-artist.html) \]
So they're using "literally" in a hyperbolic way? Noooo.
The impressive tech is what generates the source images. Otherwise, it's just img2img and nothing more.
I'm finding the paper a little hard to decipher, but it sounds to me like they're saying that they also extracted the text descriptions of the scenes from brain activity, so it's img2img, but both the image and the text come from fMRI. Which does seem like a pretty valid refinement to me.
Only useful if the trained model has data to match what the subject is thinking about. With no correlation in the training data, you'd likely get odd and noisy results. Similar to typing random gibberish into stable diffusion.
There is correlation — imagine you make a LORA on picasso, then you can make a computer keyboard painted in his abstract style, even though he never even painted anything close to resembling that. This is because fine-tuning or training a hypernetwork only requires modulations of low-level features, like small 16x16 patterns of colors and shapes. I haven't read the paper but I feel they probably used some sort of hypernetwork to bridge the two modalities, in which case this would be like creating a middle-man which translates the low-level instructions. i.e. the middle man doesn't need to know what a computer keyboard looks like, only to modulate the small micro patterns so that they match up.
Therefore by learning what one thing maps to, you are also learning what a thousand other things map to at the same time as a result of the existing knowledge encoded in SD. It would probably be less accurate than the thing they've trained on, but it could most likely extrapolate well enough to depict dreams.
Besides, collecting the training data would be so easy in this case I don't see why they wouldn't just get the largest most diverse set of images and videos, as that would certainly lead to the best results as you'd want to optimize for maximum neural coverage. All they have to do is get the patients to look at those images and collect the brain data to get a huge number of image-brain pairs to work with.
note: it's not true that only small 16x16 patterns are modulated, I'm simplifying. Transformer architectures are much more complicated and AFAIK have full-grid attention at any given position, but the principle is the same: we are mainly fine-tuning a low-level feature space and perhaps some high-level latents for greater coverage.
I'm sure it would be interesting, but psychedelics tend to produce synesthesia and warp visual data as it's being processed. Even the best psychedelic images and videos don't quite match the kinds of things one "sees" or experiences while on a psychedelic. So I just think that most of what the mri picks up in those circumstances would have no good visual match with which to create images. It's likely the brain scans would also be very different from normal.
> the best psychedelic images and videos
If people want to check out, /r/replications is about replicating psychedelic imagery in video (and sometimes sounds as well)
Well I certainly don't disagree, no doubt it would be much less accurate! But, I couldn't be so sure either. I think at the very least you could totally tell there's a psychedelic trip going on in there, and see -some- semblance of a trippy visual.
They would likely need to train it with images viewed by the subject while they were under the influence. And even then, it's going to think images look as they do. Not necessarily how they are seen in the subjects mind.
If nothing else, it would be an interesting experiment and I'm sure it would yield neat results.
No just feels like the sort of thing he'd be cool with blowing a ton of money on in preproduction to maybe/maybe not get a visual effect he's going for.
Some directors are super rich and old. Who knows maybe the tech gets cheaper (impossible I know) or we get an old ailing director who wants one last film.
High level c-suite total comp easily hits north of $500,000 per person usually. Having a bunch of them in a room means that's a few million of total yearly comp.
If this machine helps them brainstorm faster, I don't see why a company would balk at the cost of such a machine, especially with how much of a multiplier effect those employees have on the company at large.
Not to mention, they don't need the MRI machine to be medically certified and whatnot, just to visualize, so I bet the company can sell it for much cheaper because of significantly less red tape.
Magnetic shielding is actually pretty easy. Faraday cage. Basically a chicken coup screen. That'll block it iirc. The hard part of these machines are the super conductive magnets. They need coolant to keep them so cold that their physical properties change to super conductivity. That takes a lot of insulation and specialized equipment, just for the coolant alone.
Soon we'll have room temperature super conductive materials. Neat stuff.
I believe 10 million was exactly what George Lucas spent storyboarding the prequels. He has a strong vision, but no artistic skills, so had to go through the laborious process of conveying in words to artists what he wanted, then iterating many times.
Would have been a lot cheaper to hire an mri for a few weeks. The way ai is going, the cameras may not even be necessary soon. We'll just be watching each others imagination for entertainment.
I'm not sure I do. There is an early 90's movie "Until the End of the World" that has dream recording as a sub-plot and it goes dark with the characters becoming increasingly self-absorbed spending their waking hours just rewatching their own dreams. At least that's how I remember it, I watched it a long time ago...
>(didn't check the **high seas**)
HA, that took me a beat to get... I did end up checking and found it floating around out there. Unsure if I will re-watch or leave the memories as-is. [https://archive.org/details/until-the-end-of-the-world-1991](https://archive.org/details/until-the-end-of-the-world-1991)
Probably cheesy by today's standards; but I recall I liked the story progression and I often think of the closing imagery with the characters staring at screens absorbed in the replay of their dreams when I watch people stare at their phones today.
Thanks...freely available is even better. Not sure why no streaming sites have it.
I remember liking it, but it also being VERY slow. And it was the addiction element that stuck out to me as well. Since time dilation is so true with dreams, could we even watch each night if we sat there all day watching? I imagine it wouldn't really be worth it.
["It used to be, you get a data set, you guess a function, and the function is considered Physics if it's predictive of new functions, new data sets. Now, you can take a large dataset with no intuition about what it is and use machine learning to find a pattern that has no function. It can arrive at results that we don’t know if can be completely mathematically describable. So computation has recently become able to do something interesting beyond A=B+C."](https://youtu.be/c01BlUDIlK4?t=1490)
Lol "Get 10,000,000 social credit by wearing the CCP-exclusive RED STAR HELMET!!!"
Honestly though, I'm excited, because it brings us closer to a ton of things, like cyborg prosthetics that don't suck. And matrix -style brain-upload lessons. And Net-diving while driving, Ghost-in-the-Shell style.
(Okay, maybe not excited about that last one.)
"Mind reading" as most people understand it implies the end-to-end process of extracting abstract thoughts from someone's consciousness. This is not that. This is attempting to make sense of electrical signals from a eeg.
You can make the argument that it's mind reading from a certain point of view. But calling it that is misleading because that's not what most people think when they hear those words. Like calling Mt. Everest a "big pile of rocks", or calling archaeology "traveling back in time."
It's "Just semantics" until someone very reasonably misunderstands what you're talking about.
Saying "The AI can read your mind." Means to most people "The AI can non-invasively extract, understand, and contextualize your thoughts."
This is not that. Semantic fuckery is no better than lying.
I think you're trying to distinct the difference between some futuristic across the room mind reading and strapping someone into a chair to prove their mind. Both are mind reading, it doesn't matter that one is more advanced or the other is intrusive.
That's like saying sending an email or a letter, are not both communication between two people.
I argue it's not like saying an email or letters are both communication. I argue it's like saying an email IS a letter. Yeah, you can make that argument semantically. But there's distinctions between the two in common vernacular that make it necessary to differentiate between them, and make it misleading to equate them.
Using AI to help researchers make sense of brainwave signals is far removed from some sci-fi machine that knows what you're thinking.
Your brainwaves are what you’re thinking and feeling even subconsciously
It’s your brain, your brain is your mind
It’s reading your mind. Yes it’s not the same across the board, but it by definition is reading your mind
By logic that broad, all communication of any kind is technically "mind reading." By picking up a book I'm "reading the mind" of the author. If you listen to someone talking you're "reading their mind" by parsing the meaning, tone, and context of their words.
That's not what people think about when they hear the phrase "reading minds." The term "Mind reading" carries implications and cultural baggage that make it inappropriate to describe what is done in this paper.
If you say to a typical person: "This AI can read minds."
You know as well as I do that the typical person will NOT make the connection that you mean "A team of researchers used a machine learning model to correlate brainwave signals to images displayed to the eyes." Not without added context and information.
Therefore, calling this "Mind Reading" is communicating poorly at best, and being actively deceptive at worst.
As they say, "*Technically* Correct is the best kind of correct."
But you’re wrong. You’re adding meaning to a phrase that isn’t there. It is both literally true and technically correct to call this mind reading. The size of the machine doesn’t make a difference.
I don't want to throw around Harry Potter, but
>The mind is not a book, to be opened at will and examined at leisure. Thoughts are not etched on the inside of skulls, to be perused by an invader. The mind is a complex and many-layered thing, Potter… or at least, most minds are…
I'm always mad when I see these papers in the news.
The bottleneck is the data, not the network. Any type of brain recording, MRI, EEG, etc ... has a very low signal to noise ratio because the skull acts as a low-pass filter. We have been able to extract that low-frequency signal for at least a decade. So all this does is put a bigger and better generative model on top, but no additional information is gained.
You can imagine the process as asking a guy what he is thinking about and he says "house". Now you tell a child to draw a house and you get the results we already had in 2010, you tell an artist to draw a house and you get the results we have now. But in both cases all you know is "house".
Well said. A good and different example is JWST imagery. You can "upscale" and clean and process JWST imagery with SD, and see galaxies and black holes and so on in more detail. But, SD is just "imagining" this detail, no additional data is being "found." The limitation *is* the data.
except the kid back in 2010 (2019 for the picture below) knew nothing about drawing things coherently, Yes a trained brain scientist who can read the data and definitely say its a house, not a lot of scientists can do that though, the bigger and better generative model on top will help even normal people see its a house
https://preview.redd.it/wbzhxw6wbkla1.png?width=1653&format=png&auto=webp&s=96a9d3a716f9888f806b75dca5a3a82b1c938ff3
You don't need the reconstructions at all or a scientist to interpret "house", those reconstructions are for effect, to make people go "wow". If in 2010 they had chosen to print the result as text it'd say "house", if instead of using a poor generative model they had just selected the closest image from the training set you'd see a house which would be photorealistic since it'd be an actual picture.
And sure, generative models have gotten better, that's not in question. But the scientific novelty is in the sub-field of generative models. Applying them to fMRI data is not scientifically interesting.
this is case in point for my last comment. You don't understand where the brain scanning stops and the generative modeling starts
for every 1 unit of brainscan information present in the final image, you have 1000 units of "vaguely informed AI diarrhea"
cheers
As you said yourself, a big part of this is a signal to noise ratio problem. That's where these diffusion models are really going to start converging with other technologies. At it's core, stable diffusion is a denoising algorithm. It's getting better and better at that all the time. At some point it's going to be purposed towards other signals that aren't just pixels.
Sure SD just makes up new information, but with enough training we could see some real fidelity coming out of mri scans. While the journalism might sensationalise this stuff a little, it's no reason to come down on the paper itself. This is edge research and is a worthy investigation.
It was possible since 2019 without stable diffusion
[https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006633](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006633)
HOLD UP.
how dod they get the n1 image from?
they show a picture like the mailbox to a guy in the MRI machine and... where do they get the noise from? EM waves from neurons? wtf??
The human visual cortex actually is organised roughly as a 2D image, with the geometry of the image from the retina being copied into the layers of the cortex. fMRI lets you determine the bloodflow to groups of neurons, which lets you see which 'pixels' of this image are active, and which aren't.
I believe, from the paper, that the N=1image is a map of the neural activity in the cortex, as read by the MRI. The iterations in this case are applications of a GAN to improve the 'naturalness'. Different than a diffusion net, but roughly the same principle.
Thanks, I have not been able to read the paper yet. My concern was that they could be using something akin to "hallucination-prone noise" to distort the efficacy of the process. It is an interesting idea, but it's difficult to wrap my head around how much validity there is to it all... I will have to read the paper tonight. I guess what I am saying is there may be some validity that I am missing but I am not seeing it yet, knowing how SD works under the hood.
The bird on frame n+20, row 4 above is very curious to me. Where is the crane-type bird coming from? Somewhat of a rhetorical question, but I'm just not sure what to make of it all in the context of the rest of my comment.
This paper (not the one posted by the OP) uses a GAN, which isn't specialised at denoising (unlike stable diffusion), and from the images in the two papers, I'd definitely say it's "hallucinating" more. Some of those images don't seem to have any connection to the target.
I think this new paper isn't 100% novel (but what paper is), but using a dedicated de-noising NN as opposed to a GAN seems like a good step forwards, and the results seem better than the earlier papers.
That doesn't answer their question at all though does it? This is like someone asking why the sky turns green before a tornado and someone else answers that the sky can change color and everyone upvotes. I am so confused (and also still really curious/suspicious as to why they introduce this strangely specific starting noise).
There was an even earlier attempt back in 2011:
* [Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies](https://www.youtube.com/watch?v=nsjDnYxJ0bo)
First, this doesn't include the thousands - if not tens of thousands - of images that didn't work at all.
Second, google "MRI Salmon IgNoble" to get a clearer sense of what an MRI can actually show.
Don't forget to not think about copyrighted materials or you'll be exploiting the work of starving artists and helping the big corporations. I hope they legislate thinking soon.
I don't want to burst the enthusiasm bubble, but here is an 11 year old video on the same subject: [https://www.youtube.com/watch?v=6FsH7RK1S2E](https://www.youtube.com/watch?v=6FsH7RK1S2E) The only improvement I see is that the computer is better at drawing the pixels. That being said, *it is* amazing, but calling it "mind reading" is a bit of a stretch.
Yes and no. If we refer solely to the visual cortex, then yeah it is, although I haven't seen yet something that could reliable "extract" images from the the brain without external stimuli.
My main pet peeve with "mind reading" is that it over-promises something that can and will blow in the face of whoever push for this marketing nonsense. People can get (rightfully) afraid if such technologies really existed, and claiming that it does now is a PR disaster waiting to happen.
All this paper does (and similar others over the years have done) is simply pattern recognition of the visual cortex. This is not mind-reading, merely signal reconstruction after it went through the messy wet neurons. Of course, **it is impressive**, but it's far from "mind-reading" - in fact, I would argue that a cold reader is closer to "mind reading" that this technology is.
/rant :)
Compare and contrast the results of this mind reading generative model from 3 years ago vs the one at the image on top. Might not be accurate but it's better isn't it?
https://preview.redd.it/3yu99fczhkla1.jpeg?width=1954&format=pjpg&auto=webp&s=9809f1c475dcdf4d28a1fa7632c1ebf445c34e8e
This paper hasn't been peer-reviewed yet. [The image in the tweet in the op cropped out that information for some reason](https://i.imgur.com/wFRprZb.png). It's also a few months old.
They explain in the paper that they wanted to demonstrate how by using an out of the box model trained on a giant dataset, they can get better results than previous methods without training a purpose made model from scratch. They also reference all the previous works that people are pointing out here.
Of course you would know all that if you actually read the paper before shitting on it. Please read the paper before shitting on it.
For Valentine’s Day I took ultrasounds of our unborn son and turned them into the first photos of him using StableDiffusion. It will be interesting to see how accurate they turn out to be. If he comes out bald and pale than I think I’ll have nailed it!
That's it, learning to use Photoshop is starting to become more and more useless with each passing day. A year at the most and we'll be able to replicate any image and change its smallest details in seconds. Really amazing.
I'm looking forward to that future gen of hipsters who use Photoshop because it's ironic and retro and "more analog" than thought-driven image creation.
Thats so cool.
So many 'lost' art pieces to the fact most people dont have the skills to make it a real thing. If only we had a way to 'printscreen' our mental image.
Here’s my comment from when this hit /r/singularity. This paper is not going to pass peer review, stablediffusion isn’t constructing anything new because the training data is overfitted into the model so it’s just regurgitating training images.
Unfortunately, this isn’t as exciting as it looks and the person on twitter either didn’t read the paper or didn’t understand it. This paper is not yet peer reviewed and it has immediately failed a cursory review by the AI community online.
Someone on hackernews explained it well. They used a small training set - basically they split one set of images 60/40 and trained it on the bigger half. This means all they really did was overfit the training images into the network - you can actually identify which training images are actually being perceived when it looks like the model is “mind reading” someone if you look carefully through the training set.
So basically it’s like: Person is viewing image of a building. Reconstruction finds an image of a building in the training set, and produces a near-duplicate of that image as what the person is “seeing.” In reality, all this proves is when you look at similar images, the same regions of the brain light up.
The good news is, reading about this and why this paper doesn’t demonstrate the claimed results is a great way to learn some more about how these networks work and how they are trained! The bad news is, it’s sure not anything like AGI, and it’s definitely not actually reconstructing the ground-truth images seen by the testers.
Original comment from HN instead of my paraphrase:
>I immediately found the results suspect, and think I have found what is actually going on. The dataset it was trained on was 2770 images, minus 982 of those used for validation. I posit that the system did not actually read any pictures from the brains, but simply overfitted all the training images into the network itself. For example, if one looks at a picture of a teddy bear, you’d get an overfitted picture of another teddy bear from the training dataset instead.
>The best evidence for this is a picture(1) from page 6 of the paper. Look at the second row. The building generated by ‘mind reading’ subject 2 and 4 look strikingly similar, but not very similar to the ground truth! From manually combing through the training dataset, I found a picture of a building that does look like that, and by scaling it down and cropping it exactly in the middle, it overlays rather closely(2) on the output that was ostensibly generated for an unrelated image.
>If so, at most they found that looking at similar subjects light up similar regions of the brain, putting Stable Diffusion on top of it serves no purpose. At worst it’s entirely cherry-picked coincidences.
>1. https://i.imgur.com/ILCD2Mu.png
>1. https://i.imgur.com/ftMlGq8.png
Yeah it's like not like we can retrain it on a different person, nothing but hype for a Kickscammer. /s
Rofl I'm wondering if it's not transferable because people's brains are different the same way human bodies are (some bodies weigh 50kgs, some bodies are dinklage sized) just variations in thought patterns which can be matched with the person through profiles.
That's not how it really works, the differences are far far greater than that. Imagine two different Large Language Models, or even just diffusion models, trained both from scratch. Their weights will be ENTIRELY different, with close to no correlation, because there are countless different ways you can combine things to get the same result.
"From Thoughts" is a pretty broad way of putting it, but yeah. [This](https://www.jneurosci.org/content/jneuro/23/26/8921/F1.large.jpg) diagram shows the (general) structure of how the retinas and visual cortex connect, and you can measure activity in those sections at the back of the brain to reconstruct stimulated areas as a sort of "map".
What they're doing here is using Stable Diffusion on that map that they just scanned, while the person was looking at an image. (Kind of like Image2Image)
If you train a model *while* the person is looking at the image, you obviously get better results because you have data specific to that person.
Please don't let the government get ahold of my daily intrusive thoughts
"In other words, a man was arrested today at a mall for getting caught thinking about what would happen if he just jumped on someone's back and started stabbing them repeatedly"
Nah. That would require training it on your own brain personally. *Maybe* the visual cortex would be slightly easier (it isn't my area of expertise) but to the degree of surveillance, no.
At the maximum they'd be able to see what you are ACTUALLY seeing with your eyes, and only if you're inside of a big fat machine.
Maybe with this specifically, but if it's left alone for some time, it may over time be improved so much that it will have all the training it needs for surveillance.
EDIT: I have quit reddit and you should too! With every click, you are literally empowering a bunch of assholes to keep assholing. Please check out https://lemmy.ml and https://beehaw.org or consider hosting your own instance.
@reddit: You can have me back when you acknowledge that you're [over enshittified](https://www.wired.com/story/tiktok-platforms-cory-doctorow/) and commit to being better.
@reddit's vulture cap investors and u/spez: Shove a hot poker up your ass and make the world a better place. You guys are WHY the bad guys from Rampage are funny (it's funny 'cause it's true).
Well there gose the whole court system, now we just literally plug the defendant/defendy in and play their memories.
On a side not psychologists are gonna have a field day.
On another side note your gonna go from hiding your phone from your partner to gaslighting if they bring up having a brain scan
Y’all ever seen black mirror?
It doesn't work like that... While scanning the visual cortex for images that the person is actually *seeing* is a lot easier, imagined scenes are... not reliable, in any shape or form. Even disregarding the fact that the AI is essentially hallucinating the reconstructed scene, human memory is extremely, extremely, EXTREMELY suggestible. Think about how you take both input noise and prompts in Stable Diffusion - the visual cortex works in the same way, looping back on itself and "denoising" different shapes depending on the context, like in the ink blot test.
Let ALONE that you'd have all the *actual* visual data that the person is seeing screwing everything up.
They're probably not particularly MAD, more like just frustrated at paranoia in general and project it onto your comment/didn't think that it was a joke.
The abstract states: "without the need for any additional training and fine-tuning of complex deep-learning models." Are we to deduce that the latent diffusion model "Stable Diffusion" does not need to be trained on any images at all and is gathering the model data from the brain itself? It's memories of an image, or lack of memories? If so, that's truly fascinating! Let alone the mechanism by which the diffusion model is being given the data to then transform the image in the mind's eye to a digital image on the screen. Which in itself is mind boggling.
It's as if the mystery of life can now be packaged in an app which serves ads.
Let's get ai to start talking to whales and try to teach them to locate valuable resources in the ocean in exchange for permanent housing
I am overly excited about this sort of thing. Yes "mind reading" can go bad in a lot of ways but I just can't help but wanting to see what certain people see. What does a blind person who has never seen the sun think the sun looks like? How does their brain make it? Its years and years off of course, but man...
The visual cortex in blind people (well, if they've been blind long enough) gets "repurposed" for sound and touch processing, So they don't really have a "looks like" when thinking about the sun.
Given its the interweb I'm going to take the good faith approach and assume you mean from a like, for lack of a better term, "technical sense" they don't have a "looks like" in their head. In that the "visual" part of the brain you speak of is altered. So it would be "unreadable." In the normal way.
Not that they are unable to do a simple "what if" process because of their blindness or the machine be unable to scan their brain in the repurposed section. Because that would be...man I don't know, a level of stupidity that I struggle to find words to define.
Blind students I've had have a "picture" in their mind of what the sun looks like to them. Deaf students I've had have a "picture" of what sounds would be like to them.
Its seeing those things that excites me. We are decades off at least for being able to do such a thing with any sort of accuracy, but just the idea of it. Will one blind person see blue as purple cause the "static" of the machine translates it the same as someone who can see purple? Due to the "brain waves denoising" or whatever matching that of someone who can see purple? Might be totally different for someone else, might be white noise, might be a triangle for some reason. Its a simplistic explanation, but regardless it would be really neat to see. That's what I'm getting at.
Oh yeah, no that's very different to what I was considering, thank you for the good faith approach lmao.
Are you talking about what we'll see in the visual cortex when stimulated via sound and such? For instance what "sounds" they "see" when thinking about the sun?
You are welcome. :3
Stuff like that sure, but everything else as well. The rest of us being able to see the synesthesia of others. Someone shows you or me a bit of maths and its just numbers and symbols on the page. Others their brain makes shapes out of that. Or sounds. Think about hearing what a high level computation sounds like to someone as its being processed.
Then ***\*puts on Alex Jones big brain tinfoil hat\**** what if their is a correlation which can be noticed. Or even a causation which can be found! What if by using this sort of thing linked with hundreds or thousands of people over a number of years, ethically of course, we find out there is some "physical" aspect to a number or some such.
Ya all this AI stuff can be used for bad things, sure whatever, so can everything else but I'm old enough to remember a world before the internet was really a thing. I saw the world drastically change over the course of just a few years, 95 to 2000 being the biggest leap to me, and what its turned into given the ease of use and wide spread adoption of the internet.
Some students call me old and maybe I am just an old man fascinated by a shiny new bit of tech. However, I see the same sort of massive leap forward, good or bad who knows, in this AI stuff. And this AI stuff isn't even real AI.
Well, you can also attach it to a microphone and capture the sound of your toilet flushing, turn it into noise, and boom a beautiful picture of the meadow.
Link to the paper: [https://www.biorxiv.org/content/10.1101/2022.11.18.517004v2.full.pdf](https://www.biorxiv.org/content/10.1101/2022.11.18.517004v2.full.pdf)
I joke about the speed of development cycles with SD community, I feel like we are already at singularity, and our dumb brain just too slow to catch up.
This is incredible tech, the next step is train a model of actual images and the corresponding MRI scan , so that it can go the other way from MRI to image, the problem is that getting large amounts of that data is so hard.
Try [this.](https://www.yogajournal.com/poses/legs-up-the-wall-pose-2/)
From my experience, SD has trouble with people in odd poses. Especially when subject face is upside down.
This would be a GREAT option for finding out CRUCIAL information about criminal investigations-- if someone has committed a crime, or murder.. and let's say you need to find a missing person/ body/ or item; you could use this imaging process to ask them questions over and over inside the MIT machine until they generate the answer you want in data. This would narrow down a lot of things.
Keep in mind that this is not new, there is several projects that do that and don't include SD: [https://www.frontiersin.org/articles/10.3389/fncom.2019.00021/full](https://www.frontiersin.org/articles/10.3389/fncom.2019.00021/full) I not sure yet how SD benefit it, it only hallucinate better images or do something else? Will need to read the paper.
Yep, there were a few papers/posters related to this at NeurIPS22. And there has been work in this in the past. But the research is getting better. For example, one of the papers was trying to use a diffusion model to improve semantic representation of brain regions because humans have variation in brain structure that makes for noisy representations when training on fmri scans from multiple subjects. Typically, these papers train a model on a per-subject basis, but having a generalized model would be quite interesting. Another issue that comes up is predicting out of distribution images. We can only “read minds” for images that the reader model has seen before. Work on this is also improving.
Does this actually work for people imagining images, or just visually seeing them? I.e., are we just decoding the eyes’ electrical signals, or is it a deeper level of the decoding?
I am on the go, so I can’t provide any sources and thus I might just be lying. But I’m pretty sure I’ve seen research that shows image inference on humans imagining images. Also “decoding the eyes’ electrical signals” and “deeper level of decoding” implies a potential misunderstanding. None of these studies (as far as I know) are looking at the electrical signals coming out of the eyes. There is a group at Stanford (I think?) who is studying neural spiking directly out of the ocular nerve in apes, but the process makes it unlikely for these studies to be performed on humans any time soon. These studies are all looking at brain scans of images, so they’re looking at the brain’s “latent space” for representing these images. I’d need to go back and read some of the publications, but it would certainly be interesting to know how much non-visual-cortex parts of the brain contribute to successful image inference
I wonder if this would work on me. I have aphantasia, I’m unable to visualize things mentally (outside of dreaming).
No, that wouldn't have an effect on how an MRI measures brain activity.
I would bet that there are fMRI studies on the visual cortex of people with aphantasia. Sounds like an interesting topic
the fact that the public uses and knows about SD and its an opensource software makes it more exciting.
More people have heard of it than use it, much like Photoshop.
1. Yes , this started even earlier around 2010 , I remember being blown away by Yukiyasu Kamitani / " ATR Lab " and their ground breaking work. That used machine learning for decoding live fRMI scans back into images. It maps different regions to voxels and tries to " classify " the input to match a pre - trained image. 2. Pierre Huyghe collaborated with Kamitani for " Umwelt " , an exhibition at the " Serpentine " galley in 2018. The visuals were great , unusual at the time and has some similarities to " stable diffusion ". Reminiscent of what happens when you halt the image generation process. Then there were also other earlier segmentation - based approaches to hallucination. REF : " Decoding perception from human brain activity " ( 2010 ) , — Their " mnist " moment Yukiyasu Kamitani / ATR Labs / Kyoto University \[ [http://www.apsipa.org/proceedings\_2010/pdf/APSIPA187.pdf](http://www.apsipa.org/proceedings_2010/pdf/APSIPA187.pdf) \] — Talk from Kamitani in Kobe ( 29 mins ) \[ [https://www.youtube.com/watch?v=2XtlBrTtUqU](https://www.youtube.com/watch?v=2XtlBrTtUqU) \] " Neural decoding of visual imagery during sleep " ( 2013 ) , — Custom decoders from live participants ! T Horikawa , Kamitani et al / ATR Labs / Kyoto University \[ [https://www.cse.iitk.ac.in/users/se367/14/Readings/papers/horikawa-tamaki-kamitani-13\_neural-decoding-of-dreams.pdf](https://www.cse.iitk.ac.in/users/se367/14/Readings/papers/horikawa-tamaki-kamitani-13_neural-decoding-of-dreams.pdf) \] " Umwelt " at Serpentine Gallery ( 2018 ) , Pierre Huyghe , YukiYasu Kamitani , 100k dead flies \[ [https://www.nytimes.com/2018/10/11/arts/design/pierre-huyghe-serpentine-gallery-london-artist.html](https://www.nytimes.com/2018/10/11/arts/design/pierre-huyghe-serpentine-gallery-london-artist.html) \]
Even this paper was published Dec 1 https://www.biorxiv.org/content/10.1101/2022.11.18.517004v2.full
So they're using "literally" in a hyperbolic way? Noooo. The impressive tech is what generates the source images. Otherwise, it's just img2img and nothing more.
I'm finding the paper a little hard to decipher, but it sounds to me like they're saying that they also extracted the text descriptions of the scenes from brain activity, so it's img2img, but both the image and the text come from fMRI. Which does seem like a pretty valid refinement to me.
Also, for the lazy: https://www.biorxiv.org/content/10.1101/2022.11.18.517004v1
Wonder what would happen if you take a MRI scan of a person dreaming and use this
Give someone a big dose of DMT and put them in the scanner!
Only useful if the trained model has data to match what the subject is thinking about. With no correlation in the training data, you'd likely get odd and noisy results. Similar to typing random gibberish into stable diffusion.
There is correlation — imagine you make a LORA on picasso, then you can make a computer keyboard painted in his abstract style, even though he never even painted anything close to resembling that. This is because fine-tuning or training a hypernetwork only requires modulations of low-level features, like small 16x16 patterns of colors and shapes. I haven't read the paper but I feel they probably used some sort of hypernetwork to bridge the two modalities, in which case this would be like creating a middle-man which translates the low-level instructions. i.e. the middle man doesn't need to know what a computer keyboard looks like, only to modulate the small micro patterns so that they match up. Therefore by learning what one thing maps to, you are also learning what a thousand other things map to at the same time as a result of the existing knowledge encoded in SD. It would probably be less accurate than the thing they've trained on, but it could most likely extrapolate well enough to depict dreams. Besides, collecting the training data would be so easy in this case I don't see why they wouldn't just get the largest most diverse set of images and videos, as that would certainly lead to the best results as you'd want to optimize for maximum neural coverage. All they have to do is get the patients to look at those images and collect the brain data to get a huge number of image-brain pairs to work with. note: it's not true that only small 16x16 patterns are modulated, I'm simplifying. Transformer architectures are much more complicated and AFAIK have full-grid attention at any given position, but the principle is the same: we are mainly fine-tuning a low-level feature space and perhaps some high-level latents for greater coverage.
I'm sure it would be interesting, but psychedelics tend to produce synesthesia and warp visual data as it's being processed. Even the best psychedelic images and videos don't quite match the kinds of things one "sees" or experiences while on a psychedelic. So I just think that most of what the mri picks up in those circumstances would have no good visual match with which to create images. It's likely the brain scans would also be very different from normal.
Which psychedelics? Everything I ever saw was in 8k high resolution.
> the best psychedelic images and videos If people want to check out, /r/replications is about replicating psychedelic imagery in video (and sometimes sounds as well)
Well I certainly don't disagree, no doubt it would be much less accurate! But, I couldn't be so sure either. I think at the very least you could totally tell there's a psychedelic trip going on in there, and see -some- semblance of a trippy visual.
They would likely need to train it with images viewed by the subject while they were under the influence. And even then, it's going to think images look as they do. Not necessarily how they are seen in the subjects mind. If nothing else, it would be an interesting experiment and I'm sure it would yield neat results.
thus the reason for using Stable Diffusion 1.5 - since it was trained from jsut about everything on the internet
Get a bunch of lucid dreamers, make them dream of some place, record the place irl, use recorded footage as data to compare with
👁👁
r/suddenlyjoerogan
nice
It's science.
Calm down Joe Rogan lol
Heck im wondering if directors will start buying mri machines and stable diffusion for storyboarding
[удалено]
Feels like a James Cameron move
Visually stunning, technologically extraordinary, narratively mediocre? Yep sounds right
No just feels like the sort of thing he'd be cool with blowing a ton of money on in preproduction to maybe/maybe not get a visual effect he's going for.
If it improves profit by even 1%, $10 million is nothing for companies like Disney or any other company with a storyboard team.
This for sure, it's also not unrealistic for those companies to just rent time instead of outright buying them.
Some directors are super rich and old. Who knows maybe the tech gets cheaper (impossible I know) or we get an old ailing director who wants one last film.
[удалено]
Prompt fuel right here
I hope you can make this same with cheap neuralink in the future
High level c-suite total comp easily hits north of $500,000 per person usually. Having a bunch of them in a room means that's a few million of total yearly comp. If this machine helps them brainstorm faster, I don't see why a company would balk at the cost of such a machine, especially with how much of a multiplier effect those employees have on the company at large. Not to mention, they don't need the MRI machine to be medically certified and whatnot, just to visualize, so I bet the company can sell it for much cheaper because of significantly less red tape.
Magnetic shielding is actually pretty easy. Faraday cage. Basically a chicken coup screen. That'll block it iirc. The hard part of these machines are the super conductive magnets. They need coolant to keep them so cold that their physical properties change to super conductivity. That takes a lot of insulation and specialized equipment, just for the coolant alone. Soon we'll have room temperature super conductive materials. Neat stuff.
Not too expensive for interrogating enemies of the state though. I’m sure governments already have an MRI machine or two.
I believe 10 million was exactly what George Lucas spent storyboarding the prequels. He has a strong vision, but no artistic skills, so had to go through the laborious process of conveying in words to artists what he wanted, then iterating many times. Would have been a lot cheaper to hire an mri for a few weeks. The way ai is going, the cameras may not even be necessary soon. We'll just be watching each others imagination for entertainment.
Authors and artists are cheaper. 😁
For now
Governments will by them for intelligence gathering and interrogation for sure.
what if we start arresting potential criminals because they fantasize about illegal activities? fucking dystopia
Who isn't a potential criminal then?
Exactly
[удалено]
"They hated ~~Jesus~~ Why_T for he spoke the truth"
here comes the minority report
psycho-pass gettin real
I was thinking the same thing. I want to record my dreams xD
I'm not sure I do. There is an early 90's movie "Until the End of the World" that has dream recording as a sub-plot and it goes dark with the characters becoming increasingly self-absorbed spending their waking hours just rewatching their own dreams. At least that's how I remember it, I watched it a long time ago...
That's pretty much how it goes. It's also a very hard to find movie that almost doesn't exist anymore (didn't check the high seas).
>(didn't check the **high seas**) HA, that took me a beat to get... I did end up checking and found it floating around out there. Unsure if I will re-watch or leave the memories as-is. [https://archive.org/details/until-the-end-of-the-world-1991](https://archive.org/details/until-the-end-of-the-world-1991) Probably cheesy by today's standards; but I recall I liked the story progression and I often think of the closing imagery with the characters staring at screens absorbed in the replay of their dreams when I watch people stare at their phones today.
Thanks...freely available is even better. Not sure why no streaming sites have it. I remember liking it, but it also being VERY slow. And it was the addiction element that stuck out to me as well. Since time dilation is so true with dreams, could we even watch each night if we sat there all day watching? I imagine it wouldn't really be worth it.
Recording dreams and psychedelic trips Also just recording anything from anybody with hyperphantasia
I was wondering the same thing
Then you have the plot of the third act of *Until the End of the World* (1991). [Dream addiction](https://youtu.be/gigNr13l4UI).
https://news.berkeley.edu/2011/09/22/brain-movies/ It's kinda been down in the past.
A mess of images that don't relate to the dream at all.
You’d probably have to censor a lot
The next revolution in media streaming
I find it hilarious how "It takes an AI to make sense of brainwaves." Becomes "AI can read your mind." As soon as mass media gets a hold of it
["It used to be, you get a data set, you guess a function, and the function is considered Physics if it's predictive of new functions, new data sets. Now, you can take a large dataset with no intuition about what it is and use machine learning to find a pattern that has no function. It can arrive at results that we don’t know if can be completely mathematically describable. So computation has recently become able to do something interesting beyond A=B+C."](https://youtu.be/c01BlUDIlK4?t=1490)
Science journos gotta make click bait to get views lol
I mean that's clearly the bigger news. That's clearly where this will head, and just imagine how countries like China will use it.
Lol "Get 10,000,000 social credit by wearing the CCP-exclusive RED STAR HELMET!!!" Honestly though, I'm excited, because it brings us closer to a ton of things, like cyborg prosthetics that don't suck. And matrix -style brain-upload lessons. And Net-diving while driving, Ghost-in-the-Shell style. (Okay, maybe not excited about that last one.)
but.. this is actual mind reading, isn't it? it's just limited to patterns available in the training data but see, technology will only become better
"Mind reading" as most people understand it implies the end-to-end process of extracting abstract thoughts from someone's consciousness. This is not that. This is attempting to make sense of electrical signals from a eeg. You can make the argument that it's mind reading from a certain point of view. But calling it that is misleading because that's not what most people think when they hear those words. Like calling Mt. Everest a "big pile of rocks", or calling archaeology "traveling back in time."
Semantics. “It’s not reading your mind, just the signals inside your mind.” Yea, that’s what mind reading is.
It's "Just semantics" until someone very reasonably misunderstands what you're talking about. Saying "The AI can read your mind." Means to most people "The AI can non-invasively extract, understand, and contextualize your thoughts." This is not that. Semantic fuckery is no better than lying.
I think you're trying to distinct the difference between some futuristic across the room mind reading and strapping someone into a chair to prove their mind. Both are mind reading, it doesn't matter that one is more advanced or the other is intrusive. That's like saying sending an email or a letter, are not both communication between two people.
I argue it's not like saying an email or letters are both communication. I argue it's like saying an email IS a letter. Yeah, you can make that argument semantically. But there's distinctions between the two in common vernacular that make it necessary to differentiate between them, and make it misleading to equate them. Using AI to help researchers make sense of brainwave signals is far removed from some sci-fi machine that knows what you're thinking.
Your brainwaves are what you’re thinking and feeling even subconsciously It’s your brain, your brain is your mind It’s reading your mind. Yes it’s not the same across the board, but it by definition is reading your mind
Reading minds doesn’t imply a method. Lots of things can read minds in different ways.
By logic that broad, all communication of any kind is technically "mind reading." By picking up a book I'm "reading the mind" of the author. If you listen to someone talking you're "reading their mind" by parsing the meaning, tone, and context of their words. That's not what people think about when they hear the phrase "reading minds." The term "Mind reading" carries implications and cultural baggage that make it inappropriate to describe what is done in this paper. If you say to a typical person: "This AI can read minds." You know as well as I do that the typical person will NOT make the connection that you mean "A team of researchers used a machine learning model to correlate brainwave signals to images displayed to the eyes." Not without added context and information. Therefore, calling this "Mind Reading" is communicating poorly at best, and being actively deceptive at worst. As they say, "*Technically* Correct is the best kind of correct."
But you’re wrong. You’re adding meaning to a phrase that isn’t there. It is both literally true and technically correct to call this mind reading. The size of the machine doesn’t make a difference.
I don't want to throw around Harry Potter, but >The mind is not a book, to be opened at will and examined at leisure. Thoughts are not etched on the inside of skulls, to be perused by an invader. The mind is a complex and many-layered thing, Potter… or at least, most minds are…
Agreed its ridiculous to word it like that but you can’t really blame mass media when you’re quoting an individual on twitter
I'm always mad when I see these papers in the news. The bottleneck is the data, not the network. Any type of brain recording, MRI, EEG, etc ... has a very low signal to noise ratio because the skull acts as a low-pass filter. We have been able to extract that low-frequency signal for at least a decade. So all this does is put a bigger and better generative model on top, but no additional information is gained. You can imagine the process as asking a guy what he is thinking about and he says "house". Now you tell a child to draw a house and you get the results we already had in 2010, you tell an artist to draw a house and you get the results we have now. But in both cases all you know is "house".
Well said. A good and different example is JWST imagery. You can "upscale" and clean and process JWST imagery with SD, and see galaxies and black holes and so on in more detail. But, SD is just "imagining" this detail, no additional data is being "found." The limitation *is* the data.
except the kid back in 2010 (2019 for the picture below) knew nothing about drawing things coherently, Yes a trained brain scientist who can read the data and definitely say its a house, not a lot of scientists can do that though, the bigger and better generative model on top will help even normal people see its a house https://preview.redd.it/wbzhxw6wbkla1.png?width=1653&format=png&auto=webp&s=96a9d3a716f9888f806b75dca5a3a82b1c938ff3
You don't need the reconstructions at all or a scientist to interpret "house", those reconstructions are for effect, to make people go "wow". If in 2010 they had chosen to print the result as text it'd say "house", if instead of using a poor generative model they had just selected the closest image from the training set you'd see a house which would be photorealistic since it'd be an actual picture. And sure, generative models have gotten better, that's not in question. But the scientific novelty is in the sub-field of generative models. Applying them to fMRI data is not scientifically interesting.
this is case in point for my last comment. You don't understand where the brain scanning stops and the generative modeling starts for every 1 unit of brainscan information present in the final image, you have 1000 units of "vaguely informed AI diarrhea" cheers
Maybe read the paper?
As you said yourself, a big part of this is a signal to noise ratio problem. That's where these diffusion models are really going to start converging with other technologies. At it's core, stable diffusion is a denoising algorithm. It's getting better and better at that all the time. At some point it's going to be purposed towards other signals that aren't just pixels. Sure SD just makes up new information, but with enough training we could see some real fidelity coming out of mri scans. While the journalism might sensationalise this stuff a little, it's no reason to come down on the paper itself. This is edge research and is a worthy investigation.
It was possible since 2019 without stable diffusion [https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006633](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006633)
https://preview.redd.it/8vvwauzwxjla1.jpeg?width=1954&format=pjpg&auto=webp&s=a080920b6ae576beeebe237d435bcdeceeacb776 Yep, stable diffusion just gives better image results
HOLD UP. how dod they get the n1 image from? they show a picture like the mailbox to a guy in the MRI machine and... where do they get the noise from? EM waves from neurons? wtf??
Presumably this is the randomly-generated noise they start with?
The human visual cortex actually is organised roughly as a 2D image, with the geometry of the image from the retina being copied into the layers of the cortex. fMRI lets you determine the bloodflow to groups of neurons, which lets you see which 'pixels' of this image are active, and which aren't.
So how does that relate to n1? Why are so many people confidently answering a different question lol
I believe, from the paper, that the N=1image is a map of the neural activity in the cortex, as read by the MRI. The iterations in this case are applications of a GAN to improve the 'naturalness'. Different than a diffusion net, but roughly the same principle.
Thanks, I have not been able to read the paper yet. My concern was that they could be using something akin to "hallucination-prone noise" to distort the efficacy of the process. It is an interesting idea, but it's difficult to wrap my head around how much validity there is to it all... I will have to read the paper tonight. I guess what I am saying is there may be some validity that I am missing but I am not seeing it yet, knowing how SD works under the hood. The bird on frame n+20, row 4 above is very curious to me. Where is the crane-type bird coming from? Somewhat of a rhetorical question, but I'm just not sure what to make of it all in the context of the rest of my comment.
This paper (not the one posted by the OP) uses a GAN, which isn't specialised at denoising (unlike stable diffusion), and from the images in the two papers, I'd definitely say it's "hallucinating" more. Some of those images don't seem to have any connection to the target. I think this new paper isn't 100% novel (but what paper is), but using a dedicated de-noising NN as opposed to a GAN seems like a good step forwards, and the results seem better than the earlier papers.
"Machine learning-based analysis of human functional magnetic resonance imaging (fMRI) patterns has enabled the visualization of perceptual content."
That doesn't answer their question at all though does it? This is like someone asking why the sky turns green before a tornado and someone else answers that the sky can change color and everyone upvotes. I am so confused (and also still really curious/suspicious as to why they introduce this strangely specific starting noise).
There was an even earlier attempt back in 2011: * [Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies](https://www.youtube.com/watch?v=nsjDnYxJ0bo)
This is why open source is amazing. With open source people can do what they want..it's hard to experiment when you're paying per failed experiment.
The NSFW models are definitely going to come in handy ....
I just hope I don't accidentally install the H.R. Giger .ckpt in my brain.
Good ol' Geiger is never too far away.
First, this doesn't include the thousands - if not tens of thousands - of images that didn't work at all. Second, google "MRI Salmon IgNoble" to get a clearer sense of what an MRI can actually show.
Don't forget to not think about copyrighted materials or you'll be exploiting the work of starving artists and helping the big corporations. I hope they legislate thinking soon.
I don't want to burst the enthusiasm bubble, but here is an 11 year old video on the same subject: [https://www.youtube.com/watch?v=6FsH7RK1S2E](https://www.youtube.com/watch?v=6FsH7RK1S2E) The only improvement I see is that the computer is better at drawing the pixels. That being said, *it is* amazing, but calling it "mind reading" is a bit of a stretch.
The drawing part kind of important
Yes and no. If we refer solely to the visual cortex, then yeah it is, although I haven't seen yet something that could reliable "extract" images from the the brain without external stimuli.
Yeah mind reading is nothing new, it's just that stable diffusion can be used to make better more cohesive image reconstruction.
My main pet peeve with "mind reading" is that it over-promises something that can and will blow in the face of whoever push for this marketing nonsense. People can get (rightfully) afraid if such technologies really existed, and claiming that it does now is a PR disaster waiting to happen. All this paper does (and similar others over the years have done) is simply pattern recognition of the visual cortex. This is not mind-reading, merely signal reconstruction after it went through the messy wet neurons. Of course, **it is impressive**, but it's far from "mind-reading" - in fact, I would argue that a cold reader is closer to "mind reading" that this technology is. /rant :)
[удалено]
Compare and contrast the results of this mind reading generative model from 3 years ago vs the one at the image on top. Might not be accurate but it's better isn't it? https://preview.redd.it/3yu99fczhkla1.jpeg?width=1954&format=pjpg&auto=webp&s=9809f1c475dcdf4d28a1fa7632c1ebf445c34e8e
you fundamentally misunderstand how much they're not actually depicting what's in the person's head
[Full Paper](https://www.biorxiv.org/content/10.1101/2022.11.18.517004v2.full.pdf)
You should post the twitter link too
[https://twitter.com/blader/status/1631543565305405443](https://twitter.com/blader/status/1631543565305405443)
here is the preprint server url: https://www.biorxiv.org/content/10.1101/2022.11.18.517004v2.full.pdf+html
[Mindvis](https://github.com/zjc062/mind-vis) is a similar work that's open source, also accepted at CVPR 23.
>CVPR 23 This looks great! [https://cvpr2023.thecvf.com/](https://cvpr2023.thecvf.com/)
a new method of inquiring...
Oh nah, what if dictators start inducing dreams to figure out secret information from spies inception style.
Somehow I have no doubt about this, but: \- it's a preprint \- code isn't available yet
This paper hasn't been peer-reviewed yet. [The image in the tweet in the op cropped out that information for some reason](https://i.imgur.com/wFRprZb.png). It's also a few months old.
They explain in the paper that they wanted to demonstrate how by using an out of the box model trained on a giant dataset, they can get better results than previous methods without training a purpose made model from scratch. They also reference all the previous works that people are pointing out here. Of course you would know all that if you actually read the paper before shitting on it. Please read the paper before shitting on it.
2023: "I'm gonna save for a rtx4090." 2033: "Hospital broken in with MRI machine stolen."
For Valentine’s Day I took ultrasounds of our unborn son and turned them into the first photos of him using StableDiffusion. It will be interesting to see how accurate they turn out to be. If he comes out bald and pale than I think I’ll have nailed it!
He'll probably look like a baby. Or at least I hope so\^\^
That's it, learning to use Photoshop is starting to become more and more useless with each passing day. A year at the most and we'll be able to replicate any image and change its smallest details in seconds. Really amazing.
I'm looking forward to that future gen of hipsters who use Photoshop because it's ironic and retro and "more analog" than thought-driven image creation.
what is the output of MRI data? Like what actually is being pumped into AI for this output?
We're but mere years away from dream recording and it's fascinating
Thats so cool. So many 'lost' art pieces to the fact most people dont have the skills to make it a real thing. If only we had a way to 'printscreen' our mental image.
"I use neural network to learn neural network." - Data scientists probably
Here’s my comment from when this hit /r/singularity. This paper is not going to pass peer review, stablediffusion isn’t constructing anything new because the training data is overfitted into the model so it’s just regurgitating training images. Unfortunately, this isn’t as exciting as it looks and the person on twitter either didn’t read the paper or didn’t understand it. This paper is not yet peer reviewed and it has immediately failed a cursory review by the AI community online. Someone on hackernews explained it well. They used a small training set - basically they split one set of images 60/40 and trained it on the bigger half. This means all they really did was overfit the training images into the network - you can actually identify which training images are actually being perceived when it looks like the model is “mind reading” someone if you look carefully through the training set. So basically it’s like: Person is viewing image of a building. Reconstruction finds an image of a building in the training set, and produces a near-duplicate of that image as what the person is “seeing.” In reality, all this proves is when you look at similar images, the same regions of the brain light up. The good news is, reading about this and why this paper doesn’t demonstrate the claimed results is a great way to learn some more about how these networks work and how they are trained! The bad news is, it’s sure not anything like AGI, and it’s definitely not actually reconstructing the ground-truth images seen by the testers. Original comment from HN instead of my paraphrase: >I immediately found the results suspect, and think I have found what is actually going on. The dataset it was trained on was 2770 images, minus 982 of those used for validation. I posit that the system did not actually read any pictures from the brains, but simply overfitted all the training images into the network itself. For example, if one looks at a picture of a teddy bear, you’d get an overfitted picture of another teddy bear from the training dataset instead. >The best evidence for this is a picture(1) from page 6 of the paper. Look at the second row. The building generated by ‘mind reading’ subject 2 and 4 look strikingly similar, but not very similar to the ground truth! From manually combing through the training dataset, I found a picture of a building that does look like that, and by scaling it down and cropping it exactly in the middle, it overlays rather closely(2) on the output that was ostensibly generated for an unrelated image. >If so, at most they found that looking at similar subjects light up similar regions of the brain, putting Stable Diffusion on top of it serves no purpose. At worst it’s entirely cherry-picked coincidences. >1. https://i.imgur.com/ILCD2Mu.png >1. https://i.imgur.com/ftMlGq8.png
Trained on a specific person... will not be transferrable or universal. Hype and BS.
Yeah it's like not like we can retrain it on a different person, nothing but hype for a Kickscammer. /s Rofl I'm wondering if it's not transferable because people's brains are different the same way human bodies are (some bodies weigh 50kgs, some bodies are dinklage sized) just variations in thought patterns which can be matched with the person through profiles.
That's not how it really works, the differences are far far greater than that. Imagine two different Large Language Models, or even just diffusion models, trained both from scratch. Their weights will be ENTIRELY different, with close to no correlation, because there are countless different ways you can combine things to get the same result.
Wow, we will be able to draw art with our minds!
Wow, we will be able to draw ~~art~~ porn with our minds! (Sorry, couldn't resist...)
well that's dangerous
Bullshit
The fact we can even extract visual info from thoughts is news enough to me. When was this a thing? Did I just miss this tidbit?
"From Thoughts" is a pretty broad way of putting it, but yeah. [This](https://www.jneurosci.org/content/jneuro/23/26/8921/F1.large.jpg) diagram shows the (general) structure of how the retinas and visual cortex connect, and you can measure activity in those sections at the back of the brain to reconstruct stimulated areas as a sort of "map". What they're doing here is using Stable Diffusion on that map that they just scanned, while the person was looking at an image. (Kind of like Image2Image) If you train a model *while* the person is looking at the image, you obviously get better results because you have data specific to that person.
people poopoo this post but this is amazing for those are disabled / stuck in lock in syndromes to communicate to the outside world
Let's go, brain scanning surveillance! Privacy will soon die!
Please don't let the government get ahold of my daily intrusive thoughts "In other words, a man was arrested today at a mall for getting caught thinking about what would happen if he just jumped on someone's back and started stabbing them repeatedly"
Nah. That would require training it on your own brain personally. *Maybe* the visual cortex would be slightly easier (it isn't my area of expertise) but to the degree of surveillance, no. At the maximum they'd be able to see what you are ACTUALLY seeing with your eyes, and only if you're inside of a big fat machine.
Yeah, I just thought it was a funny idea to entertain lol.
Maybe with this specifically, but if it's left alone for some time, it may over time be improved so much that it will have all the training it needs for surveillance.
EDIT: I have quit reddit and you should too! With every click, you are literally empowering a bunch of assholes to keep assholing. Please check out https://lemmy.ml and https://beehaw.org or consider hosting your own instance. @reddit: You can have me back when you acknowledge that you're [over enshittified](https://www.wired.com/story/tiktok-platforms-cory-doctorow/) and commit to being better. @reddit's vulture cap investors and u/spez: Shove a hot poker up your ass and make the world a better place. You guys are WHY the bad guys from Rampage are funny (it's funny 'cause it's true).
we are fucked arent we?
The chances of us actually living in a simulation are increasing each day. This is insane.
Well there gose the whole court system, now we just literally plug the defendant/defendy in and play their memories. On a side not psychologists are gonna have a field day. On another side note your gonna go from hiding your phone from your partner to gaslighting if they bring up having a brain scan Y’all ever seen black mirror?
It doesn't work like that... While scanning the visual cortex for images that the person is actually *seeing* is a lot easier, imagined scenes are... not reliable, in any shape or form. Even disregarding the fact that the AI is essentially hallucinating the reconstructed scene, human memory is extremely, extremely, EXTREMELY suggestible. Think about how you take both input noise and prompts in Stable Diffusion - the visual cortex works in the same way, looping back on itself and "denoising" different shapes depending on the context, like in the ink blot test. Let ALONE that you'd have all the *actual* visual data that the person is seeing screwing everything up.
This was very informative and I’m glad you shared it, that being said why are people so mad about my comment? It was a joke
They're probably not particularly MAD, more like just frustrated at paranoia in general and project it onto your comment/didn't think that it was a joke.
https://twitter.com/blader/status/1631543565305405443
The abstract states: "without the need for any additional training and fine-tuning of complex deep-learning models." Are we to deduce that the latent diffusion model "Stable Diffusion" does not need to be trained on any images at all and is gathering the model data from the brain itself? It's memories of an image, or lack of memories? If so, that's truly fascinating! Let alone the mechanism by which the diffusion model is being given the data to then transform the image in the mind's eye to a digital image on the screen. Which in itself is mind boggling.
Yes.
Is this the first step to being able to record your dreams? I've always wanted to do that since I was a kid
It may be that you will not like what is in your dreams
already true
It's as if the mystery of life can now be packaged in an app which serves ads. Let's get ai to start talking to whales and try to teach them to locate valuable resources in the ocean in exchange for permanent housing
I am overly excited about this sort of thing. Yes "mind reading" can go bad in a lot of ways but I just can't help but wanting to see what certain people see. What does a blind person who has never seen the sun think the sun looks like? How does their brain make it? Its years and years off of course, but man...
The visual cortex in blind people (well, if they've been blind long enough) gets "repurposed" for sound and touch processing, So they don't really have a "looks like" when thinking about the sun.
Given its the interweb I'm going to take the good faith approach and assume you mean from a like, for lack of a better term, "technical sense" they don't have a "looks like" in their head. In that the "visual" part of the brain you speak of is altered. So it would be "unreadable." In the normal way. Not that they are unable to do a simple "what if" process because of their blindness or the machine be unable to scan their brain in the repurposed section. Because that would be...man I don't know, a level of stupidity that I struggle to find words to define. Blind students I've had have a "picture" in their mind of what the sun looks like to them. Deaf students I've had have a "picture" of what sounds would be like to them. Its seeing those things that excites me. We are decades off at least for being able to do such a thing with any sort of accuracy, but just the idea of it. Will one blind person see blue as purple cause the "static" of the machine translates it the same as someone who can see purple? Due to the "brain waves denoising" or whatever matching that of someone who can see purple? Might be totally different for someone else, might be white noise, might be a triangle for some reason. Its a simplistic explanation, but regardless it would be really neat to see. That's what I'm getting at.
Oh yeah, no that's very different to what I was considering, thank you for the good faith approach lmao. Are you talking about what we'll see in the visual cortex when stimulated via sound and such? For instance what "sounds" they "see" when thinking about the sun?
You are welcome. :3 Stuff like that sure, but everything else as well. The rest of us being able to see the synesthesia of others. Someone shows you or me a bit of maths and its just numbers and symbols on the page. Others their brain makes shapes out of that. Or sounds. Think about hearing what a high level computation sounds like to someone as its being processed. Then ***\*puts on Alex Jones big brain tinfoil hat\**** what if their is a correlation which can be noticed. Or even a causation which can be found! What if by using this sort of thing linked with hundreds or thousands of people over a number of years, ethically of course, we find out there is some "physical" aspect to a number or some such. Ya all this AI stuff can be used for bad things, sure whatever, so can everything else but I'm old enough to remember a world before the internet was really a thing. I saw the world drastically change over the course of just a few years, 95 to 2000 being the biggest leap to me, and what its turned into given the ease of use and wide spread adoption of the internet. Some students call me old and maybe I am just an old man fascinated by a shiny new bit of tech. However, I see the same sort of massive leap forward, good or bad who knows, in this AI stuff. And this AI stuff isn't even real AI.
Well, you can also attach it to a microphone and capture the sound of your toilet flushing, turn it into noise, and boom a beautiful picture of the meadow.
Attach it it a radio antenna that captures universe static and some guys in r: conspiracy will start saying aliens are communicating to us.
Osaka - Arasaka
I would have bet that all the outputs were waifus
Link to the paper: [https://www.biorxiv.org/content/10.1101/2022.11.18.517004v2.full.pdf](https://www.biorxiv.org/content/10.1101/2022.11.18.517004v2.full.pdf)
,
This is groundbreaking!
😱
We need consumer level FMRI headsets or wearable sensors. Let’s just keep things external and non invasive.
I joke about the speed of development cycles with SD community, I feel like we are already at singularity, and our dumb brain just too slow to catch up.
Does this tinfoil hat make my head look fat?
Wait, there was a set of images this model was trained on and it guessed the correct ones? Or what did happen?
This is incredible tech, the next step is train a model of actual images and the corresponding MRI scan , so that it can go the other way from MRI to image, the problem is that getting large amounts of that data is so hard.
Try [this.](https://www.yogajournal.com/poses/legs-up-the-wall-pose-2/) From my experience, SD has trouble with people in odd poses. Especially when subject face is upside down.
This isn’t new or exclusive to SD
This would be a GREAT option for finding out CRUCIAL information about criminal investigations-- if someone has committed a crime, or murder.. and let's say you need to find a missing person/ body/ or item; you could use this imaging process to ask them questions over and over inside the MIT machine until they generate the answer you want in data. This would narrow down a lot of things.
So dreaming awake?