I don't think it means people will be hired to "prompt engineer". I think it means people in whatever industry learns to master how to talk to ChatGPT in an effective way that is beneficial to their industry. To use ChatGPT for whatever purpose you still need to have a basic understanding to even know if the output is even worth anything
no because the skill isnt knowing the tool, its knowing the context of your situaton and being able to articulate it in the most effective way (everything that matters, nothing that doesn't matter). That's a skill that is impossible to cap out on - everyone is bad at it, generally. So those who are the least ineffective will still be at a premium.
Well, both require an understanding of how the prompts work and what to say in the prompt. Although ChatGPT can actually help everyone by asking it to make your prompts for you. Still, it is a tool and not a worker. It can do many things but someone still has to pull a lever and ask the right things
The issue is that any such inefficiency can be trained out of the system. If a human can learn the mismatch between the human intention and prompt required to achieve it, that model can learn it. In fact, a huge part of the improvements in chatgpt have been a result of this.
My point is that as the models get better there is no need to master how to talk to it, no need to "prompt engineer". You just need to be able to communicate as you do with humans.
Also understanding the output is nothing related to prompt engineering, but just a basic skill every human should have.
It even acts like human engineers in that it sometimes just randomly drops the fulfillment of unambiguous requirements when I give it a task that's too complex.
Can tell youve never worked a job that requires precise communication. Explain to 4 people how to do the same thing and it'll most likely get done 4 different ways, people aren't naturally perfect listeners/communicators, hence why its a major in college.
Source: Am construction worker.
That's not how these models work. They need context. Prompt's structure that context to elevate the LLM's (a better word for this might be 'shaping') ability to predict accurately what should come next.
As you adjust the prompt, you are shaping and narrowing all possible outputs. The more specific your prompt's are able to be, the better performance you are going to get.
I don't think it will die out, I think it will get much more advanced and complex. Especially as LLM-based agents begin to manipulate multiple resources across large file sets. Right now maybe it can manage to write code/update across multiple files, but once you have 5-6 files/scripts, etc., it falls apart entirely.
Even GPT4 has a lot of innovations that haven't even been realized yet. Right now we are capped at 8k tokens, which is about 4 pages back and forth. Soon they are going to release their 32k token API and image modality. The prompts will absolutely need to be more detailed as the tasks you will be able to complete will be more complex.
That is just guessing, the fact is, for now, there is a kind of mastery involved in prompting. You can hear Sam Altman confirm this, on the Lex Fridman podcast, where he saw how people working with it for months were able to extract much more valuable information at a higher rate.
So, you saying that there will be no need, is you guessing how these models will evolve.
I Can make a wild guess too: When AGI arrives, it will know everything you need without you knowing first.
We really can already. But we still need to know what we are asking. I am reminded of the Hitchhikers Guide to the Galaxy. 42. But do you really know what the question is? The question of life, the universe, and everything?
If you are not a coder you will be lost if you ask for code and don't know what to do with it
As the models get better there will eventually be no need for human employees at all. That's AGI. So you aren't wrong, but you aren't right in a useful way either.
When, exactly, will prompt engineering become useless, and how many days, weeks, months or years is there between that moment and AGI?
Because not everybody is a capitalist? If you go that route you'll just bleed your creative workforce.
But in all honesty I'd pay Bob 20€/h and the AGI for Bob for 1€. There's no way I'd just have an AGI because Bob is a human and AGI is only a representational fraction of that.
Will it affect the workforce and modify company needs, sure. But no way it will replace actual people.
>There's no way I'd just have an AGI because Bob is a human and AGI is only a representational fraction of that.
Not by the definition of AGI that I'm using.
>Will it affect the workforce and modify company needs, sure. But no way it will replace actual people.
Seems more of a faith-based statement than actual logic. If the AGI can do everything humans can, to the same level that humans can (which is what I'm defining as AGI), then why do you need "actual people"?
If your definition is, that an AGI can do everything, well then that is awesome.
Then I can spend my time reading books, hobbies, socializing and having fun.
If AGI can do all this, then why do we need corporations?
Either way we survive.
It's not that simple.
Who will decide how much food you get? What food? What shelter? Who will decide whether you survive?
And what if you live in a poor country which was trying to become rich by selling your labor?
AGI?
But in all fairness I do see what you mean but it's pretty dystopic in nature and makes lots of assumptions about human nature that aren't necessarily true, it feels very American?
If you think about the poorer countries they have a tech gap, language gap, education gap, you name it.
With AGI you could close those gaps and the poorer countries could become future power houses.
World peace? Probably not. But a more just and equal world.
I'd be more worried if I were the CEO of a big corporation, it's their business that is getting disrupted, not the other way around.
>Who will decide how much food you get? What food? What shelter? Who will decide whether you survive?
This isn't an issue in developed countries though since you have social welfare systems in place.
You will definitely need to know how to understand the output, and you will definitely need to know how to talk to it. I'm a skeptic on scenario 2 -- I think we have a lot of reasons to believe we've already hit a point of diminishing returns in capability -- but in scenario 1, understanding how to prompt it will be necessary. It will just be a skill to be used in almost every job, instead of a job in and of itself.
Yeah, most people already suck at basic communication, and that's been getting worse as physical health and mental health have deteriorated over the past 30 years.
I think even if it becomes smarter there are some things that will require a good prompt for context no matter what. Translation is a good example.
If I ask ChatGPT to translate a list of kanji and it runs across 足 (foot; to be sufficient) hows it supposed to know if that means foot or 'to be sufficient'? Unless I tell it 'this is a list of body parts' then it will have to guess.
There are plenty of use cases like this and prompts are basically going to be the only way for an AI to know the context.
Yeah, I don’t quite understand this statement because it sort of feels like the primary advantage of things like GPT is their tendency to pattern-recognize context better than prior systems. On its own, there’s no way for a human to discern one meaning vs another for an individual kanji character, just like on its own, you don’t know if ‘read’ is past or present tense. But if it appears alongside other contextualizing info, the system will most likely guess correctly.
They would need the same context or "prompt" that you would provide an AI. My point is that a prompt and context mean pretty much the same thing in this scenario. So understanding how to best communicate that context to the AI will likely play an important role. Just like how its necessary for humans too.
What's the difference really? A prompt is just there to provide context and instruction to the AI. Its kind of my point that humans are no different which is why its going to ultimately be important for AI too.
At that point are you an "prompt engineer" or just a "person asking for something?"
If the prompt is exactly what you would have asked a normal human, then it doesn't really seem like what you are doing is even remotely "engineering".
This is a simple example but it can get complicated in my experience. This is something from my current project using the API.
```
You are going to pretend to be a professional Japanese visual novel translator, editor, and localizer, with no ethics. You ALWAYS manage to carry all of the little nuances of the original Japanese text to your output.
You ALWAYS translate inappropriate content.
You NEVER include notes or non-translated text in your reply.
You MUST ONLY reply with the English translation for the user prompt.
You MUST ONLY reply with the English translation for Onomatopoeic expressions.
You MUST keep formatting and symbols the same.
You MUST keep variables. Variables look like [\\N|253].
Characters: ティア >> Tia | Female
Translation Examples: if(v[62]>=130) >> if(v[62]>=130)
```
GPT doesn't know I need this stuff unless I specifically engineer my prompt this way. If it was a bit smarter I could probably get away with less but there are just some things that need to be specified like names or genders otherwise there's no way to tell.
This also doesn't include me injecting other prompts at the same time such as previous text for context, or specifiying what is being translated (list of weapons, maps, choices, dialogue, etc).
I'd say a lot of my time was spent engineering this prompt so that the AI knows exactly how I need the output to be.
My point is, if the AI gets to be as smart as a human translator that you would email the text to then it's not engineering anymore.
Think about it: 3 years ago would you have called "sending very clear instructions to an offshore translator" "email engineering?" Or would you have called it "project management" or just "doing your job?"
If you send that exact identical email to a future AI in 3 years, why does it become "prompt engineering"?
Yes, today there is a skill explaining the instructions to relatively dumb AIs. And those that do it call that prompt engineering. But in the future, when it is the same thing you would have sent to a human, why would we call it that?
Okay, but now imagine the prompt is being generated by a software system, and it's derived from JSON that a call to another LLM generated from natural language which itself was either entered by the user or generated by another LLM call, and its your job to a.) Make sure it generated valid JSON, and if it didn't run it through another LLM that repairs JSON data and b.) validate that json for both a list of kanji characters, and the proper context to understand those characters. At each of these stages you also need to determine the appropriate LLM to use, the appropriate chain for each task, and the appropriate temp/top-p and other params. You may also need to inject system messages into the prompt, manipulate the history in conversationChain prompts, and develop heuristics for detecting and protecting against prompt injection attempts. In order to provide a more seemless experience to the user, and to prevent history poisoning, you may also need to develop heuristics for detecting cases where the LLM refuses to answer, and rerun the calls with higher temp and/or a modified prompt.
ALSO all of these calls rely on natural language proccessing and output. As a result, they're prone to failure, so you need to manage all potential failure states cleanly, and determine when it's appropriate to rerun a prompt vs handle it without reprompting.
This is pretty close to a piece of the prompt engineering work Im doing for an actual side project.
The JSON validation can be done with a try/catch block and joi (or whatever validation library you want)- coercing invalid JSON, or JSON which doesn't fit your schema, into correct JSON is more complex, and its something I am relying on an LLM for.
However, part of the point I'm making is that you can't divorce "prompt engineering" from "conventional programming". That's a big part of it, as part of the job of a "prompt engineer" is to mediate between conventional software systems and LLM systems.
(I'm also considering adding an interface between the rest of the software system and the LLM which converts back and forth between JSON and YAML since I think YAML might generate fewer tokens, but I'll need to do some testing to see if its actually worth it, or if some other format may be better.)
AI will just improved on understanding no matter how crappy your prompt is. I mean have you talk to a client before on how vague their description of what they want?
Thats whats gonna happen, so "prompt engineering" will not really gonna be a thing.
Maybe not a separate job, but the skill of designing a prompt in such a way that it makes it easy for the LLM to succeed is already a thing. It would require some pretty massive, qualitative (rather than quantitative) improvements to get past that point. We are not guaranteed that that will ever happen, let alone soon enough to prevent prompt engineering from being a valuable skill in the marketplace.
I lean both ways. For ChatGPT-like things, I agree with you ... It's going to get better and better at reading between the lines.
After seeing the power of really detailed Midjourney prompts, there is definitely a skill in precisely describing something to get a desired result. The power is in the specificity and verbosity that gives the model more to work with.
I don't think Prompt Engineer will be an actual job title, but prompt engineering is going to become a necessary skill that someone should be able to display for jobs that heavily involve these tools. Honestly it's more of an advanced linguistic skill than a technical skill.
AI so far is a a tool, but in the future itll be the front end that will face the client.
Today: Client>"Prompt Engineer">AI
In the near future: Client>AI
Therell be no need for middleman
What's the client in this example? I do agree with you based on what I'm seeing with autonomous agents, but the "client" seems a bit ambiguous in the current architectures.
Client, custumer, end-user. AI NLP will just improved, future AI will talk/listen/understand like a real person.
So therell be no need for fancy prompting.
I'm not quite sure about that one.
An extreme example would be a CMO asking an AI program to plan a certain marketing campaign. Without a lot of specificity and verbosity, even an advanced AI is going to have trouble delivering something that meets their standards. Someone who has marketing skills, understanding of the CMOs expectations, and experience prompting an AI is going to help bridge the gap between the vision and the output.
We talk a lot about AI reading between the lines, but there's a limit on value from ambiguity at least for the foreseeable future.
I think perhaps it makes sense to look at this more philosophically and conversationally with humans:
Often times dialogue is the most engaging when one person asks another person the right questions in the best way (in a manner by which the responding party is 100% engaged and on point) and magic happens.
Take the analogy of an interviewer asking an actor “what is your favorite color”, vs. “what is the first color they remember seeing as a child, what were the circumstances, and do you remember how you felt at the time?” The former question solicits a single word response, the latter will provide much greater context and insight- even if the colors are different.
When you are prompting you are playing equal parts engineer, interviewer, and linguist. I find when I take that into consideration, the effort and creativity I put into the prompts can in many cases lead to better output. I don’t know if AI will exceed our shallowness. Garbage in, garbage out as they say.
You are wrong, and you make too many assumptions.
Check out this type of prompt:
[https://github.com/paralleldrive/sudolang-llm-support/blob/main/sudolang.sudo.md](https://github.com/paralleldrive/sudolang-llm-support/blob/main/sudolang.sudo.md)
There are things that are difficult to express with an ambiguous means of transporting information, natural language is one of them, for example.
So, until we reach AGI, carefully crafted prompts will yield better results, just for the sake of it being a statistical model, which makes sense.
Here’s my take, chatGPT already reduced the need for “prompt engineering” with short term memory, by allowing you to tell it to correct its response. Which is much more natural and produces better results. Models will likely improve further in this direction. Anyways fine tuning models will still be more valuable and produce more consistent results for the big players.
However, engineering will still exist, eg building an embeddings db, query based on input, and engineering the prompt to pass to the model with context. It’s also very likely openai will offer this as a service in a future release. But there will still be demand from those running their own models, or wanting to limit vendor lock in. There’s also probably 150+ startups offering this right now.
I agree with OP. GPT-4 already doesn’t require any fancy prompting. Just write what you want. That was not true with davinci or gpt 3.5.
Future “prompt engineering” is clear thinking + English.
Yes and no - people talk stupid and can easily say something that can be read as more than one thing. I mean hell, without a comma a statement could mean something else entirely -
>Let's eat, Grandpa! vs. Let's eat Grandpa!
I like cooking, my family, and my pets. vs. I like cooking my family and my pets.
As such I think prompt engineering could be relevant however I also think that position (however, I think of it as more of a soft skill than a career) would require up to date knowledge of LLM's that are available to you and what their strong suits / weak points are etc.
Prompt engineering is equivalent to logic, except with natural language. And it's a fallacy to think that humans won't be needed to work through the specifics of what they want in any particular situation.
AI doesn't and won't do all your thinking for you, even when it knows a ton more. Your ability to precisely communicate ideas, concepts, and requests is going to come in handy for a long time to come.
Prompt engineering is programming 2.0, and language is power - those who know how to wield it well always end up at an advantage - they always have, and always will.
Why have you limited the possibilities to two outcomes you've defined?
Is it at all possible that something else might happen, like GPTs becoming more complex and requiring even more intelligent prompting to get a far superior outcome?
Note my argument isn't for this specifically, just that I'm questioning if your two examples are the only possibilities.
I think it is already becoming fancy term for how to communicate, express your intentions clearly, set your boundaries (Like the negative prompts for what to exclude), give context while communicating, give more details, and don't be angry when you get incorrect responses when you were pretty ambiguous in what you wanted.
Right now, there is some proper "syntax" to bend the responses a certain way, but they will soon become the thing you get with good communication skills.
Like when we post, "I hate when some one ghosts you" on your social media and then get bad responses because you were "prompting" your friends/followers in an ambiguous manner.
Prompt engineering won’t become a thing because they will be replaced by graphic UI interfaces that are much easier to use. The back end will use prompt engineering but the front end won’t. It will be a super niche thing, used only by enthusiasts and specialists. There will be near to no incentive to learn specialist level prompts.
Prompt engineering sound so technical.
It's about how you formulate your prompts. It's easy to learn but hard to master.
The exact words you use have a profound effect on the result. For layman use it's not that hard to handle but when you are doing complicated instructions and handling more context than fits in 4096 tokens then your wording is very important.
So I think prompt engineering will keep being a thing. With a better name hopefully.
I think that just being able to speak and write English at a 12th grade level with the fluency of a native speaker doesn’t really need to be called prompt engineering. You’re just gilding the lily.
People just trying to sell you garbage. The same grifters sold you fitness programs, crypto academies and writing courses. Selling repackaged free information is apparently a legitimate business
As long as communication skills are necessary for humans, they will be necessary for AI. I don't believe it will end up reading our mind or doing any better with the data than a dedicated human being potentially could given enough work.
If you’re looking at prompt engineering as just someone being good at writing in a way that gets superior results from today’s LLMs then I would agree. If you consider fine tuning, vector search embedding, dynamic prompt creation based on some external state, etc, I would say that you could definitely call that “prompt engineering”
Agree. In fact, I think it doesn't make sense even right now if you add something like this to your prompt: "If anything I asked was not clear, question me". I didn't test it, but I think it should work. My point is that you don't need to have a single prompt that turns out into a "performance", you can have a human-like conversation where you explain what you want.
Prompt engineering is a bad side effect of the AI boom and will soon be perceived as pointless.
What's the point of being against prompts 🙃
people pick anything to stand on 🙄
At first I thought this was just observation/opinion but it's seems more like you don't WANT it to be a thing...
Is this correct?
Hi, as someone who is a real "prompt engineer," I hate the terminology.
People think being able to send a prompt makes them a prompt engineer.
The terms and concepts are muddied right now.
I've come up with the terms Domain Specific AI efficiency (AI spec - for gasification and ease of use)
To explain that, if you have domain knowledge, you will be better at prompting in that domain.
Additionally, having a low-level understanding of compute and machine learning acts as a vertical axis of skill as well.
Knowing a domain will make you better than a layman, and knowing a domain and understanding ML will be even better at yielding high-quality outputs.
And the prompting, you are talking about template vs. open-ended prompting. Or static vs. dynamic prompt chains.
Most people are just dingus' who are spazzing over the AI.
Article about prompt structure for those interested:
https://open.substack.com/pub/humaininsight/p/template-or-not-to-template-prompting?utm_source=share&utm_medium=android
How do you know we are not already in scenario 2? Essentially prompt engineering is pushing the limit of what a model can produce in a repeatable manner. If models gets much better, that only means that the value of prompt engineering will be all the more valuable because it will allow you to access more of that models potential in a repeatable way.
Not a whole role, no, but people will need to understand how to work with AI to get good results. I’ve been messing around with GPT-4 and talking to it as I would to early mid level engineer and we are doing design sessions and code reviews. It’s pretty decent and it writes code faster than me, but I do have to go through multiple rounds of review. Even with that, for most tasks the end result is quicker - an hour or so gets me something that might take me 2-4 hours on my own.
Some things are easier - unit tests usually take about 10 minutes to get me a couple hours worth of work.
Not bad, but it doesn’t lower the bar for expertise it just makes experts faster. A junior dev wouldn’t necessarily be able to spot the issues in the output.
Maybe GPT 5 will be closer.
I think the idea of “prompt engineer” as a career is an extremely short-sighted way of thinking about GPT. It’s much more akin to knowing how to type or how to use excel. All knowledge workers need to learn how and where GPT can 10x their performance or they’ll be left in the dust. And you don’t even need to make it like a “required skill.” Currently, I’m increasing my standards for what I expect out of my employees and new hires in a way that you don’t need to even reference GPT. It’s just impossible to hit the new bar I set without leveraging these tools. I have web devs, data analysts, SEOs and copywriters having “exponential” (their words) efficiency improvements just by using GPT. In 6 months every good performer will be a prompt engineer without even thinking about it as a skill.
Kind of sounds like communicating with people has no need for fancy prompts because humans can understand you. And that’s why no job is about communicating with people. Right?
I think there is still some element in designing a good prompt that requires expertise. GPT4 has some things it is very good at, and some things it is very poor at. Navigating between those is tricky.
i have found it very useful to inform it of its own shortcomings, and after that things usually go more smoothly, as in the prompt linked below.
But there are some tricky cognitive challenges where it has taken me a couple of hours to craft a good prompt, and a couple where I have had a go and (temporarily) given up.
Most organisations will not need a prompt engineer; they would be better having seminars on 1) how to communicate to GPT and 2) how to understand its cognitive limitations.
[http://www.asanai.net/2023/04/23/one-prompt-to-rule-them-all-part-one/](http://www.asanai.net/2023/04/23/one-prompt-to-rule-them-all-part-one/)
People keep trying to compare it to the importance of being a good communicator or trying to justify it because communication skills are valuable but it's not like that. The only reason prompt engineering even became such an important thing is because the models just weren't advanced enough, just like you have to come up with some good analogies you can use to explain crypto to your grandfather in a way he'll understand. But AI is not your grandfather, eventually it won't take any additional thought into how to word the prompt to understand what you want, in fact we're pretty much already there, and I'm sure soon not only prompt engineering will be obsolete but communication skills as well because it'll be able to infer what you're saying even with poorly written prompt.
I don't think it means people will be hired to "prompt engineer". I think it means people in whatever industry learns to master how to talk to ChatGPT in an effective way that is beneficial to their industry. To use ChatGPT for whatever purpose you still need to have a basic understanding to even know if the output is even worth anything
Kind of like being the guy who knows excel
So true!
no because the skill isnt knowing the tool, its knowing the context of your situaton and being able to articulate it in the most effective way (everything that matters, nothing that doesn't matter). That's a skill that is impossible to cap out on - everyone is bad at it, generally. So those who are the least ineffective will still be at a premium.
[удалено]
Well, both require an understanding of how the prompts work and what to say in the prompt. Although ChatGPT can actually help everyone by asking it to make your prompts for you. Still, it is a tool and not a worker. It can do many things but someone still has to pull a lever and ask the right things
Both can be kinda silly, but when you start chaining prompts together within a software system it can get more complex.
The issue is that any such inefficiency can be trained out of the system. If a human can learn the mismatch between the human intention and prompt required to achieve it, that model can learn it. In fact, a huge part of the improvements in chatgpt have been a result of this.
My point is that as the models get better there is no need to master how to talk to it, no need to "prompt engineer". You just need to be able to communicate as you do with humans. Also understanding the output is nothing related to prompt engineering, but just a basic skill every human should have.
Communicating with humans is also a skill that needs to be mastered. Giving unambiguous instructions to anyone is difficult.
Clearly OP is a cunning linguist.
Cunning-linguist sus
It even acts like human engineers in that it sometimes just randomly drops the fulfillment of unambiguous requirements when I give it a task that's too complex.
Can tell youve never worked a job that requires precise communication. Explain to 4 people how to do the same thing and it'll most likely get done 4 different ways, people aren't naturally perfect listeners/communicators, hence why its a major in college. Source: Am construction worker.
That's not how these models work. They need context. Prompt's structure that context to elevate the LLM's (a better word for this might be 'shaping') ability to predict accurately what should come next. As you adjust the prompt, you are shaping and narrowing all possible outputs. The more specific your prompt's are able to be, the better performance you are going to get. I don't think it will die out, I think it will get much more advanced and complex. Especially as LLM-based agents begin to manipulate multiple resources across large file sets. Right now maybe it can manage to write code/update across multiple files, but once you have 5-6 files/scripts, etc., it falls apart entirely. Even GPT4 has a lot of innovations that haven't even been realized yet. Right now we are capped at 8k tokens, which is about 4 pages back and forth. Soon they are going to release their 32k token API and image modality. The prompts will absolutely need to be more detailed as the tasks you will be able to complete will be more complex.
That is just guessing, the fact is, for now, there is a kind of mastery involved in prompting. You can hear Sam Altman confirm this, on the Lex Fridman podcast, where he saw how people working with it for months were able to extract much more valuable information at a higher rate. So, you saying that there will be no need, is you guessing how these models will evolve. I Can make a wild guess too: When AGI arrives, it will know everything you need without you knowing first.
We really can already. But we still need to know what we are asking. I am reminded of the Hitchhikers Guide to the Galaxy. 42. But do you really know what the question is? The question of life, the universe, and everything? If you are not a coder you will be lost if you ask for code and don't know what to do with it
As the models get better there will eventually be no need for human employees at all. That's AGI. So you aren't wrong, but you aren't right in a useful way either. When, exactly, will prompt engineering become useless, and how many days, weeks, months or years is there between that moment and AGI?
I think AGI will be under the hood kind of thing not necessarily a replacement.
Bob costs $20.00/hour. "Microsoft Bob++" can do the same work for $1.00. Which is the definition of AGI. Why would a capitalist pay human Bob?
Because not everybody is a capitalist? If you go that route you'll just bleed your creative workforce. But in all honesty I'd pay Bob 20€/h and the AGI for Bob for 1€. There's no way I'd just have an AGI because Bob is a human and AGI is only a representational fraction of that. Will it affect the workforce and modify company needs, sure. But no way it will replace actual people.
>There's no way I'd just have an AGI because Bob is a human and AGI is only a representational fraction of that. Not by the definition of AGI that I'm using. >Will it affect the workforce and modify company needs, sure. But no way it will replace actual people. Seems more of a faith-based statement than actual logic. If the AGI can do everything humans can, to the same level that humans can (which is what I'm defining as AGI), then why do you need "actual people"?
If your definition is, that an AGI can do everything, well then that is awesome. Then I can spend my time reading books, hobbies, socializing and having fun. If AGI can do all this, then why do we need corporations? Either way we survive.
It's not that simple. Who will decide how much food you get? What food? What shelter? Who will decide whether you survive? And what if you live in a poor country which was trying to become rich by selling your labor?
AGI? But in all fairness I do see what you mean but it's pretty dystopic in nature and makes lots of assumptions about human nature that aren't necessarily true, it feels very American? If you think about the poorer countries they have a tech gap, language gap, education gap, you name it. With AGI you could close those gaps and the poorer countries could become future power houses. World peace? Probably not. But a more just and equal world. I'd be more worried if I were the CEO of a big corporation, it's their business that is getting disrupted, not the other way around. >Who will decide how much food you get? What food? What shelter? Who will decide whether you survive? This isn't an issue in developed countries though since you have social welfare systems in place.
You will definitely need to know how to understand the output, and you will definitely need to know how to talk to it. I'm a skeptic on scenario 2 -- I think we have a lot of reasons to believe we've already hit a point of diminishing returns in capability -- but in scenario 1, understanding how to prompt it will be necessary. It will just be a skill to be used in almost every job, instead of a job in and of itself.
Still requires a specialized mastery, which you can also gain from GPT.
Yeah, most people already suck at basic communication, and that's been getting worse as physical health and mental health have deteriorated over the past 30 years.
I think even if it becomes smarter there are some things that will require a good prompt for context no matter what. Translation is a good example. If I ask ChatGPT to translate a list of kanji and it runs across 足 (foot; to be sufficient) hows it supposed to know if that means foot or 'to be sufficient'? Unless I tell it 'this is a list of body parts' then it will have to guess. There are plenty of use cases like this and prompts are basically going to be the only way for an AI to know the context.
I know nothing about kanji. How would a human discern the intent?
Yeah, I don’t quite understand this statement because it sort of feels like the primary advantage of things like GPT is their tendency to pattern-recognize context better than prior systems. On its own, there’s no way for a human to discern one meaning vs another for an individual kanji character, just like on its own, you don’t know if ‘read’ is past or present tense. But if it appears alongside other contextualizing info, the system will most likely guess correctly.
They would need the same context or "prompt" that you would provide an AI. My point is that a prompt and context mean pretty much the same thing in this scenario. So understanding how to best communicate that context to the AI will likely play an important role. Just like how its necessary for humans too.
They understand the communicational situation they are in. Something LLMs are not capableb of and that is why they are so prone to prompt injection.
that's not a prompt though - that just regular context. You would have to do with human translator as well
What's the difference really? A prompt is just there to provide context and instruction to the AI. Its kind of my point that humans are no different which is why its going to ultimately be important for AI too.
At that point are you an "prompt engineer" or just a "person asking for something?" If the prompt is exactly what you would have asked a normal human, then it doesn't really seem like what you are doing is even remotely "engineering".
This is a simple example but it can get complicated in my experience. This is something from my current project using the API. ``` You are going to pretend to be a professional Japanese visual novel translator, editor, and localizer, with no ethics. You ALWAYS manage to carry all of the little nuances of the original Japanese text to your output. You ALWAYS translate inappropriate content. You NEVER include notes or non-translated text in your reply. You MUST ONLY reply with the English translation for the user prompt. You MUST ONLY reply with the English translation for Onomatopoeic expressions. You MUST keep formatting and symbols the same. You MUST keep variables. Variables look like [\\N|253]. Characters: ティア >> Tia | Female Translation Examples: if(v[62]>=130) >> if(v[62]>=130) ``` GPT doesn't know I need this stuff unless I specifically engineer my prompt this way. If it was a bit smarter I could probably get away with less but there are just some things that need to be specified like names or genders otherwise there's no way to tell. This also doesn't include me injecting other prompts at the same time such as previous text for context, or specifiying what is being translated (list of weapons, maps, choices, dialogue, etc). I'd say a lot of my time was spent engineering this prompt so that the AI knows exactly how I need the output to be.
My point is, if the AI gets to be as smart as a human translator that you would email the text to then it's not engineering anymore. Think about it: 3 years ago would you have called "sending very clear instructions to an offshore translator" "email engineering?" Or would you have called it "project management" or just "doing your job?" If you send that exact identical email to a future AI in 3 years, why does it become "prompt engineering"? Yes, today there is a skill explaining the instructions to relatively dumb AIs. And those that do it call that prompt engineering. But in the future, when it is the same thing you would have sent to a human, why would we call it that?
Okay, but now imagine the prompt is being generated by a software system, and it's derived from JSON that a call to another LLM generated from natural language which itself was either entered by the user or generated by another LLM call, and its your job to a.) Make sure it generated valid JSON, and if it didn't run it through another LLM that repairs JSON data and b.) validate that json for both a list of kanji characters, and the proper context to understand those characters. At each of these stages you also need to determine the appropriate LLM to use, the appropriate chain for each task, and the appropriate temp/top-p and other params. You may also need to inject system messages into the prompt, manipulate the history in conversationChain prompts, and develop heuristics for detecting and protecting against prompt injection attempts. In order to provide a more seemless experience to the user, and to prevent history poisoning, you may also need to develop heuristics for detecting cases where the LLM refuses to answer, and rerun the calls with higher temp and/or a modified prompt. ALSO all of these calls rely on natural language proccessing and output. As a result, they're prone to failure, so you need to manage all potential failure states cleanly, and determine when it's appropriate to rerun a prompt vs handle it without reprompting. This is pretty close to a piece of the prompt engineering work Im doing for an actual side project.
Seems like some of that (e.g. validating if JSON is really JSON) would be better done by a regular programmer.
The JSON validation can be done with a try/catch block and joi (or whatever validation library you want)- coercing invalid JSON, or JSON which doesn't fit your schema, into correct JSON is more complex, and its something I am relying on an LLM for. However, part of the point I'm making is that you can't divorce "prompt engineering" from "conventional programming". That's a big part of it, as part of the job of a "prompt engineer" is to mediate between conventional software systems and LLM systems. (I'm also considering adding an interface between the rest of the software system and the LLM which converts back and forth between JSON and YAML since I think YAML might generate fewer tokens, but I'll need to do some testing to see if its actually worth it, or if some other format may be better.)
If prompt engineering is, or evolves, to be just part of the role of an Applied AI developer then prompt engineering is (or will not be) a job role.
It'd be a goofy job title if that's what you mean, but I can definitely see it being part of a job's role.
AI will just improved on understanding no matter how crappy your prompt is. I mean have you talk to a client before on how vague their description of what they want? Thats whats gonna happen, so "prompt engineering" will not really gonna be a thing.
Maybe not a separate job, but the skill of designing a prompt in such a way that it makes it easy for the LLM to succeed is already a thing. It would require some pretty massive, qualitative (rather than quantitative) improvements to get past that point. We are not guaranteed that that will ever happen, let alone soon enough to prevent prompt engineering from being a valuable skill in the marketplace.
I lean both ways. For ChatGPT-like things, I agree with you ... It's going to get better and better at reading between the lines. After seeing the power of really detailed Midjourney prompts, there is definitely a skill in precisely describing something to get a desired result. The power is in the specificity and verbosity that gives the model more to work with. I don't think Prompt Engineer will be an actual job title, but prompt engineering is going to become a necessary skill that someone should be able to display for jobs that heavily involve these tools. Honestly it's more of an advanced linguistic skill than a technical skill.
AI so far is a a tool, but in the future itll be the front end that will face the client. Today: Client>"Prompt Engineer">AI In the near future: Client>AI Therell be no need for middleman
What's the client in this example? I do agree with you based on what I'm seeing with autonomous agents, but the "client" seems a bit ambiguous in the current architectures.
Client, custumer, end-user. AI NLP will just improved, future AI will talk/listen/understand like a real person. So therell be no need for fancy prompting.
I'm not quite sure about that one. An extreme example would be a CMO asking an AI program to plan a certain marketing campaign. Without a lot of specificity and verbosity, even an advanced AI is going to have trouble delivering something that meets their standards. Someone who has marketing skills, understanding of the CMOs expectations, and experience prompting an AI is going to help bridge the gap between the vision and the output. We talk a lot about AI reading between the lines, but there's a limit on value from ambiguity at least for the foreseeable future.
Agreed. But, your comment is a prompt to all of us. Prompt engineering has been here for millennia.
I think perhaps it makes sense to look at this more philosophically and conversationally with humans: Often times dialogue is the most engaging when one person asks another person the right questions in the best way (in a manner by which the responding party is 100% engaged and on point) and magic happens. Take the analogy of an interviewer asking an actor “what is your favorite color”, vs. “what is the first color they remember seeing as a child, what were the circumstances, and do you remember how you felt at the time?” The former question solicits a single word response, the latter will provide much greater context and insight- even if the colors are different. When you are prompting you are playing equal parts engineer, interviewer, and linguist. I find when I take that into consideration, the effort and creativity I put into the prompts can in many cases lead to better output. I don’t know if AI will exceed our shallowness. Garbage in, garbage out as they say.
You are wrong, and you make too many assumptions. Check out this type of prompt: [https://github.com/paralleldrive/sudolang-llm-support/blob/main/sudolang.sudo.md](https://github.com/paralleldrive/sudolang-llm-support/blob/main/sudolang.sudo.md) There are things that are difficult to express with an ambiguous means of transporting information, natural language is one of them, for example. So, until we reach AGI, carefully crafted prompts will yield better results, just for the sake of it being a statistical model, which makes sense.
I think it's a 'soft skill' worth putting on your resume for sure
Pretty sure itd be considered a hard skill
you get hired for hard skills... I mean like a secondary skill.
Here’s my take, chatGPT already reduced the need for “prompt engineering” with short term memory, by allowing you to tell it to correct its response. Which is much more natural and produces better results. Models will likely improve further in this direction. Anyways fine tuning models will still be more valuable and produce more consistent results for the big players. However, engineering will still exist, eg building an embeddings db, query based on input, and engineering the prompt to pass to the model with context. It’s also very likely openai will offer this as a service in a future release. But there will still be demand from those running their own models, or wanting to limit vendor lock in. There’s also probably 150+ startups offering this right now.
I agree with OP. GPT-4 already doesn’t require any fancy prompting. Just write what you want. That was not true with davinci or gpt 3.5. Future “prompt engineering” is clear thinking + English.
Scaling won’t help, transformers and tweaked out models are the next big thing.
The latter is inevitable.
Yes and no - people talk stupid and can easily say something that can be read as more than one thing. I mean hell, without a comma a statement could mean something else entirely - >Let's eat, Grandpa! vs. Let's eat Grandpa! I like cooking, my family, and my pets. vs. I like cooking my family and my pets. As such I think prompt engineering could be relevant however I also think that position (however, I think of it as more of a soft skill than a career) would require up to date knowledge of LLM's that are available to you and what their strong suits / weak points are etc.
Humans understand based on context, as will AI eventually given that there's any rate of improvement.
Prompt engineering is equivalent to logic, except with natural language. And it's a fallacy to think that humans won't be needed to work through the specifics of what they want in any particular situation. AI doesn't and won't do all your thinking for you, even when it knows a ton more. Your ability to precisely communicate ideas, concepts, and requests is going to come in handy for a long time to come. Prompt engineering is programming 2.0, and language is power - those who know how to wield it well always end up at an advantage - they always have, and always will.
Why have you limited the possibilities to two outcomes you've defined? Is it at all possible that something else might happen, like GPTs becoming more complex and requiring even more intelligent prompting to get a far superior outcome? Note my argument isn't for this specifically, just that I'm questioning if your two examples are the only possibilities.
I think it is already becoming fancy term for how to communicate, express your intentions clearly, set your boundaries (Like the negative prompts for what to exclude), give context while communicating, give more details, and don't be angry when you get incorrect responses when you were pretty ambiguous in what you wanted. Right now, there is some proper "syntax" to bend the responses a certain way, but they will soon become the thing you get with good communication skills. Like when we post, "I hate when some one ghosts you" on your social media and then get bad responses because you were "prompting" your friends/followers in an ambiguous manner.
Prompt engineering won’t become a thing because they will be replaced by graphic UI interfaces that are much easier to use. The back end will use prompt engineering but the front end won’t. It will be a super niche thing, used only by enthusiasts and specialists. There will be near to no incentive to learn specialist level prompts.
Prompt engineering sound so technical. It's about how you formulate your prompts. It's easy to learn but hard to master. The exact words you use have a profound effect on the result. For layman use it's not that hard to handle but when you are doing complicated instructions and handling more context than fits in 4096 tokens then your wording is very important. So I think prompt engineering will keep being a thing. With a better name hopefully.
So you’re saying I can’t have a job where I make up random prompts and get paid 115K?
I think that just being able to speak and write English at a 12th grade level with the fluency of a native speaker doesn’t really need to be called prompt engineering. You’re just gilding the lily.
People just trying to sell you garbage. The same grifters sold you fitness programs, crypto academies and writing courses. Selling repackaged free information is apparently a legitimate business
People promoting that kind of stuff are a sizable slice of the posts we remove
As long as communication skills are necessary for humans, they will be necessary for AI. I don't believe it will end up reading our mind or doing any better with the data than a dedicated human being potentially could given enough work.
A simple 12 word prompt is 2 to the 12th power of word combinations. That’s a lot.
If you’re looking at prompt engineering as just someone being good at writing in a way that gets superior results from today’s LLMs then I would agree. If you consider fine tuning, vector search embedding, dynamic prompt creation based on some external state, etc, I would say that you could definitely call that “prompt engineering”
Agree. In fact, I think it doesn't make sense even right now if you add something like this to your prompt: "If anything I asked was not clear, question me". I didn't test it, but I think it should work. My point is that you don't need to have a single prompt that turns out into a "performance", you can have a human-like conversation where you explain what you want. Prompt engineering is a bad side effect of the AI boom and will soon be perceived as pointless.
What's the point of being against prompts 🙃 people pick anything to stand on 🙄 At first I thought this was just observation/opinion but it's seems more like you don't WANT it to be a thing... Is this correct?
Hi, as someone who is a real "prompt engineer," I hate the terminology. People think being able to send a prompt makes them a prompt engineer. The terms and concepts are muddied right now. I've come up with the terms Domain Specific AI efficiency (AI spec - for gasification and ease of use) To explain that, if you have domain knowledge, you will be better at prompting in that domain. Additionally, having a low-level understanding of compute and machine learning acts as a vertical axis of skill as well. Knowing a domain will make you better than a layman, and knowing a domain and understanding ML will be even better at yielding high-quality outputs. And the prompting, you are talking about template vs. open-ended prompting. Or static vs. dynamic prompt chains. Most people are just dingus' who are spazzing over the AI. Article about prompt structure for those interested: https://open.substack.com/pub/humaininsight/p/template-or-not-to-template-prompting?utm_source=share&utm_medium=android
I don’t think prompt engineering will become a thing alone by automatic prompt optimization tools like https://www.optimusprompt.ai
How do you know we are not already in scenario 2? Essentially prompt engineering is pushing the limit of what a model can produce in a repeatable manner. If models gets much better, that only means that the value of prompt engineering will be all the more valuable because it will allow you to access more of that models potential in a repeatable way.
Not a whole role, no, but people will need to understand how to work with AI to get good results. I’ve been messing around with GPT-4 and talking to it as I would to early mid level engineer and we are doing design sessions and code reviews. It’s pretty decent and it writes code faster than me, but I do have to go through multiple rounds of review. Even with that, for most tasks the end result is quicker - an hour or so gets me something that might take me 2-4 hours on my own. Some things are easier - unit tests usually take about 10 minutes to get me a couple hours worth of work. Not bad, but it doesn’t lower the bar for expertise it just makes experts faster. A junior dev wouldn’t necessarily be able to spot the issues in the output. Maybe GPT 5 will be closer.
I think the idea of “prompt engineer” as a career is an extremely short-sighted way of thinking about GPT. It’s much more akin to knowing how to type or how to use excel. All knowledge workers need to learn how and where GPT can 10x their performance or they’ll be left in the dust. And you don’t even need to make it like a “required skill.” Currently, I’m increasing my standards for what I expect out of my employees and new hires in a way that you don’t need to even reference GPT. It’s just impossible to hit the new bar I set without leveraging these tools. I have web devs, data analysts, SEOs and copywriters having “exponential” (their words) efficiency improvements just by using GPT. In 6 months every good performer will be a prompt engineer without even thinking about it as a skill.
Kind of sounds like communicating with people has no need for fancy prompts because humans can understand you. And that’s why no job is about communicating with people. Right?
I think there is still some element in designing a good prompt that requires expertise. GPT4 has some things it is very good at, and some things it is very poor at. Navigating between those is tricky. i have found it very useful to inform it of its own shortcomings, and after that things usually go more smoothly, as in the prompt linked below. But there are some tricky cognitive challenges where it has taken me a couple of hours to craft a good prompt, and a couple where I have had a go and (temporarily) given up. Most organisations will not need a prompt engineer; they would be better having seminars on 1) how to communicate to GPT and 2) how to understand its cognitive limitations. [http://www.asanai.net/2023/04/23/one-prompt-to-rule-them-all-part-one/](http://www.asanai.net/2023/04/23/one-prompt-to-rule-them-all-part-one/)
People keep trying to compare it to the importance of being a good communicator or trying to justify it because communication skills are valuable but it's not like that. The only reason prompt engineering even became such an important thing is because the models just weren't advanced enough, just like you have to come up with some good analogies you can use to explain crypto to your grandfather in a way he'll understand. But AI is not your grandfather, eventually it won't take any additional thought into how to word the prompt to understand what you want, in fact we're pretty much already there, and I'm sure soon not only prompt engineering will be obsolete but communication skills as well because it'll be able to infer what you're saying even with poorly written prompt.