When I’ve been building applications with
large language models, I don’t think I’ve ever come to the prompt that
I ended up using in the final application on my first attempt.
And this isn’t what matters. As long as you have a good process
to iteratively make your prompt better, then you’ll
be able to come to something that works
well for the task you want to achieve.
You may have heard me say that when I train a machine learning model,
it almost never works the first time. In fact, I’m very surprised if the first
model I train works. I think we’re prompting, the odds
of it working the first time is maybe
a little bit higher, but as he’s saying, it doesn’t matter if the
first prompt works. What matters most is the process for getting
to the prompts that work for your application.
So with that, let’s jump into the code and let me show
you some frameworks to think about how to
iteratively develop a prompt. Alright, so if you’ve taken
a machine learning class with me, before you
may have seen me use a diagram saying that with
machine learning development, you often have an idea and
then implement it. So write the code, get the
data, train your model, and that gives you an experimental result. And you
can then look at that output, maybe do error analysis, figure out
where it’s working or not working, and then
maybe even change your idea of exactly what problem
you want to solve or how to approach
it, and then change your implementation and run another experiment and so
on, and iterate over and over to get
to an effective machine learning model. If you’re not familiar with machine learning
and haven’t seen this diagram before, don’t worry about it,
not that important for the rest of this presentation. But
when you are writing prompts
to develop an application using an OOM, the process can be
quite similar where you have an idea for what you want to
do, the task you want to complete, and you can then
take a first attempt at writing a prompt
that hopefully is clear and specific and maybe,
if appropriate, gives the system time to think,
and then you can run it and see what result you get.
And if it doesn’t work well enough the first time, then
the iterative process of figuring out why the instructions,
for example, were not clear enough or why it didn’t give
the algorithm enough time to think, allows you
to refine the idea, refine the prompt, and so on, and to
go around this loop multiple times until you
end up with a prompt that works for your application. This too
is why I personally have not paid as
much attention to the internet articles that say
30 perfect prompts, because I think there probably isn’t a perfect
prompt for everything under the sun. It’s more important that
you have a process for developing a good
prompt for your specific application. So let’s look
at an example together in code. I have here the starter
code that you saw in the previous videos,
have been port open AI and port OS. Here we get the open
AI API key, and this is the same helper function that you
saw as last time.
I’m going to use as the running example in
this video the task of summarizing a fact
sheet for a chair. So let me just paste that in here. Feel
free to pause the video and read this more carefully
in the notebook on the left if you want. But here’s a
fact sheet for a chair with a description saying it’s part of
a beautiful family of mid-century inspired, and so on. Talks about the construction,
has the dimensions, options for the chair,
materials, and so on. Comes from Italy.
So let’s say you want to take this fact sheet and help a marketing
team write a description for an online retail
as follows, and I’ll just…
and I’ll just paste this in,
so my prompt here says your task is to help a marketing
team create the description for retail
website or product based on a techno fact sheet,
write a product description, and so on. Right? So this is my
first attempt to explain the task to the large-language
model. So let me hit shift enter, and
this takes a few seconds to run,
and we get this result. It looks like it’s
done a nice job writing a description, introducing a stunning mid-century inspired
office chair, perfect edition, and so on, but when
I look at this, I go, boy, this is really long. It’s done a
nice job doing exactly what I asked it to, which is start
from the technical fact sheet and write a
But when I look at this, I go, this is kind of long.
Maybe we want it to be a little bit shorter.
So I have had an idea. I wrote a prompt, got the result.
I’m not that happy with it because it’s too
long, so I will then clarify my prompt and say
use at most 50 words to try to give better guidance on
the desired length of this, and let’s run it again.
Okay, this actually looks like a much nicer short
description of the product, introducing a mid-century
inspired office chair, and so on, five you just, yeah, both
stylish and practical. Not bad.
And let me double check the length that this is. So I’m
going to take the response, split it according to where
the space is, and then you’ll print out the length. So it’s 52 words.
Actually not bad.
Large language models are okay, but not that great
at following instructions about a very precise word count,
but this is actually not bad. Sometimes it will print
out something with 60 or 65 and so on words, but it’s
kind of within reason. Some of the things you
Let me run that again. But these are different
ways to tell the large-language model what’s the length of the output
that you want. So this is one, two, three. I count
these sentences. Looks like I did a pretty good job. And then
I’ve also seen people sometimes do things like, I don’t know, use at
most 280 characters. Large-language models, because of the way they
interpret text, using something called a tokenizer, which I won’t talk about.
But they tend to be so-so at counting characters. But
let’s see, 281 characters. It’s actually surprisingly close. Usually a
large-language model doesn’t get it quite
this close. But these are different ways they can play
with to try to control the length of the output that you
get. But then just switch it back to use at most
And that’s that result that we had just now.
As we continue to refine this text for our website,
we might decide that, boy, this website isn’t
selling direct to consumers, it’s actually intended to sell
furniture to furniture retailers that would
be more interested in the technical details of the chair and the
materials of the chair. In that case, you can
take this prompt and say, I want to modify this prompt to get it to
be more precise about the technical details.
So let me keep on modifying this prompt.
And I’m going to say,
this description is intended for furniture retailers,
so it should be technical and focus on materials,
products and constructs it from.
Well, let’s run that.
And let’s see.
Not bad. It says, coated aluminum base
and pneumatic chair.
High-quality materials. So by changing the prompt, you
can get it to focus more on specific characters, on
specific characteristics you want it to. And
when I look at this, I might decide, hmm, at the end of the description,
I also wanted to include
the product ID. So the two offerings of this chair,
SWC 110, SOC 100. So
maybe I can further improve this prompt.
And to get it to give me the product IDs,
I can add this instruction at the end of the description,
include every 7 character product ID
in the technical specification. And let’s run it
and see what happens.
And so it says, introduce you to our mid-century
inspired office chair, shell colors, talks about plastic coating
practical, some options,
talks about the two product IDs. So this looks pretty good.
And what you’ve just seen is a short example of the iterative
prompt development that many developers will
And I think a guideline is, in the last video,
you saw Yisa share a number of best practices. And so what I
usually do is keep best practices like that in mind,
be clear and specific, and if necessary,
give the model time to think. With those in mind, it’s
worthwhile to often take a first attempt at
writing a prompt, see what happens, and then go from there
to iteratively refine the prompt to get closer
and closer to the result that you need. And
so a lot of the successful prompts that
you may see used in various programs was
arrived at an iterative process like this. Just
for fun, let me show you an example of an even
more complex prompt that might give you a sense of what ChatGPT
can do, which is I’ve just added a few extra
instructions here. After description, include a
table that gives the product dimensions, and then
you’ll format everything as HTML. So let’s run
And in practice, you would end up with a prompt like this,
really only after multiple iterations. I don’t think I know anyone
that would write this exact prompt the first
time they were trying to get the system
to process a fact sheet.
And so this actually outputs a bunch of HTML. Let’s
display the HTML to see if this is even valid
HTML and see if this works. And I don’t actually know it’s going to
work, but let’s see. Oh, cool. All right. Looks like a rendit.
So it has this really nice looking description of
a chair. Construction, materials, product dimensions.
Oh, it looks like I left out the use at most 50 words instruction,
so this is a little bit long, but if you want that,
you can even feel free to pause the video, tell it to be more
succinct and regenerate this and see what results you get.
So I hope you take away from this video that prompt development
is an iterative process. Try something,
see how it does not yet, fulfill exactly what you want,
and then think about how to clarify your instructions,
or in some cases, think about how to give
it more space to think, to get it closer to
delivering the results that you want. And I think the
key to being an effective prompt engineer isn’t
so much about knowing the perfect prompt, it’s about
having a good process to develop prompts that are
effective for your application. And in
this video I illustrated developing a prompt using
just one example. For more sophisticated applications, sometimes you
will have multiple examples, say a
list of 10 or even 50 or 100 fact sheets, and iteratively
develop a prompt and evaluate it against a
large set of cases.
But for the early development of most applications,
I see many people developing it sort of the way
I am with just one example, but then for more mature applications,
sometimes it could be useful to evaluate prompts against
a larger set of examples, such as to test
different prompts on dozens of fact sheets to
see how this average or worst case performance
is on multiple fact sheets. But usually you end up doing
that only when an application is more mature and you have to
have those metrics to drive that incremental last few
steps of prompt improvement.
So with that, please do play with the Jupyter code notebook
examples and try out different variations and see
what results you get. And when you’re done, let’s go
on to the next video where we’ll talk about one very common use of large
language models in software applications, which is to