Effective Prompting for Planners: Unlocking Large Language Models (LLMs)

The advent of Large Language Models (LLMs) has opened up the world of AI to everyone, allowing us to interact using natural language prompts instead of complex coding.

GPT-3.5 and GPT-4 models from OpenAI as well as Claude and many other open-source models are prompt-based. With prompt-based models, the user interacts with the model by entering a text prompt, to which the model responds with a text completion. This completion is the model’s continuation of the input text.

When used correctly, generative models can produce highly valuable results, including for planners and other built and natural environment professionals.

Getting your prompts right is therefore necessary to harness LLMs’ enormous potential, especially when you’re looking to potentially refer to or rely on its output as part of your work.

An entire industry has formed around the topic of “prompt engineering” with many self-proclaimed experts in the field (one only needs to spend some time on X or LinkedIn).

In this guide, I’ll explain the basics of prompt engineering for planners, and some tips I have learned through use of these tools.

This image has been generated on Midjourney, another AI tool. A group of planners working around a table, in old comic book style.

What are LLMs?

This guide assumes some basic understanding of what an LLM is.

LLMs, such as GPT-3 and GPT-4 from OpenAI, are artificial intelligence models that have been trained on an expansive range of internet text. But what makes them special is their ability to generate human-like text based on the prompts or inputs they're given. This can range from answering questions and writing essays, to creating poetry or even code.

For planners and other built and natural environment professionals, LLMs have the potential to assist is with large areas of our work, particularly the more mundane elements, such as summarising or finding information. They can help automate certain tasks, such as drafting reports and emails, or even generating code that you can then run locally on your computer to undertake more advanced tasks like calculations or spreadsheet collation. Additionally, they can provide valuable insights by analysing large datasets, identifying patterns, and predicting trends. We’ve even developed a whole set of planning tools on our website through use of GPT.

If you’re interested in learning more, I ran a free webinar via the New Zealand Planning Institute on AI in Planning and how it can benefit the profession. You can watch the recording here.

What is a prompt?

Think of a prompt as a task or question that you pose to an LLM like ChatGPT. The magic of these models lies in their ability to understand and respond to these prompts, ideally providing an answer that is accurate, appropriately detailed and concise, and fitting to your needs.

Essentially, prompting is about packaging your intent in a natural-language query that will cause the model to return the desired response.

Is GPT actually smart?

GPT models attempt to produce the next series of words that are most likely to follow from the previous text.

As you develop more complex prompts and work with LLM more regularly, it's helpful to keep this in mind.

Regardless of the prompt that is provided, the model is simply responding with what it determines is most likely (given its training data and training targets).

If you ask a question in your prompt, the model isn't following a separate “Q&A” code path, but rather it appears to answer the question because an answer is the most likely sort of response for the given question as input.

Models however use varying degrees of “intelligence” to do this.

For example, GPT-4 has been shown to have “emerging qualities” that border on general intelligence.

I recommend always using the latest models available. For everyday use, GPT-4 is perfect, but if you have documents or lengthy slabs of text that you want to work with as context, GPT-4 32k context length via Poe is a good backup to have.

We guess this is what a group of planners would look like if we were a moody Nolan film.

What LLM tools should I start with?

While I assume if you’re reading this guide, you have probably played around with an LLM already, here are a set of my go to tools, in order of personal usage:

1. ChatGPT. You can use the free GPT-3.5 model however I highly recommend paying for the GPT-4.0 model, in particular for access to Code Interpreter.

2. Bing. With new AI functionality, it is perfect for undertaking rapid fire research and getting access to citations that you can reference back to. Just make sure you check the references, as sometimes the model is prone to hallucination. Bing is built off the back of GPT-4.0.

3. Poe. Gives you access to the 32k context length GPT-4.0 model, as well as Claude, which offers context length of up to 100k. Claude is not as reliable as GPT-4.0 but might be suitable for some tasks that don’t require a high degree of accuracy and reliability. You can access some of the models with limited usage for free, and others will cost, either at a monthly or annual subscription.

Summary of Top Tips

Don’t have time to read this full guide? Here are some quick tips to get you started:

1. Be detailed and specific, including in terms of what information to rely on, outcome, length, format and style.

2. Include instructions at the beginning of your prompt, and then add context or information after using “inverted commas”.

3. Tell the model what to do, not what it should NOT do.

4. Ask the model to “think step by step” in order to improve factuality and quality of its evaluative output.

5. Give examples of how you want the output to look. For example, if you’re asking it to summarise an assessment for a report you are writing, give it an example of how you would usually structure that summary.

6. Omit needless words, as they just use up unnecessary tokens.

7. Ask the model for help with prompt writing. I often do this where I need research done via Bing AI. I tell ChatGPT what the problem is and what information I believe I need, and it then drafts very clear and specific questions that I can give to Bing AI for online research.

8. Refine your prompts over and over again until you get what you need. ChatGPT lets you edit your previous conversational prompts, which is also good if the model goes off in the wrong direction and you want to reset to an earlier stage of the conversation.

How to come up with good prompts

Effective prompts are rooted in two key principles: clarity and specificity. Clarity involves using straightforward language that's free from jargon and complicated vocabulary. So rather than keeping your queries short and snappy, it is in fact better to be specific and make your point sufficiently clear to the LLM.

You may consider this to be counterproductive. If you have to write something of length, perhaps you should just attempt to ‘do the work’ yourself. However, there are tools like Text Blaze that you can use to store your commonly use prompts and then reuse them easily for more repetitive LLM tasks in future.

Here is an example of an unclear prompt:

What is a condition I can use for landscaping?

You can see from the below that the model does not even know that the question is about a condition on a resource consent.

Here is an example of a clear prompt:

Instructions: 

1.  Write an example resource consent condition that requires a consent holder to implement the details shown on a proposed landscaping plan, titled “Landscape Plan”, prepared by Best Landscapers, dated 24 June 2023.

2.  Any references in the condition to the regulatory authority should be to 'the council'.

3.  Use 'must' instead of 'shall'.

4.  Use 'certified by the council' instead of 'approved'.

5.  Explain step by step after the draft condition is written how it meets Newbury principals.

The output from the model is considerably better:

Specificity, on the other hand, refers to the need for context. Tell the LLM as much as it needs to know to answer your question.

Example of an unspecific prompt:

Identify the potential environmental impacts of a proposed wind farm development.

Example of a specific prompt:

Identify the potential environmental impacts of a proposed wind farm consisting of 200 turbines, planned for undulating land north of Wellington, currently used for horticulture. Also, consider the proximity of a sensitive ecological area located 3km north-west of the proposed wind farm boundary.

Let’s now look at a few tricks to make our prompts even better.

Prompt engineering hacks

Prompting is not a science and calling it “engineering” is an overreach, suggesting that it might be all too hard for those of us not good at maths (and hence became planners and not architects).

But researchers and users of LLMs have identified some tricks that consistently improve the models’ responses.

Do say “do,” don’t say “don’t”

This point continues the specificity theme because “do” instructions are by nature more specific than “don’t”. So rather than telling the model what not to do, it’s usually better to specify exactly what you want it to do – if you know what that “thing” is.

Let’s imagine that, in our previous example, I wanted to make sure that the LLM doesn’t produce overly long descriptions of the potential effects of a wind farm (models tend to try and use all their tokens, unnecessarily). Rather than saying:

Don't provide long descriptions of each effect.

it is better to be specific:

Each effect should be listed as a bullet between two and five words long.

Use few-shot prompting

In our example, I asked for a particular item (the potential environmental impacts of wind farms) in a certain form (listed as bullets) of a certain length (two to five words). This type of instruction without any concrete examples is known as “zero-shot prompting.”

Most models, however, benefit from “few-shot prompting,” where you feed your model some examples to learn the desired pattern. In our prompt, we can add an example to help the model get a feel for what we’re looking for.

Here are some examples:

·       Ecological effects

·       Noise pollution

·       Shadow flicker

Structure your prompt in a meaningful way

Elements like quotation marks, bullet points, line breaks, etc. make it easier for humans to parse and understand text. And guess what? The same is true for LLMs.

Let’s apply that insight to our previous specificity example and spell out the full prompt once more.

Identify the potential environmental impacts of a proposed wind farm consisting of 200 turbines, planned for undulating land north of Wellington, currently used for horticulture. Also, consider the proximity of a sensitive ecological area located 3km north-west of the proposed wind farm boundary.

Each effect should be listed as a bullet between two and five words long.

Here are some examples:

- Ecological effects

 -Noise pollution

- Shadow flicker

Now that we’ve perfected our prompt, what does the LLM’s output look like? Here’s what ChatGPT came up with in response:

Here is what the response would have looked like if we didn’t use any of the practices that I set out above:

The latter response would in fact be more helpful to those looking for more detailed, but less project-specific information, and this is fine, you can tailor your prompt accordingly.

The great thing about ChatGPT is that you can continually edit your prompt and use some ‘trial and error’ to get it the LLM response to a form you find useful.

Use leading words

Sometimes, the simplest techniques are the most effective. Researchers from Tokyo University have shown that telling a model to “think step by step” before its response can generate more accurate results and help the model correct its own mistakes. This method is known as using “leading words,” because you gently guide the model towards a more effective plan for problem-solving.

Using leading words forces the model to break down its solution into multiple, more manageable steps rather than being allowed to just hazard a guess.

This is less necessary for smarter models like GPT-4, and even less so for the Code Interpreter plugin, however I still find it helpful to include this where I want the model to carefully step through its response.

Prompting pitfalls

In addition to these techniques for improvement, it helps to know which elements in an instruction can throw the LLM model off.

Some people, for example, try to incorporate requests for short, one-word answers into their prompts to prevent the models from rambling.

However, as the last example has shown with listing the environmental effects of a wind farm, wordier answers might not be a bad thing when it comes to LLMs – but rather, a technique these models use to arrive at a correct, or at least, more helpful response.

LLMs may soon learn to keep their “thought” processes to themselves, and in fact this could be something you include in an initial instruction to them at the start of a conversation. In the meantime, there could be a good case for letting them ramble and refrain from requests for one-word answers.

Another problem happens when you exceed the model’s context length. Due to the nature of these large models, an LLM’s “context window” (which is the portion of text it can process at once) comprises both the prompt and its response. If you exceed a model’s context length, its output will deteriorate.

Context lengths vary widely between models, for example, if you sign up for Poe, you can get access to a number of models that have very long context lengths, for example:

• Claude – 100k context

• GPT-4 – 32k context

• GPT-4 - 16k context

If the length of your prompt becomes an issue, you should break the problem down into smaller steps.

Note that token lengths can increase rapidly when working with numbers, as shown by the example below which demonstrates token boundaries for different date formats. In this case, spelling out the entire month is more space efficient than a fully numeric date.

Image Source: Microsoft Azure LLM Documentation

Example Prompts to Try

If you’re not sure where to start, I recommend trial and error. Think of tasks that you do for your work, whether it’s summarising information, analysing context or detail, or trying to drill down into a complex interpretation. You might also be doing some mathematics of data analysis in which case GPT-4 with the Code Interpreter plugin is perfect to use.

If you’re still stuck, here are some trial prompts you could use right now:

1. "Summarise the text below and give me a list of bullet points with key insights and important facts. [insert text]”

2. "Rewrite the text below and make it easy for a beginner to understand."

   This one can be very helpful in understanding complicated texts like research papers and technical documents.

3. “Review the information below and undertake an expert objectivity analysis. Consider whether there is any subjectivity present and identify step-by-step ways to improve the text.”

   This one can be helpful when reviewing expert reports including your own to identify whether subjectivity is identifiable.

4. “[Insert a problem or issue] Give me a step-by-step solution to the problem above with clear instructions on how to execute each step."

5. “Analyse what the potential impacts of [include details of a proposed transportation infrastructure project] could be on local traffic patterns, accessibility, and connectivity for different modes of transport, including public transit, cycling, and walking. Draw on data from [uploaded report or spreadsheet]." 

   This one would work best when using Poe or ChatGPT Code Interpreter, as you can upload documents.

6. “Please review the information below and provide constructive feedback, suggestions for improvement, and any potential omissions or inconsistencies.”

I have been publishing a regular newsletter on AI and prompting for New Zealand planners called AI-mpowered.

Most newsletter editions include a “Prompt of the Week” which is very specific to resource management planning in New Zealand.

You can see all the past editions with prompt ideas here.

Best Practice Recap

• Be Specific. Leave as little to interpretation as possible. Restrict the model’s “wiggle room” if you are looking for very specific outputs.

• Be Descriptive. Use analogies, and examples of how you want something to be presented back to you.

• Double Down. Sometimes you may need to repeat yourself to the model. Give instructions before and after your primary content, use an instruction and a cue, etc. and don’t be afraid to continually edit or refine your prompts until you get exactly what you need.

• Order Matters. The order in which information is presented to the model may impact the output. Whether you put instructions before your content (“summarise the following…”) or after (“summarise the above…”) can sometimes make a difference in output, depending on the ‘smarts’ of the model (GPT-4 it generally doesn’t matter, but if you’re using the free GPT-3.5 model, then it will be more influencing).

• Give the model an “out”. It can sometimes be helpful to give the model an alternative path if it is unable to complete the assigned task. For example, when asking it to complete a task given a set of information, you might include something like "you must ask me questions to clarify if you’re unsure" This can help the model get everything it needs from you in order to prepare a comprehensive response.

Become a better prompter - further reading

This guide has been adapted from information contained in the following resources:

• “The Beginner’s Guide to LLM Prompting” by Isabelle Nguyen

• “Prompt Engineering Guidelines” by haystack

• “Introduction to prompt engineering” Microsoft Azure Documentation

• “Advanced ChatGPT: Full Guide”

• “GPT Best Practices” by OpenAI

• “Introduction to AI for Business Users” by Microsoft

• “Career Essentials in Generative AI” by Microsoft and LinkedIn

While not so relevant to planning, you can find a comprehensive list of prompt examples on this Notion site.

Training

We offer one-on-one and team training for those interested in learning more about prompt engineering or would simply like to learn more about LLMs and how to use AI tools like ChatGPT and Midjourney. We can also help you develop specific prompts that can be used in your organisation for any type of work. Please contact us via the form below if you are interested in learning more.