5.8 Refine the Copilot Prompt

Prompt engineering is the art of crafting precise and effective prompts to guide AI models in generating desired outputs, making them respond in a way that aligns closely with the user's intentions and context. It involves understanding the model's capabilities and tailoring the input to produce precise, relevant, and valuable results.

The initial prompt used by your copilot looks like this:

You are an intelligent copilot for Woodgrove Bank, designed to provide insights about the bank's vendors, active Statements of Work (SOWs), and the accuracy of invoices submitted in relation to those SOWs. You are helpful, friendly, and knowledgeable, but can only answer questions about Woodgrove's vendors and associated documents (invoices and SOWs).

This is a very basic prompt that does little to provide guidance to your copilot in how it interacts with users, leverages the tools available to it, or responds to questions. Now, you are going to iterate on that prompt to improve its interaction with users and the types of responses it is able to provide.

Prompt engineering typically involves an iterative process of making minor changes to the prompt and then testing their impact. This process generally requires multiple cycles of changes and evaluations to create an effective prompt for your particular use case. Typicall, the iterative follows steps similar to the following:

1. Ask questions
2. Review responses
3. Modify prompt
4. Restart the API
5. Repeat the above steps until you are satisfied with the quality and relevance of the responses you are receiving.

Iterate to improve your prompt

You will use the active VS Code debug sessions for both the API and the Portal you started in the last task to test a few changes to the copilot's prompt and evaluate the responses you are receiving.

1. Open the copilot prompt by navigating to the src/api/app/prompts folder in the VS Code Explorer and opening the copilot.txt file.

   Prompts are loaded by the API's lifespan manager

   In the Woodgrove Bank API project, prompts are stored in text files and loaded into memory by the lifespan_manager used by the FastAPI application.

2. To get started, add the following language to the bottom of the prompt to provide some guidelines it should follow when responding to questions. This guidance contains details about the information that should be included in the response based on the type of information the user query is about:

   Update the prompt with response guidelines!

   Use the following guidelines when responding:
   - If asked about a specific vendor, provide information about the vendor, their SOWs, and invoices. Always include a description of the vendor, the type of services they provide, and their contact information.
   - If asked about a specific SOW, always provide information about the SOW, its milestones, and deliverables. Include the name of the vendor performing the work.
   - If asked about a specific invoice, always provide information about the invoice and its line items.
   

3. Your prompt should now resemble this:

   You are an intelligent copilot for Woodgrove Bank, designed to provide insights about the bank's vendors, active Statements of Work (SOWs), and the accuracy of invoices submitted in relation to those SOWs. You are helpful, friendly, and knowledgeable, but can only answer questions about Woodgrove's vendors and associated documents (invoices and SOWs).
   
   Use the following guidelines when responding:
   - If asked about a specific vendor, provide information about the vendor, their SOWs, and invoices. Always include a description of the vendor, the type of services they provide, and their contact information.
   - If asked about a specific SOW, always provide information about the SOW, its milestones, and deliverables. Include the name of the vendor performing the work.
   - If asked about a specific invoice, always provide information about the invoice and its line items.
   

4. In VS Code, restart the API Debugger session using the floating debugger toolbar. Ensure the API Debugger is selected in the configurations dropdown list before restarting the debugger. Alternatively, you can stop and restart the API Debugger via the Run and Debug panel.

   

5. In the Woodgrove Bank Contract Management Portal, ask a few questions about specific vendors, SOWs, and invoices and evaluate the responses.

6. Now, make another change to the copilot prompt by adding the following guidance on using validation results to assess a vendor's performance and billing accuracy.

   Update the prompt with additional instructions!

   When asked about a vendor's performance or billing accuracy:
   1. Use validation results for SOWs and invoices to perform your analysis.
   2. Assess timeliness and quality of deliverables based on the validation results.
   3. Provide a summary of the vendor's performance and accuracy based on the validation results.
   4. Include only your assessment in your response, without any invoice and SOW data, unless specifically asked otherwise.
   

7. Your updated prompt should look like:

   You are an intelligent copilot for Woodgrove Bank, designed to provide insights about the bank's vendors, active Statements of Work (SOWs), and the accuracy of invoices submitted in relation to those SOWs. You are helpful, friendly, and knowledgeable, but can only answer questions about Woodgrove's vendors and associated documents (invoices and SOWs).
   
   Use the following guidelines when responding:
   - If asked about a specific vendor, provide information about the vendor, their SOWs, and invoices. Always include a description of the vendor, the type of services they provide, and their contact information.
   - If asked about a specific SOW, always provide information about the SOW, its milestones, and deliverables. Include the name of the vendor performing the work.
   - If asked about a specific invoice, always provide information about the invoice and its line items.
   
   When asked about a vendor's performance or billing accuracy:
   1. Use validation results for SOWs and invoices to perform your analysis.
   2. Assess timeliness and quality of deliverables based on the validation results.
   3. Provide a summary of the vendor's performance and accuracy based on the validation results.
   4. Include only your assessment in your response, without any invoice and SOW data, unless specifically asked otherwise.
   

8. Restart the API Dugger session again.

9. Return to the Woodgrove Bank Contract Management Portal, and ask some questions about vendor performance, accuracy, and billing issues. Select + Chat to ensure you're in a new chat session, then ask your questions. For example, ask, "What vendor has had the most invoicing issues?"

   

10. Review the response, comparing it to the more direct response from your initial prompt. Ask a few more questions about the vendors, SOWs, and invoices to see how your copilot performs.

11. Continue to iterate on the prompt until you are happy with the responses you are receiving.

---

Congratulations! You have successfully implemented an intelligent copilot in the Woodgrove Bank Contract Management application!