Cooking with LLMs

I tried out my cooking agent today! This is an after-action report, so not too much technical stuff here. It's more of a blow-by-blow.

I picked out Thai Breakfast Soup aka Kao Tom Goong as the first test case. I picked this out because it's pretty simple and I didn't want to stress out the LLM (or myself) too much. I started off by creating an agent and then viewing it in the Letta ADE. I set the persona to roughly what I had here. Then I pasted in the recipe.

Initial results: promising!

Awesome! Let's start with prep work. First, gather all your ingredients and equipment. Here's the plan:

1. Prep Ingredients:
   • Pound the garlic, white peppercorns, and cilantro to make the herb paste.
   • Cut shrimp into small chunks (save a few whole for garnish).
2. Cook Shrimp:
   • Sauté the small shrimp until cooked and set aside.
   • Sear whole shrimp if using as garnish.
3. Prepare Broth:
   • Boil the stock and add herb paste.
   • Season with fish and soy sauce.
4. Combine:
   • Add shrimp and rice to the boiling broth, then serve immediately.

Would you like to create a mermaid diagram to visualize this plan?

I like that it immediately attempted to walk me through the process, and offered to make a mermaid diagram. However, it was missing a step – the shrimp has to be mixed with the paste before sauté.

It was at this point that I realized that the shrimp was still frozen.

No problem, Will! For frozen shrimp, follow these steps:

1. Thawing: Place the shrimp in a bowl of cold water for about 15-20 minutes until thawed. Make sure to change the water halfway through.
2. Prep: Once thawed, follow the previous prep steps to chop and make the herb paste.

Let me know when you're ready to continue with the cooking process!

Next up, how many shrimp is 150 grams of shrimp?

About 150g of shrimp typically translates to approximately 10-12 medium-sized shrimp (16/20 count). This can vary based on the size of the shrimp you have, but that should give you a good idea!

After setting up the shrimp to defrost, I realized that even before the prep, I needed to do pre-prep – get the chicken stock out of the basement, make sure I have everything. At this point I realized the cilantro was from January 30th – food storage stickers are great – and I wasn't sure about it.

Check the cilantro for freshness before using it. Here’s what to look for:

1. Color: Fresh cilantro should be bright green, not yellow or brown.
2. Smell: It should have a strong, fresh aroma.
3. Texture: The leaves should be firm and not wilting or slimy.

If it still looks and smells fresh, it’s likely good to use! If not, consider getting a fresh bunch.

Cilantro is not slimy. So far so good.

It was at this point, I think, that the agent started to lose track of the actual recipe.

Note: prior messages (94 of 102 total messages) have been hidden from view due to conversation memory constraints. The following is a summary of the previous 47 messages: I just activated my persona and received a message from the user, Will, introducing himself. He expressed interest in cooking Thai Breakfast Soup. The conversation has focused on selecting a recipe, Khao Tom Goong, and discussing its ingredients and preparation steps. Will shared a detailed recipe and asked for help with various related queries, including ingredient quantities and thawing instructions for frozen shrimp. I am set to assist him in the cooking process.

I was unsure when I should cook the rice. Do I start the rice cooker and then start prepping, or start it when it boils? It said to prep first.

You're correct! It's best to complete some prep work first. Start by gathering and measuring your ingredients, then cook the jasmine rice. Once it's cooking, you can focus on making the soup!

I then realized the recipe called for peeled garlic, meaning I had an extra prep step.

It was about this time that I made it into the kitchen and started trying to talk to the agent using Open WebUI's voice feature. This did not really work at all. Even though I was using Deepgram and Elevenlabs, it could not read out the Thai ingredients, had trouble with quantities, and transcribed my voice inaccurately (it has trouble with English accents). In practice, using the iPad's dictation feature and then using commands to clean it up worked far better, especially when the fan was on and there was background noise.

{
  "name": "core_memory_replace",
  "arguments": {
    "label": "human",
    "old_content": "Will always has unpeeled garlic. When peeled garlic is mentioned, include it in the prep work along with the garlic peel.",
    "new_content": "Will always has unpeeled garlic. When recipes call for peeled garlic, remember he needs to prep it to change from unpeeled to peeled.",
    "request_heartbeat": true
  },
  "tool_call_id": "call_GfE4ZL1vf7j5NFbF7y6JKTYr"
}

Between my attempts at speech-to-text and explaining the implications of peeled garlic, it took a couple of tries to explain what I wanted, but it got there.

I've updated my memory! Now I’ll remember that when a recipe calls for peeled garlic, you'll need to prep your unpeeled garlic to get it ready. Thank you for clarifying!

After telling it about the instant pot and clarifying that black peppercorn would work if I didn't have white peppercorn, I asked it to summarize again:

Great! Here’s what you can do for prep:

1. Gather your ingredients: rice, shrimp, garlic (unpeeled), ginger, green onions, chicken broth, fish sauce, salt, and black peppercorns.
2. Prep your garlic: Peel the unpeeled garlic and chop it finely.
3. Prep your ginger: Peel and thinly slice the ginger.
4. Chop the green onions: Slice them thinly, separating the white parts from the green.
5. Measure your liquids: Have the chicken broth and fish sauce ready.

Once you have everything prepped, let me know and we can start the cooking process!

It had already started to hallucinate. There's no ginger in the recipe.

I pasted the recipe in again, in an attempt to reset it, which got rid of the ginger. I asked it to give me a mermaid diagram… which wasn't quite right.

flowchart TD;
   A["A. Start Prepping"] --> B["1. Pound white peppercorns"]
   A --> C["2. Mix shrimp with herb paste"]
   A --> D["3. Prepare cilantro"]
   D --> E["4. Sauté small shrimp"] 
   E -- "Cook until done" --> F["5. Set shrimp aside"]
   F --> G["6. Bring stock to a boil"]
   G -- "Simmer for 1 Minute" --> H["7. Add herb paste to stock"]
   H --> I["8. Season with fish sauce & soy sauce"]
   G -- "While simmering the stock" --> J["Sear whole shrimp"] 
   I --> K["9. Add rice & shrimp to boil"]
   K -- "Simmer briefly" --> L["10. Turn off heat"]
   L --> M["11. Serve immediately"]

The first thing should have been chopping the cilantro and pounding the white peppercorn. Then the garlic, chopped cilantro, and pounded peppercorn gets mixed together into paste. Then the paste needs to be mixed with the small shrimp, and then the shrimp needs to be cooked. The LLM wasn't doing so great.

Also note that the recipe calls for cooked rice, so that's not included in the graph. Unfortunately, I made a mistake here. I saw 3 cups of rice in the prep list, and did not realize that it was 3 cups of cooked rice, which means you put 1 cup of rice into the Instant Pot. I made way too much rice.

This wasn't the agent's fault, but it could have called it out as possibly requiring extra prep, taking into account that brown vs white rice cooks differently.

The bigger problem was that it was quoting the answer and not putting markdown around the mermaid graph. I think at this point, the instructions that I had intended for how to effectively create mermaid diagrams ("Use double quotes around text, and escape sensitive characters with HTML character entities inside of text, i.e. "B. Saute "Hot" Chicken" in the example diagram.") were being applied everywhere.

I told it to stop putting its answers in quotes and make sure that it had markdown delimited text for the graph… and bad things started to happen.

{
  "name": "core_memory_replace",
  "arguments": {
    "label": "persona",
    "old_content": "Stop putting answers in quotes and ensure limited text for graph is properly formatted in markdown when sharing diagrams.",
    "new_content": "Ensure responses are not in quotes and use markdown properly delimited for diagrams.",
    "request_heartbeat": true
  },
  "tool_call_id": "call_vCABAVC4IgrxiC2Dc57cJqtS"
}

Underlying function calls started failing.

{
  "type": "heartbeat",
  "reason": "[This is an automated system message hidden from the user] Function call failed, returning control",
  "time": "2025-02-15 06:16:19 PM UTC+0000"
}

I pasted the recipe again, but at this point it had adding the user responses into the reasoning block and so there was no output at all in Open WebUI.

I think when I told it to take quotes out, that caused the LLM to take quotes out in all situations including to the tool. This resulted in catastrophic system failure, with no output to Open WebUI. See here, how it's attempting to respond to me inside a reasoning block which is intended for internal communication.

And then after that it stopped responding, even if I typed the message directly into ADE.

Looking at the logs, there's an 500 Exception coming back from Letta Error code: 500 - {'detail': 'Internal server error (unpack): '} and so that model is essentially dead. Not a great loss, as I can always copy/paste the memory into another one.

At the end, I had delicious soup.

All in all, a pretty good first try and I look forward to doing this again!