A dev in Denver told me to stop tuning prompts and start tuning context

Last month a guy at a meetup said my AI chatbot kept going off the rails because my system prompt was overloaded, so I trimmed it down and put the real instructions in the context window instead. Night and day difference, cut my error rate from like 30% down to maybe 8% in two days. Has anyone else had better luck shifting the weight away from the system prompt?

1 comments

1 Comment

uma_baker881mo ago

Same thing happened to me back in March. I had this giant system prompt with every rule I could think of, and the bot kept ignoring half of it anyway. Trimmed it way down to just the core personality stuff, then started dropping the actual task instructions into the conversation right before asking it to do something. That shift alone fixed most of my context bleeding issues too, where it would forget stuff from earlier in the chat. The dev was right, the model pays way more attention to what's near the end of the context than to the system prompt, at least for the models I'm using. Also helps that I can test different context setups way faster than rewriting the whole system prompt each time. That 30% to 8% drop sounds about right, mine went from like 25% to under 10% in a week.