Is Prompt Engineering Dead?

A video from one of my favorite AI creators, Nate B. Jones (@nate.b.jones), made a claim worth testing: that newer models are so good at interpreting intent that the structure of your prompt no longer matters (Figure 1). Prompt engineering, in other words, is dead.

So I ran a small experiment.

The setup

The task was to generate Page 1, the title page, of a HARPER protocol for a hypothetical RWE study of Shingrix vaccination and dementia risk (maybe not that hypothetical).

HARPER, the Harmonized Protocol Template to Enhance Reproducibility, is a widely adopted protocol template developed by a joint ISPE/ISPOR task force with regulatory stakeholder participation from the FDA, EMA, and PMDA (Wang et al., PDS 2023). Its main use is in the development of RWE and pharmacoepidemiology protocols.

In this experiment, every run was zero-shot. I wanted one clean pass with no back-and-forth with the model. Overall, I compared four prompting strategies:

1. Simple prompt, no context.
2. Added context. The PDS manuscript plus a pointer to the HARPER website.
3. Persona plus a structured prompt (Role, Context, Task, Constraints, Format), no added context.
4. Persona plus the same structured prompt, with added context.

I ran all four on two model families: Claude Opus 4.8 (high) and ChatGPT 5.5 (extended). My primary objective was to see whether the prompting schema, not the model, drove the output.

The results

Finding 1. Every run over-delivered. One pattern held across every cell on both models. The output gave me more than I asked for. I requested a title page as prescribed by the HARPER protocol template. I never handed over a study design, yet every run produced one, or a labeled slot for one, along with comparators, outcome definitions, and target populations I never specified. The request "make a title page" quietly became "make my design decisions for me."

Finding 2. The two families over-deliver differently, with a caveat. Claude committed to specific content, naming a design and filling fields. ChatGPT's structured prompts handed back a scaffold of placeholders for me to complete. The caveat here is that I am a heavy Claude user and built an MCP server for protocol writing, so my Claude account knows HARPER in a way ChatGPT does not. That likely pushed it toward committing to specific content instead of just including placeholders.

Finding 3. The prompting schema mattered, in small and model-specific ways. No clean rule emerged that structure always helps. For ChatGPT, the structured prompt with and without added context were close, but not identical. The added-context run pulled in two regulatory sections, Study Registration and Conflicts of Interest, so context shifted the output rather than leaving it untouched. For Claude, the added-context run drifted toward a sponsor house style and was by far the heaviest in tokens. The clearest schema effect was simpler than any of that. Within ChatGPT, the simple prompt named a study design while the structured prompt left it as a placeholder to specify. Same model, different scaffold, different willingness to commit.

So is prompt engineering dead?

No. It moved.

The old skill was using some sort of structuring syntax (there's several!). Defining roles, context, task, constraints, and format was useful to get a usable answer out of a weaker model. But that part has faded. The models now interpret your intent so well that a plain request often works.

The lesson is not that prompt engineering is dead, but rather that you must specify intent, then audit what the model filled in.

What replaced it is harder. The model will always try to hand you something polished. The open question is what it decided on your behalf to get there, and whether those decisions are yours. Across both families, every run either committed to design choices I never made or papered over them with placeholders. Neither is wrong on its face. Both demand the same thing from you. Read what the model assumed, then own it or override it.

A note on tokens

There is a real benefit to using fewer tokens, but it is mostly cost and latency, not raw model quality. Trimming irrelevant or noisy context can also help the model use what remains more reliably. The caveat runs the other way too. Under-specifying has a failure mode, and you can see it here. With no context, the model had to invent an entire study design to fill the gap. There is even a repo that leans all the way into brevity by helping you prompt like a caveman (JuliusBrussee/caveman). Why use many words when few words will do? Fun, and could be useful, but few words also means more room for the model to guess.

The fine print

This was a single pass per cell, four strategies across two model families with one task. This experiment was purely exploratory. Also, the token ranges above for Claude are rough model estimates, since the desktop app has no counter and models count their own tokens poorly. The ChatGPT times are single-run wall-clock on a chat interface, which tracks reasoning effort and server load more than prompt length, so I would not lean on them. And the Claude runs also carry whatever my personal account and protocol-writing work have taught it about HARPER, which a fresh account would not likely share. A clean rerun would use fresh accounts, several passes per strategy, and exact token counts from the API usage field.

A disclaimer to this experiment is that this was just one use case, and it should not be interpreted as giving the same result if we were generating code, or cleaning up a dataset.

Takeaway: prompt engineering is not dead, but in the absence of some intent or direction, models will try their best to fill in the gap.