Chapter 6 Using generative AI to code

By Jade Benjamin-Chung and Anna Nguyen

AI can be enormously useful in writing code for research. Currently, there are two main ways to use it, and the distinction is important:

• Chat-based help: asking a chatbot to explain a function or debug a pasted error in an app or website. You copy/paste code back and forth. If data is ever provided to it, it must be manually uploaded. You have a high degree of agency over data access and what code to run, when, and where.
• Agentic coding tools run directly in your terminal or code editor (e.g., RStudio), can read and edit files across your project, execute code, run tests, and iterate on its own output — all with a degree of autonomy that you grant it. It can plan a multi-step task, write and execute code across several files, install packages, and run analyses with minimal intervention. This is powerful, but it also means more can happen without your direct review at each step. That’s why coding with these tools is called “vibe coding”. Agentic tools include Claude Code, Codex, Gemini CLI, Cursor, and Github Co-Pilot.

Both modes are useful, but they each pose different risks to learning and to research safety and accuracy. This chapter includes our Lab Policy on use of AI in Research and then discusses the rationale.

6.0.1 Benjamin-Chung Lab Policy on use of AI in Research

• Lab members must use Stanford-licensed enterprise AI tools (Claude for Education, ChatGPT Edu, Gemini Enterprise) for all research-related work involving lab data, code, or drafts. These tools carry Stanford’s data protection and institutional agreements; personal/free-tier accounts do not.

• Tool choice must match the data risk level and requirements in the IRB protocol and Data Use Agreement, if applicable.

  • Low risk (e.g., public data, non-sensitive code, general drafts): any Stanford-licensed enterprise tool may be used.
  • Moderate risk (e.g., de-identified data, unpublished results, internal code, grant/manuscript drafts): Stanford-licensed enterprise tools only — no personal or free-tier accounts.
  • High risk / PHI (e.g., identifiable or participant-level data, protected health information): only tools explicitly approved for this risk tier may be used, and only after confirming with Jade that the specific tool and use case are permitted under the project’s IRB protocol and data use agreement. Some Stanford-licensed tools may not be approved for PHI use even if generally approved for lab work — check before proceeding.

• No lab data, code, or manuscripts can be used in non-approved tools. This includes participant data (even de-identified), unpublished results, grant drafts, IRB materials, and internal code — never paste or upload these into a personal ChatGPT account, a non-Stanford API key, or any tool outside the approved list. If a project genuinely requires a tool outside the approved list, you must get explicit permission from Jade in writing before use.

• Agentic tools (Claude Code, Codex, Cowork, Gemini CLI, etc.) require extra caution. Because these tools can read, write, and execute code or take actions autonomously, always: Review generated code/output before running or committing it. Never grant an agentic tool access to production data, credentials, or shared drives without lab PI approval. Disclose if an agentic tool was used to produce analysis code, and document this in commit messages or code comments.

• Follow lab guidelines on Agentic AI permissions: Always leave the tool’s permission mode on default (the setting that asks before making edits or running commands) — never switch to a mode that lets the AI act without asking first. This applies to the shared project .claude/settings.json, which is the only settings file you should edit; do not create a personal settings.local.json or a user-level ~/.claude/settings.json override for lab projects.

• Humans are responsible for Git commit and Git push, not AI. Because research errors are analytical rather than technical and often can’t be caught by automated checks, every lab member must personally write and execute their own commits and pushes — this keeps the reasoning behind each change with the person responsible for it, and builds the judgment mentees need to develop.

• When in doubt, ask first. It’s always acceptable — and preferred — to check with Jade before using a new tool or changing permission settings, rather than after.

• You must complete a short quiz and agree to the lab’s AI use policy before using any AI tool for research in the Benjamin-Chung Lab AI Use in Research Permission Form

• Use of AI tools in research must be documented in the Benjamin-Chung Lab AI Use in Research Documentation Form

6.1 Three key issues

1. Learning and skill development. Heavy reliance on AI coding tools may impede the development of coding fluency and statistical/computational intuition — particularly for trainees who are still building these skills.
2. Research accuracy. Using these tools on datasets, analysis types, or workflows you aren’t already familiar with raises the risk of subtle errors going undetected, because you may lack the intuition to recognize when something looks wrong.
3. Data protection. These tools can access files on your machine, which introduces new and sometimes subtle data-handling risks in a research context governed by IRB protocols, data use agreements, and participant privacy commitments.

Each is discussed in more detail below, along with suggested safeguards.

6.2 Bottom line

AI coding assistants are a powerful addition to our computational toolkit, but they are not a substitute for the judgment that comes from hands-on coding experience, nor for careful attention to data governance. Use them deliberately: lean on them more for familiar, low-stakes tasks, and less for unfamiliar or high-stakes ones, with unit tests, replication checks, and careful permission review as standing practice regardless of experience level.

When in doubt about whether a workflow is compliant, ask Jade before proceeding.

6.2.1 Research accuracy and the value of experience

In 2026, most seasoned quantitative researchers have been writing code for a decade or more — tens of thousands of cumulative hours. There’s a popular “rule” that it takes 10,000 hours to master a new skill– equivalent to 5.5 years of full time work. That experience builds a kind of gut instinct: when you look at code, or watch it run, you develop a sense for what might be going wrong, even when nothing throws an obvious error. Examples of silent failures:

• A merge that drops rows unexpectedly
• A factor variable that got silently releveled
• A date field parsed in the wrong timezone

After dealing with these silent errors hundreds or thousands of times, the human brain recognizes the shape of a subtle bug before you can necessarily articulate exactly what it is. That experience is what lets seasoned researchers know what safeguards to put in place:

• Which unit tests are worth writing
• Which portions of a work flow require human or independent agentic replication checks
• Which outputs deserve a manual sanity check versus which can be trusted.

AI coding tools do not have this judgment about your specific project. They can write tests if asked, but deciding which tests matter for a given dataset or analysis is still a human judgment call, grounded in domain and data familiarity.

This means the risk of using these tools is not constant. As your mentor, I am working on developing systems to help you learn when you might be in the danger zone. When you’re in the danger zone, you are outside your area of established expertise, working with a new data source, an unfamiliar analysis method, or a computational workflow you haven’t run by hand before. The challenge is, when you’re new to a project or to a field of study, you’re always in the danger zone.

6.2.2 Impeding learning? An open question

Little to nothing is currently known about the pedagogy of learning to code with AI assistance versus learning to code without it. If you never manually write code — if you never sit with a bug for twenty minutes and slowly figure out that the join key had a leading whitespace — how do you develop the gut instinct described above? We don’t know.

It’s also possible that this concern will turn out to be less important than it seems. The future of research computing may not involve much hands-on human coding at all. But even if that’s true, we don’t yet know how to teach someone — a rotation student, a new Research Assistant, a first-year PhD student — to “vibe code” well: to direct an AI tool effectively, to catch its mistakes, and to build sound judgment about when to trust its output. Until the pedagogy of computational research with agentic AI exists, in our lab, we consider risks of agentic AI use to be high for early-career skill-building to be high.

6.2.3 Data safety in a research context

Uploading data to a chat-based AI tool poses risks, including:

• Breaches of participant confidentiality
• Violations of IRB protocols and data use agreements,
• Loss of control over how the data is stored, retained, or used to train future models. Even de-identified datasets can sometimes be re-identified when combined with other information, and many AI tools’ terms of service permit them to retain or process uploaded content in ways that are not fully transparent to the user.

The risks are substatially higher for agentic tools such as Claude Code that run locally and, with your permission, can read and modify files on your laptop — including, potentially, files containing participant-level data governed by IRB protocols and data use agreements.

The practical risk is subtler than “the AI leaks your data.” During a coding session, the tool may generate hundreds of permission requests (to read a file, run a command, install a package, access the network, etc.). These prompts are sometimes phrased in dense technical language that is genuinely hard to parse quickly. It is easy to click “allow” reflexively and inadvertently grant access to a raw data directory, or permission to send data over the network (e.g., installing a package, calling an external API), without fully registering what you approved. This is not primarily a malicious-AI risk; it’s a human-factors risk.

As of June 2026, Stanford’s Approved Connectors for AI Services have different different risk categories by different products. Note that Google and OpenAI can be used with High Risk Protected Health Information (PHI) data, while using PHI with Claude requires routing through the Stanford AI API Gateway. If you are not sure whether you your project is using PHI, check with Jade before proceeding.

6.2.4 Understanding agentic AI permission prompts

Note: the following is based on Claude Code, but other agentic coding tools have similar permission prompts.

The prompt and approval interface differs depending on whether you’re working in a terminal or a GUI/editor-based tool, but the underlying pattern is the same:

• AI’s own plain-English summary of the action, not a guarantee of what the command actually does. It’s usually accurate, but it can omit exactly the detail — a hardcoded file path, a write to an unexpected directory — that matters for review. Read the actual command or diff underneath, not just the summary.

• Long commands or file paths that are often truncated. Claude Code may write a snippet of code and ask you for permission to run it, but it might not be in your project’s language, and often these code snippets are not saved in your repository at all. For example, if you’re working in an R codebase and ask it to double-check a calculation or validate an assumption, it might quietly reach for Python — writing a small throwaway script to cross-check the logic, run it, read the output, and then report back to you in English. That scratch script often never gets saved as a file you’d see in a pull request; it’s more like a post-it note the AI wrote to itself, used once, and threw away. The problem is that if you don’t read Python, you have no way to actually verify what that post-it note said or did — you’re not reviewing the check, you’re reviewing Claude’s summary of the check. That might be perfectly fine for something low-stakes, but for a step that’s supposed to be your safeguard against a subtle analytic error, it’s worth pausing on: if you can’t read the language a verification was written in, you aren’t really approving the verification — you’re approving a description of it, written by the same tool whose work it’s supposed to be checking.

Here’s an example from a code base entirely written in R, where Claude Code asked for permission to check for duplicates using a “post-it” note in Python.

• “Always allow” sits right next to the one-time approval, in the terminal as option 2 of 3, and in the GUI as a persistent button with its own keyboard shortcut. It’s one keystroke away from a single-use approval, which is exactly what makes reflexive over-granting easy, especially late in a long session.

When Claude Code (or a similar AI coding assistant) asks for permission to run a command, recognizing the command itself — not just reading the plain-English gloss above it — is the fastest way to know what you’re actually approving. The tables below are organized by risk level.

Command	What it does	Why it matters
Low-risk commands
ls, cat, head, tail	List directory contents; print file contents
cd, pwd	Change / print working directory
cp	Copy files
| (pipe)	Send one command’s output into another as input
&&	Run the next command only if the previous one succeeded
High-risk commands
rm	Delete files	No undo. Watch for this near raw data paths
rm -rf	Force-delete a directory and everything in it, no confirmation	Riskiest pattern to recognize — recursive + force, can wipe a folder tree silently
mv	Move files	Can silently overwrite a destination file
>, >>	Redirect output to a file	> silently overwrites the whole target file; easy to miss mid-command
sed -i, awk	Edit files in place / process text	-i modifies the original directly — no diff preview
chmod	Change file permissions	Can make a data file world-readable/writable
chown	Change file ownership	Rare in normal analysis work; worth a second look
find ... -exec	Search files, run a command on each match	Can apply rm/chmod across many files — read the -exec clause carefully
curl, wget	Download content from a URL	Network access — relevant if data shouldn’t leave the machine
scp, rsync	Copy files to/from a remote machine	Data transfer off-machine — scrutinize unfamiliar destinations
git push, git commit	Send code / record a commit	Notice if Claude is committing on your behalf
sudo	Run with admin privileges	Should almost never be needed for data analysis
export	Set an environment variable	May set API keys/credentials — check what’s exported

6.3 Best practices

6.3.1 For learning and skill development

• Spend meaningful time writing core analysis code by hand before leaning on agentic tools for that same type of task — e.g., write your first TMLE or IPW pipeline yourself before asking an AI tool to generate one.
• When using an AI tool for a task you’re still learning, ask it to explain its reasoning, not just produce output. Treat it as a tutor for code you then re-derive yourself, not a black box.
• Before fully letting agentic AI write a script for you, try writing out detailed pseudo-code – aka, a step-by-step plan for what the code should do. Then ask the AI to implement that plan, and check that it actually does what you intended.

6.3.2 For research accuracy

• Match AI use to familiarity. Use these tools most freely for boilerplate, refactoring, and workflows you already know well. Increase manual scrutiny — or avoid heavy AI use — for new datasets, unfamiliar methods, or unfamiliar computational environments.

• Use unit tests for AI-assisted analysis code, especially for anything going into a manuscript or grant deliverable. Ideally, you write these, not an agent, and you confirm the tests actually check the right things. Learn more in the chapter: Data validation and unit testing in Chapter 12.

• Spot-check intermediate outputs, not just final results — row counts after joins/merges, variable distributions before and after recoding, and sample sizes at each analytic step.

• Set up automated logging to track what files agents have edited. Read about it here.

• Perform independent replication for key analyses: have a second team member reproduce a critical result from raw data, ideally without heavy reliance on the same AI-generated code path. In the kind of research we do, the code most likely to lead to a lack of replication arises during data processing, i.e., converting survey data to the form needed for analyses. Two reasonable people or agents could make different assumptions, none of which are wrong, that result in different data. Replication brings these differences to light and allows them to be transparently reconciled. Learn more here.

• Practice password safety: Don’t put passwords, SSH keys, or API tokens in any file that an AI tool can read. Don’t paste passwords or keys into code or notebooks. Don’t leave credential files (like SSH keys) in folders that an AI coding tool has access to. Keep passwords and keys in a password manager, not in your files.

6.3.3 For data protection

• Ensure your datasets are backed up in locations that cannot be accessed by agentic AI tools to prevent accidentally overwriting or deleting data.
• Never point an AI coding tool at a directory containing raw participant-level data unless you have specifically confirmed this is permitted under the relevant IRB protocol and data use agreement — and ideally, work with de-identified or synthetic data during code development, applying finished code to real data only in a controlled step.
• Read permission prompts before approving them, especially requests to access the network, install packages, or read/write outside the current project folder.
  
• Use project-level permission scoping instead of granting broad or session-wide access. This is why our lab uses a shared .claude/settings.json (allow-listing specific directories and commands, mode set to default) rather than individual overrides (see below). Don’t work around it with a personal settings.local.json or broader access than the project file grants.

When in doubt about whether a workflow is compliant, ask Jade before proceeding.

6.4 Configuring permissions for agentic AI

This section covers Claude Code’s permission system. This guidance won’t necessarily work in other agentic tools (Cursor, Codex CLI, OpenCode, etc.), each of which has its own config format and varying hook support. If you’re using a different agent, you would need to look up that tool’s own permissions/hooks documentation and check with Jade to ensure compliance.

6.4.1 Permission modes

Claude Code has several permission modes that set the baseline approval behavior for a session. default mode requires explicit approval for every action other than reads — every:

• file edit
• shell command
• network call

gets a permission prompt. This is the only mode that guarantees a human is in the loop, which matters given that our work involves participant data and code that feeds into published analyses. Other modes exist, but they trade oversight for speed in ways that aren’t appropriate for research in our lab without explicit permission from Jade.

6.4.1.1 Setting up permissions

Below, we recommend lab-specific permissions that apply to all projects. They adjustthe standard default permissions in Claude to:

• Be more permissive for named safe tools
• Be more restrictive for accessing raw data file paths

Permission rules live in a permissions block in settings.json and let you pre-approve safe, repetitive actions while keeping prompts (or hard blocks) on anything sensitive. Rules are written as Tool(specifier) patterns and are evaluated in a fixed order — deny, then ask, then allow — with the first match winning. A deny rule always wins even if a more specific allow rule also matches; you cannot carve an allow list exception out of a deny rule.

Note: lab policy is that humans are responsible for Git commit and Git push, Not AI. Though in industry, agents are increasingly performing Git commits on their own, research code differs from software engineering in ways that change how much autonomy is appropriate. In engineering, continuous integration and test suites can verify correctness before a commit lands, and multiple reviewers typically stand between an AI-drafted commit and production.

In our lab, the errors that matter most are analytical, not technical — a mishandling of a missing observation, an incorrect ICD-11 code, an inappropriate mediator — and no automated check can catch these. Further, most of our team are mentees still building that judgment, and if an AI agent can commit once a person clicks “approve,” the person approving may not yet have the expertise to catch what’s wrong. Requiring each person to write and run their own git commit/git push keeps a deliberate human decision point before any change becomes part of our shared code base and promotes learning.

For each code repository, set up permissions by saving a .claude/settings.json in each project’s root folder. Push it to Github so it’s version-controlled and shared by everyone working in that repo:

{
  "permissions": {
    "defaultMode": "default",
    "allow": [
      "Read(./**)",
      "Bash(git show*)",
      "Bash(git status*)",
      "Bash(git diff*)",
      "Bash(git branch*)",
      "Bash(git log*)",
      "Bash(git fetch*)",
      "Bash(Rscript *)",
      "Bash(quarto *)"
    ],
    "ask": [
      "Bash(git checkout*)",
      "Bash(pip install *)",
      "Bash(pip3 install *)",
      "Bash(install.packages(*)*)",
      "Bash(curl *)",
      "Bash(rm *)",
      "Bash(npm install *)",
      "Bash(conda install *)",
      "Bash(apt-get install *)",
      "Bash(apt install *)",
      "Bash(python *)",
      "Bash(python3 *)"
    ],
    "deny": [
      "Bash(git add*)",
      "Bash(git commit*)",
      "Bash(git push*)",
      "Bash(git merge*)",
      "Bash(git rebase*)",
      "Bash(git reset*)",
      "Read(./data/raw/**)",
      "Edit(./data/raw/**)",
      "Write(./data/raw/**)",
      "Bash(rm -rf *)"
    ],
    "additionalDirectories": []
  }
}

What this does, line by line:

defaultMode: "default" — pins the session to the mode described above, so the setting travels with the repo.

allow — actions that proceed without a permission prompt because they’re read-only or low-risk:

• Reading any file in the project (Read(./**))
• Checking git status/diff/log/show/branch/fetch (informational or non-destructive — none of these rewrite history or touch the working directory’s uncommitted changes)
• Running R scripts and Quarto documents within this project repo (Rscript *, quarto *) — the core of most analysis work. If a specific script is particularly computationally intensive or destructive, you can move that script’s path from allow to ask (e.g. Bash(Rscript scripts/expensive_model.R)).

ask — actions that change state or reach outside the repo, so we want a checkpoint rather than a hard block:

• git checkout — can silently discard uncommitted work, so it’s gated even though it doesn’t touch remote history.
• Installing packages or dependencies (pip install, install.packages, npm install, conda install, apt install/apt-get install) — these reach outside the repo and change your environment.
• curl — any outbound network request.
• rm (plain delete) — file deletion that isn’t the -rf pattern blocked outright below.
• Running Python scripts (python *, python3 *) — Python isn’t part of our core R-based analysis workflow. Claude typically writes these on its own to run quick tests or check its work. For new users of agentic AI, it’s best to approve these individually at first so you can see what it’s doing; once you trust a specific recurring script, you can move its exact path to allow.

deny — hard blocks, regardless of any other rule:

• Git history commands (git add, git commit, git push, git merge, git rebase, git reset) — humans are responsible for these, not AI. Lab members must personally write and execute their own commits, pushes, merges, rebases, and resets — this keeps the reasoning behind each change with the person responsible for it, and builds the judgment our team needs to develop. Claude can still draft a suggested commit message or explain what changed, but cannot run these commands itself, even with approval.
• Reading, editing, or writing anything in data/raw/ (our raw, unprocessed field data — see the hook section for why a path rule alone isn’t sufficient here, and remember to update this path per-project)
• rm -rf (recursive force delete) — blocked outright rather than merely asked, since this is the highest-consequence single command available.

Nothing in ask or allow can override any of these.

additionalDirectories — left empty here; add paths only if a project genuinely needs to read/write outside its own folder (e.g., a shared reference-data directory), and prefer read access over write access when you do.

6.4.1.2 Tailor the template to your project

• Modify the data/raw/** path in the deny block Enter the path to your project’s raw, unprocessed field data. If a project has more than one raw-data location, add a separate line for each.

  ⚠️ This step matters more than it might look — if the path doesn’t match reality, the deny rule silently protects nothing.

• Check whether the project needs additional allow tools Does the project need to list tools in allow other than git, Rscript, python, and python3? If so, modify this section.

• Review the ask list This assumes a standard git workflow. Bash(curl *), Bash(rm -rf *), and the overall structure of the file are generic safety defaults and typically don’t need to change per project.

6.5 Disclosing use of generative AI in research

Lab members must disclose GenAI use in research at four points:

1. Within the lab, to Jade using the Benjamin-Chung Lab AI Use in Research Documentation Form
2. To collaborators, whenever their data or materials are involved
3. To journals, at time of submission (see below)
4. To funders, when their policies require it.

6.6 Know your target journal’s AI policy before you start

Before you start a project that will be published in a peer-reviewed journal, keep in mind that you will likely need to disclose any AI assistance in your methods section, acknowledgments, or cover letter. Journal policies on AI use vary widely (see table below).

• Before relying on AI tools for a substantial part of your workflow, check the current policy for your intended journal to learn what it does and doesn’t cover — a policy permitting AI for language editing is not the same as one permitting it for analysis or figure generation.
• Document what tools you use throughout the project so you can disclose them at the time of journal submission.
• Given how quickly tools are changing, you’ll need to check the policy again closer to submission.

Here is an example of the type of acknowledgement language you might include in a manuscript:

Claude (Sonnet 4.6, Anthropic) was used for language editing and improving clarity of expression. Claude Code (Sonnet 4.6, Anthropic) was used to assist with data processing, statistical coding, and figure and table generation. All code was reviewed by two independent team members to verify accuracy and reproducibility. These tools were not used to develop research questions, interpret findings, or draw conclusions. The authors take full responsibility for the interpretations, conclusions, accuracy, and integrity of the work.

The following was generated by Claude in July 2026:

Table 6.1: Journal AI policies as of proposal writing — policies change, verify current policy before heavy AI use.

Publisher	Journals	Policy (🚫 not permitted / ✅ allowed w/ disclosure / ❓ not addressed)
Policy documented
Springer Nature	Nature, Nat. Comms/Med/Health, Sci. Reports, BMC, Discover Env.	🚫 Authorship; ✅ copy-edit w/ disclosure; name tool + use in Methods
PLOS	Glob. Public Health, Medicine, NTD	🚫 Authorship; tool/use/validation disclosed in Methods/Ack.
Elsevier/Lancet	Lancet GH/PH, eBioMed, eClinicalMed, Vaccine, IJHEH, Heliyon	🚫 Authorship; ✅ copy-edit w/ disclosure; declaration before references
BMJ	BMJ Open, BMJ Glob. Health	🚫 Authorship; described in manuscript + flagged at submission
Oxford UP	AJE, IJE, Epi Reviews, CID, JID, OFID	🚫 Authorship; ✅ coding/analysis w/ disclosure; cover letter + Methods/Ack.
Wiley	Stats in Med, TMIH, Paediatric & Perinatal Epi	🚫 Authorship; ✅ copy-edit w/ disclosure; 🚫 data alteration; Methods/Ack. if substantial
ACS	Env. Sci. & Tech.	🚫 Authorship; disclosed in Ack., Methods if substantial
ASM	Applied & Env. Micro.	🚫 Authorship; cover letter + Materials & Methods
AAAS/Science	Science	🚫 Authorship; cover letter + Ack. + full prompts in Methods
F1000	Gates Open Research	🚫 Authorship; not detailed — check w/ editor
Policy not yet located — check before use
ASTMH	AJTMH	❓ Not located — verify before use
APHA	AJPH	❓ Not located; 🖼️ AI images must be disclosed
eLife	eLife	❓ Not located — verify before use
NIEHS	EHP	❓ Not located — verify before use
Wolters Kluwer	Epidemiology	❓ Not located — verify before use
Cambridge UP	J. Clin. & Trans. Sci.	❓ Not located — verify before use

Sources: publisher policy pages accessed July 2026 (springernature.com, journals.plos.org, elsevier.com, authors-bmj-com, academic.oup.com, wiley.com, researcher-resources.acs.org, journals.asm.org, science.org, f1000research.com). Cells marked “not located” should be re-checked directly before submission — this space is evolving fast.