Software Development: Code That Leaks Data

In part 4, I covered where responsibility lies. Here, I address a problem that sometimes gets little attention yet affects nearly every company that develops custom software: AI in software development and how data can leak through the code itself.

It's almost taken for granted that developers use AI today

It's nearly assumed today that developers use AI in their work. That doesn't mean everyone does, but it's reasonable to expect it.

Since we can assume developers use AI, it's also reasonable to assume other employees do too, depending on their roles. IT teams, for example, use these tools heavily for troubleshooting—they're very good at it and speed things up: you paste in logs, error messages, or configurations and start asking AI questions. There's no real difference between that and a developer—it's often similar data, the same paths, and the same questions about where things went wrong, but I'll stay focused on software development here. It's important to remember that nearly everything that follows applies equally where AI touches real data.

That's where the problem lies, and it doesn't get enough focus: companies' and organizations' data processing agreements exist, but they might not be reviewed with these scenarios in mind.

Most people think about this as a productivity boost—the developer writes code faster because AI completes sentences for them, suggests functions, and finds bugs. That's true, and it's great, but there's another side to this that gets much less attention.

Code is rarely just code

When a developer is experimenting with a new solution, the code often contains real customer data—test data, examples, database records, error messages with content from real records. This is a natural part of development: you can't test whether code works without running it against data that resembles the real thing.

When AI is then asked to help with debugging or improving the code, this data goes along in the request. Sometimes it's an entire table pasted in to show "this is what the problem looks like," or error messages containing ID numbers, email addresses, or other personal information, often without the developer noticing—they're just trying to solve the problem in front of them.

This is the same problem I discussed in part 4 about that single click: one click and the content is gone beyond borders. The difference is that here it's not a system administrator pressing a button that affects the processing environment—it's a developer doing their job, maybe using a personal subscription running in the US, defaulting to sharing it for training in the worst cases.

This goes beyond software development queries anymore

Many of these new tools like Cursor, Claude Code, and similar ones automatically read entire code repositories into context so you can work with them better—not just what you point to, but what these tools think is relevant. This means everything in project folders can be shared without anyone making a conscious decision to send it.

It's not just personal data that leaks

Other things that could leak are technical in nature, like API keys, access credentials, and connection strings buried in the code. Personal information is a problem, but system credentials in queries are valuable intelligence for hackers.

The code that comes back 

Finally, there's the side worth remembering when the focus is on data going out: the code that comes back. AI sometimes suggests code that uses outdated packages, known vulnerabilities, or package names that simply don't exist (a practice called "dependency confusion"), opening doors to attacks. Someone who takes this uncritically isn't just speeding up—they're moving risk directly into code that then runs in production.

Who owns the code that AI writes?

Here's another question worth considering: who owns the copyright and intellectual property of code created with AI assistance?

When a contractor delivers a solution, the client typically assumes they own the code fully unless contracts say otherwise. But if a large part of it was created with AI, who owns the intellectual property? Different tools' terms address this differently, and some reserve rights to use inputs and outputs in various ways. For most clients, this isn't a practical issue, but for companies building competitive advantage on custom software, it can matter. 

This is something I wanted to highlight but not go deep on here—just point out that there are different terms around these issues.

The questions to ask

Whether an outside team is brought in, an internal department handles it, or IT operations uses AI, the questions are the same. Here are the ones I'd want answers to:

•     Which tool is being used? GitHub Copilot, Cursor, Claude Code, or something else?
•     What model is running underneath? GPT, Claude, Gemini, or an open model on different infrastructure?
•     Where is it running? In the US, Europe, or another data center?
•     Does code or data go into queries? What about real data being tested and appearing in test files, error messages, or debug logs sent to the AI?
•     Does the tool automatically read the code repository into context? If so, what sits in project folders that goes along?
•     Do access keys or secrets ever end up in a query? API keys, connection strings, and so on?
•     Is it a business subscription or a personal one? The difference determines whether data is used for training and where it's stored.
•     Who owns the code and intellectual property? And what does the tool's terms say about inputs and outputs?
•     Can you track which code was created with AI later? Without traceability, you can't do copyright or security audits when needed.
•     Is the code AI suggests reviewed? Outdated packages, known vulnerabilities, and package names that don't exist could otherwise make their way straight into the solution.
•     Who bears the cost of these tools? The contractor, the client, or is it rolled into hourly rates or project fees?

Data protection, copyright, and cost allocation are woven together here—far from being just a technical detail. Most important is how real data in code is handled, since traditional workflows don't necessarily solve it. Risk depends on how each line is tested, which tools are used, and what data passes through them.

To apply this to your situation, whether you're buying, building, or running custom software, you can go through the self-assessment below. The goal is to bring these issues to the surface rather than let them hide away.

Why not look deeper into the bag

I'm not saying no one thinks about this—rather, it's easy to let it fall through the cracks because these issues can be complex. Everyone benefits from moving faster. People fear missing out on the latest technology, so they choose not to worry instead.

The contractor benefits from being faster, delivering sooner, and being more likely to land the next project with the client, and they benefit because the work costs less and finishes faster. Neither has a strong incentive to pause and ask these uncomfortable questions about where the data ended up along the way.

That's why this conversation needs to happen now that AI has become a standard part of modern software development. The simple problem is: when no one decides who should manage these issues, the answers hang in limbo.

As a reminder of where each path stands with the EU Data Boundary, you can use the guide from part 3—it's helpful when you evaluate where your AI tools actually operate.

This was part five of six.

Here was the data hiding in code and how data can leak during development, and hopefully this has made you think if you prefer to keep these issues front and center rather than let them slide.

In the final part, part 6, I'll weigh in on model selection and why it rarely works to pick one system and stop thinking about the others. How false security forms when entire sectors chase each other into the same solution. And what I'd want to see in a conversation with a service provider if I were a buyer today.