Software Development: Code That Leaks Data

In part 4, I discussed where the responsibility lies. Here, I address a problem that sometimes receives little attention but affects almost every single company that has software developed for them: artificial intelligence in software development and how data can leak through the code itself.

It is almost given that developers use AI today

It is almost a given today that developers use artificial intelligence in their work. That is not to say that everyone does, but rather that it can be assumed.

Since it can be assumed that developers use AI, it is almost possible to assume that other employees do so as well, based on the nature of their tasks. IT personnel, for example, work extensively in troubleshooting, where these tools are very effective and speed up the process: you input logs, error messages, or configurations and start asking the AI questions. The difference between that and the developer is essentially non-existent; this is often similar data, the same channel, and the same questions about where it ended up, but I will stick to software development here. It is good to keep in mind that almost everything that follows applies equally wherever AI touches real data.

Therein lies the problem that is not getting enough focus: processing agreements of companies and institutions exist but are potentially not reviewed with these scenarios in mind.

Most people think of this as an increase in productivity; the developer writes code faster because the AI completes the sentences for them, suggests functions, and finds errors. That is true, and it is great, but there is another side to this that receives much less attention.

Code is rarely just code

When a developer is experimenting with a new solution, real customer data often resides in the application, such as test data, examples, database entries, and error messages containing the content of a real transaction. This is a normal part of development: you cannot test whether the code works without running it on data that resembles the real thing.

When the AI is then asked to help with debugging or improving the code, this data goes along with the prompt. Sometimes this is an entire data table pasted to show "this is what the problem looks like," or error messages containing national ID numbers, email addresses, or other personally identifiable data, and often without the developer actively noticing, as they are just trying to solve the problem in front of them.

This is the same issue I discussed in part 4 regarding that single checkbox: one click and the content has gone abroad. The difference is that here, it is not a system administrator pushing a button that affects the processing area, but a developer doing their job, perhaps using a personal subscription running in the US, which by default shares data for training in the worst cases.

This no longer applies only to software development prompts

Many of these new tools, such as Cursor, Claude Code, and similar applications, read entire code repositories into context automatically to work with them more effectively—not necessarily just what you point to, but what these tools deem relevant at any given time. This means that everything in project folders can be sent without anyone making a conscious decision to do so.

It is not just personal data that leaks

Other things that could leak are of a technical nature, such as API keys, credentials, and connection strings residing in the code. Personal data is an issue, but system credentials in prompts are highly valuable information for hackers.

The code that comes back 

Finally, there is the side to remember when the focus is on the data going out: the code coming back. AI sometimes suggests code that uses outdated packages, known vulnerabilities, or package names that simply do not exist (slopsquatting), which opens doors for attacks. Anyone who accepts this uncritically is not only rushing, but is importing risk directly into the code that will subsequently be used in operations.

Who owns the code that the AI writes?

Here is another question one has pondered: what is the copyright and ownership of code created with the help of artificial intelligence?

When a contractor delivers a solution, the client usually assumes they own the code entirely unless agreements state otherwise. But if a large part of it was generated by AI, who owns the intellectual property? The terms of various tools answer this differently, and some reserve the right to use input and output in various ways. For most clients, this is not a practical problem, but for companies that build competitive advantage on custom software, this can matter. 

This is an issue I wanted to draw attention to without going into great depth here, and to point out that there are differing terms surrounding these matters.

The questions that should be asked

Whether an external team is hired for the project, an internal department does the work, or IT operations uses AI, the questions remain the same. Here are the ones I would want answers to:

•     Which tool is being used? GitHub Copilot, Cursor, Claude Code, or something else?
•     What model is running underneath? GPT, Claude, Gemini, or an open model on other infrastructure?
•     Where is it running? In the US, in Europe, or in another data center?
•     Do code snippets or data go into prompts? What about real data used for testing that appears in test files, error messages, or debug files sent to the AI?
•     Does the tool automatically read the codebase into context? And if so, what lies in the project folder that gets sent along?
•     Do access keys or secrets ever end up in a prompt? API keys, connection strings, etc.?
•     Is a corporate or personal subscription being used? The difference determines whether the data is used for training and where it is stored.
•     Who owns the code and the intellectual property? And what do the tool's terms say about input and output?
•     Can it be shown retrospectively what code was generated by AI? Without traceability, neither copyright nor security audits are possible when it matters.
•     Is the code suggested by the AI reviewed? Outdated packages, known vulnerabilities, and non-existent package names could otherwise get directly into the solution.
•     Who bears the cost of these tools? The contractor, the client, or is it included in the hourly rate or project fee?

Here, data protection, copyright, and cost allocation intertwine into a single knot that is far from being just a technical detail. The most important thing is to look at how real data in the code is managed, as traditional processes do not necessarily solve it; risk is determined more by how every single line is tested, which tools are used, and what data flows through them.

To apply this to your own situation, whether you buy, develop, or operate custom software, you can go through the self-assessment below. The purpose of self-assessment is to bring these matters to the surface rather than letting them simmer underneath.

Why aren't more people looking inside the bag?

I am not claiming that no one thinks about this, but rather that it is easy to let it slide because of how complex these issues can be. Everyone gains from rushing, so people tend to feel like they are missing the boat if they do not utilize the latest technology, choosing instead not to worry.

The contractor benefits from being faster, delivering sooner, and being more likely to get the next project from the client, and the client benefits because the project costs less and is finished sooner. Neither has a strong incentive to pause and ask those uncomfortable questions about where the data ended up along the way in the project.

Therefore, this conversation needs to be opened up, as AI has become a natural part of modern software development. The problem is simply that when no one decides who should manage these matters, the answers are left hanging.

To review where each path stands regarding the EU Data Boundary, you can use the roadmap from part 3, which is useful when assessing where your AI tools actually process data.

This was part five of six.

Here we discussed the data hiding in the code and how data can leak during development, and hopefully, this was food for thought for some who prefer to have these matters on the surface rather than letting them simmer underneath.

In the final part, part 6, I will address model selection and why it rarely works to choose one system and stop thinking about the others. How a false sense of security is created when entire sectors follow each other into the same solution? And what I would personally want to see in a conversation with a service provider if I were a buyer today.