Every dataframe that holds a customer ID, a device fingerprint, or a free-text field is a GDPR object. The regulation applies to the work you do in notebooks, feature stores, and training pipelines, not only to the legal team’s policies. This guide maps each GDPR obligation to a concrete data science task, with code you can adapt.
Quick Takeaways
- Personal data is broad. Any data relating to an identifiable person counts, including cookie IDs, IP addresses, and pseudonymized records.
- Six principles drive your workflow: purpose limitation, data minimization, accuracy, storage limitation, integrity/confidentiality, and accountability (Article 5).
- Pseudonymization is not anonymization. Hashed IDs reduce risk, but the data stays regulated if re-identification is reasonably possible.
- Document everything. Lawful basis, feature rationale, retention, and model decisions must be demonstrable on request.
| Your Task | GDPR Hook | What To Do |
|---|---|---|
| Collecting/ingesting data | Art. 5(1)(b), Art. 6 | Record purpose and lawful basis before ingestion |
| Feature engineering | Art. 5(1)(c) | Drop fields you cannot justify |
| Joining datasets | Art. 5(1)(b), Art. 9 | Check for purpose drift and special categories |
| Model training | Art. 6, Art. 35 | Run a legitimate interest assessment or DPIA |
| Automated scoring | Art. 22 | Provide human review and explanations |
| Data deletion requests | Art. 17 | Propagate erasure to features and training sets |
Who Is Responsible: Controller, Processor, and You
GDPR assigns legal duties to the controller (decides why and how data is processed) and the processor (processes on the controller’s behalf). Individual data scientists rarely carry the legal liability. You carry the operational duty to build systems that let your organization comply.
In practice, that means three things:
- You are the person who knows which columns feed which model.
- You are often the only one who can tell whether a field is a proxy for a protected attribute.
- You can build privacy by design (Article 25) into pipelines, or you can create a compliance problem the legal team discovers later.
What Counts as Personal Data in a Dataset
Personal data is any information relating to an identified or identifiable natural person. Direct identifiers are obvious: name, email, national ID. Indirect identifiers cause most violations:
- IP addresses, cookie IDs, advertising IDs
- Precise geolocation traces
- Rare combinations such as ZIP code + birth date + gender
- Free-text fields (support tickets, reviews) that contain names or addresses
Special category data (Article 9) receives stricter treatment. It covers health, biometrics, ethnicity, political opinions, religion, union membership, sexual orientation, and genetic data. You can infer these categories from innocuous features, so audit your feature space and not only your schema.
Step 1: Establish a Lawful Basis Before You Touch the Data
Every processing purpose needs one of six lawful bases under Article 6. The three you will meet most often:
| Lawful Basis | Typical Data Science Use | Key Constraint |
|---|---|---|
| Consent | Personalization, marketing models | Must be specific, informed, and withdrawable |
| Contract | Fraud checks on a purchase, credit scoring for a loan | Processing must be necessary for the contract |
| Legitimate interest | Churn prediction, security analytics, model improvement | Requires a documented balancing test |
For model development, legitimate interest is the most common route. The EDPB’s Opinion 28/2024 says a controller can rely on it for AI development and deployment, subject to a case-by-case assessment. The opinion also requires a three-step test: a legitimate interest, the necessity of the processing, and a balancing of interests where data subject rights must not override the controller’s interests.
Write the assessment before you train. It is far harder to justify retroactively.
Purpose Limitation: The Reuse Trap
Data collected for one purpose cannot be silently repurposed. Support-ticket data gathered to resolve issues does not automatically become training data for a sales-propensity model. Before reusing a dataset, check whether the new purpose is compatible with the original one. If it is not, you need a new lawful basis or fresh consent.
Step 2: Minimize Data and Features
Article 5(1)(c) requires data to be adequate, relevant, and limited to what is necessary. For data scientists, this is a feature selection discipline. Each feature needs a defensible reason to exist.
import pandas as pd
# Declare the purpose-bound feature allowlist explicitly.
# Anything not listed never enters the training dataframe.
CHURN_MODEL_FEATURES = [
"tenure_months",
"monthly_spend",
"support_tickets_90d",
"plan_type",
]
TARGET = "churned"
def minimize(df: pd.DataFrame) -> pd.DataFrame:
"""Return only allowlisted columns; fail loudly on missing ones."""
missing = set(CHURN_MODEL_FEATURES + [TARGET]) - set(df.columns)
if missing:
raise ValueError(f"Missing expected columns: {missing}")
# .loc with an explicit list drops everything else, including raw identifiers
return df.loc[:, CHURN_MODEL_FEATURES + [TARGET]].copy()
raw = pd.read_parquet("customers.parquet")
train_df = minimize(raw)
An explicit allowlist beats a blocklist. A new column added upstream cannot leak into your model by accident.
Step 3: Pseudonymize Correctly
Pseudonymization replaces direct identifiers with tokens so that data cannot be attributed to a person without additional information kept separately. It is a recommended safeguard. It does not take data out of GDPR scope by itself.
A plain SHA-256 hash of an email is weak. Anyone with a list of candidate emails can recompute the hashes. Use a keyed hash (HMAC) with a secret stored in a secrets manager, never in the repository.
import hmac
import hashlib
import os
import pandas as pd
# Load the key from a secrets manager or environment, never hard-code it
SECRET_KEY = os.environ["PSEUDONYM_KEY"].encode("utf-8")
def pseudonymize(value: str) -> str:
"""Deterministic keyed hash: same input -> same token, irreversible without the key."""
normalized = value.strip().lower()
return hmac.new(SECRET_KEY, normalized.encode("utf-8"), hashlib.sha256).hexdigest()
df = pd.read_parquet("customers.parquet")
df["customer_token"] = df["email"].map(pseudonymize) # stable join key across datasets
df = df.drop(columns=["email", "full_name", "phone"]) # remove direct identifiers
Keep the key in a different system, under different access controls, from the pseudonymized data. That separation is what gives pseudonymization its protective value.
Pseudonymization vs. Anonymization
| Aspect | Pseudonymization | Anonymization |
|---|---|---|
| Re-identification | Possible with extra information | Not reasonably possible by anyone |
| GDPR applies? | Yes | No |
| Utility for analytics | High (stable joins, longitudinal analysis) | Often reduced |
| Typical techniques | HMAC tokens, tokenization vaults | Aggregation, generalization, noise addition |
| Main risk | Key compromise, linkage attacks | Overestimating how anonymous the data is |
The legal picture here is in flux. The European Commission’s Digital Omnibus proposal would clarify that pseudonymised data held by an entity that cannot re-identify the individual is not personal data for that entity. That position conflicts with the EDPB’s January 2025 guidance, which treats pseudonymised data as always personal. The Omnibus is a legislative proposal and I could not confirm its final adopted text, so verify its current status with your legal team. Until the law changes, treat pseudonymized data as personal data.
Test Your Anonymization with k-Anonymity
If you claim a dataset is anonymous, test it. A quick k-anonymity check counts how many records share each combination of quasi-identifiers.
import pandas as pd
def k_anonymity(df: pd.DataFrame, quasi_identifiers: list[str]) -> int:
"""Return the smallest group size across quasi-identifier combinations."""
group_sizes = df.groupby(quasi_identifiers, observed=True).size()
return int(group_sizes.min())
qi = ["age_band", "postcode_prefix", "gender"]
k = k_anonymity(df, qi)
print(f"k = {k}") # k=1 means at least one person is uniquely identifiable
# Flag the risky groups for generalization or suppression
risky = df.groupby(qi, observed=True).filter(lambda g: len(g) < 5)
print(f"{len(risky)} rows sit in groups smaller than 5")
A result of k = 1 means someone is unique in your data. Generalize the attributes (wider age bands, shorter postcodes) or suppress those rows. K-anonymity alone does not protect against attribute disclosure, so combine it with l-diversity or differential privacy for sensitive attributes.
Step 4: Enforce Storage Limitation
Data must not be kept longer than necessary. Build retention into the pipeline so it runs on a schedule.
import pandas as pd
RETENTION_DAYS = 730 # set from your documented retention policy
def apply_retention(df: pd.DataFrame, ts_col: str = "event_ts") -> pd.DataFrame:
cutoff = pd.Timestamp.now(tz="UTC") - pd.Timedelta(days=RETENTION_DAYS)
kept = df[df[ts_col] >= cutoff]
# Log counts for the accountability record, never the rows themselves
print(f"Retention applied: removed {len(df) - len(kept)} of {len(df)} rows")
return kept
Snapshots, feature store versions, and notebook outputs are the places where old data survives unnoticed. Include them in the retention policy.
Step 5: Handle Data Subject Rights in Your Pipelines
Individuals can request access (Art. 15), rectification (Art. 16), erasure (Art. 17), restriction (Art. 18), portability (Art. 20), and objection (Art. 21). Each one is an engineering problem if your pipeline was not designed for it.
| Right | Pipeline Implication |
|---|---|
| Access | Be able to query every store by the subject’s token |
| Erasure | Delete from raw tables, feature stores, caches, and future training sets |
| Rectification | Correct source data so downstream features refresh |
| Portability | Export data in a structured, machine-readable format |
| Objection | Exclude the subject from profiling and marketing models |
A practical pattern is an erasure registry: a table of pseudonymous tokens that must be excluded. Every training job filters against it.
import pandas as pd
def exclude_erased(df: pd.DataFrame, erased_tokens: set[str]) -> pd.DataFrame:
"""Drop records for subjects who exercised the right to erasure."""
return df[~df["customer_token"].isin(erased_tokens)]
erased = set(pd.read_parquet("erasure_registry.parquet")["customer_token"])
train_df = exclude_erased(train_df, erased)
Models already trained on erased data are a harder question. Regulators have not settled it, and the answer depends on whether the model itself retains personal data. Document your reasoning and your retraining cadence.
Step 6: Automated Decisions and Profiling (Article 22)
Individuals have the right not to be subject to a decision based solely on automated processing that produces legal or similarly significant effects. Credit approvals, hiring filters, and insurance pricing are typical examples.
For models in scope, you need:
- Human review with real authority to change the outcome, not a rubber stamp
- Meaningful information about the logic involved
- A way to contest the decision
This is where model choice has compliance weight. A gradient-boosted model with SHAP explanations is easier to defend than an opaque ensemble with no explanation layer.
| Model Type | Interpretability | Article 22 Fit |
|---|---|---|
| Logistic regression | High (coefficients) | Strong |
| Decision tree (shallow) | High (rules) | Strong |
| Gradient boosting + SHAP | Medium (post-hoc) | Acceptable with documentation |
| Deep neural network | Low | Needs strong explanation tooling and human review |
Step 7: Run a DPIA When Risk Is High
A Data Protection Impact Assessment (Article 35) is mandatory for processing likely to result in high risk. Large-scale profiling, systematic monitoring, and large-scale use of special category data are typical triggers. A DPIA records the purpose, necessity, risks, and mitigations. Start it at design time and update it when the model’s scope changes.
Practical Examples and Real-World Use Cases
Predicting Customer Churn with Python
Scenario: A subscription business wants a churn model.
- Lawful basis: Legitimate interest, documented in a balancing test.
- Minimization: Use tenure, spend, and ticket counts. Exclude free-text ticket bodies unless scrubbed of identifiers.
- Pseudonymization: Replace the email with an HMAC token before the data reaches the notebook environment.
- Rights: Filter the erasure registry before each retraining run.
- Output use: Targeting retention offers is low-risk. Auto-denying service to high-risk customers would trigger Article 22 review.
Handling Sensor Data from Connected Devices
Scenario: A smart-home company streams device telemetry.
Device IDs linked to households are personal data. Aggregate readings to hourly or daily levels at the edge where possible. Drop precise timestamps if the model does not need them, because usage patterns reveal when people are home. Apply retention at the stream sink, not only in the warehouse.
Training on Text Data
Scenario: An NLP team fine-tunes a model on support conversations.
Run entity recognition to detect and mask names, emails, and phone numbers before the text enters the training set. Assess whether the resulting model could regurgitate personal data. The EDPB takes the position that not all AI models trained on personal data can be considered anonymous, and that the threshold for anonymity is high. Document your testing, such as membership inference and extraction attempts.
Data Scientist’s GDPR Checklist
| Stage | Check |
|---|---|
| Before collection | Purpose defined, lawful basis recorded |
| Ingestion | Feature allowlist applied, special categories flagged |
| Preparation | Direct identifiers removed, keyed pseudonymization used, key stored separately |
| Modeling | Legitimate interest assessment or DPIA completed, bias and proxy-variable audit done |
| Deployment | Article 22 safeguards in place, explanation method documented |
| Operations | Retention job scheduled, erasure registry enforced, model retraining cadence set |
| Accountability | Records of processing updated, decisions logged |
FAQ
Is pseudonymized data still personal data under GDPR?
Yes, under current EDPB guidance. Pseudonymized data can be attributed to a person with additional information, so GDPR still applies. The Digital Omnibus proposal would change this for entities that cannot re-identify the data, but check whether it has been adopted before relying on that.
Can I use personal data to train a machine learning model without consent?
Often yes, if you have another valid lawful basis. Legitimate interest is the most common one for model development. You must document a three-step assessment covering purpose, necessity, and balancing against individuals’ rights. Special category data needs an additional Article 9 condition.
Does the right to erasure apply to trained models?
The right applies to the personal data you hold, including training datasets and feature stores. Whether a trained model itself contains personal data depends on the model and is assessed case by case. Delete the source records, exclude them from future training runs, and document your retraining policy.
Do data scientists need to complete a DPIA?
A DPIA is required for processing likely to result in high risk to individuals, such as large-scale profiling or systematic monitoring. The controller owns the obligation, but data scientists supply the technical details on data flows, model logic, and risk mitigations that the assessment needs.



