We don’t just have an AI data protection problem - we have an AI intimacy problem
Plus: Your chatbot is building a model of you whether you configure it or not
Hi everyone, thanks for coming back to Customer Futures.
Each week I unpack the disruptive shifts around Empowerment Tech. AI Agents, digital wallets, Personal AI in and the future of the digital customer relationship.
If you haven’t yet signed up, why not subscribe:
Hi folks,
We’re jumping straight into the weeds this week.
It’s never been more important to understand the future of the digital customer. Welcome back to the Customer Futures newsletter.
In this week’s edition:
Is the car the next browser for a privacy backlash?
We don’t just have an AI data protection problem - we have an AI intimacy problem
Your chatbot is building a model of you whether you configure it or not
… and much more
Let’s Go.
Is the car the next browser for a privacy backlash?
Modern digital cars are already gathering a staggering amount of personal data.
Where you go. How fast (and how slow). Breaking patterns. How many people are in the car. In some cases, who is in the car. In some cases, what is said inside the car. Then there’s the biometrics (weight, driver blink rate, temperature and all sorts of funky stuff).
Samsung famously announced about the internet of things: if it can be connected it will be connected. So the digital - and now AI - age, we must also remember: if the data can be collected, it will be collected.
Ah yes.
Qn for you: When was the last time you read the privacy notice for your car? Me neither.
Fun fact: check out the T&Cs from Ford Sync, which records “Voice utterances” from anyone in the car when it’s in a certain state. .
“Ford’s Service provider 24/7 Customer, Inc. (”24/7”), may record and retain user voice utterances (”recorded utterances”), which are recordings of sounds made when SYNC Services is in listen state and waiting for a user command or response.
“These recorded utterances may include all sounds in the vehicle, including the voice of the user and voices of other vehicle occupants, while the service is in listen state. 24/7 may also, at Ford’s request, randomly record and assemble in sequence, all voice communications made from the time the Service is connected (by the user pressing the VOICE button) to the time the Service is disconnected. (”Whole call recordings (WCRs)”).
“WCRs will include voice utterances and may include any other sounds in the vehicle, including the voices of the user and other vehicle occupants, during the entire time the Service is connected. Both recorded utterances and WCRs may be associated with you or the cell phone number assigned to the Service.”
Hmmm
So what’s the equivalent of a private browser, but for a car? Incognito mode? Maybe even flight mode?
Remember the Good Old Days where not everything was a subscription, or being tracked every moment of every day.
The thing is, I probably want and need much of that car data. There are some important patterns in there to help with safe driving, managing fuel, personalisation etc.
But that data is for me.
Not for the insurance company. Or to help the manufacturer ‘improve the service’. And not when it’s just another API to feed the Hungry Data Borg. To help with risk pricing and accidental tracking.
But so far, so surveillance economy. Much as the EFF’s own research into car data policies uncovered a couple of years ago.
But now, ah yes, we have AI. And AI agents are about to do all sorts of interesting things with that data too. Automated service bookings, insurance policies, driver warning flags, hyper personalisation.
But in whose interests will those car-fuelled AI platforms be acting? The car is possibly the next browser in terms of the privacy backlash.
It’s just that folks aren’t paying attention yet.
READ, EFF PRIVACY REPORT ON CARS
We don’t just have an AI data protection problem - we have an AI intimacy problem
Another Ben Evans knockout insight recently:
“Playing with new Siri, it occurs to me that regulators are focussed on allowing you to install multiple assistants,
“But the more important competitive factor might be whether the individual apps allow access to competing assistants for personal context.
“Neither Siri nor chatGPT can see inside WhatsApp, and that’s not Apple’s call. And of course, even thinking about opening that up prompts about a dozen complex questions”
As I’ve been saying for a while. It’s not enough to be able to delegate to your Digital Assistant powered by AI. And it’s not enough to simply trust that personal agent with comfortable answers about the commercial model, the UX transparency, the data governance, the compliance and so on.
Your personal agent will also need portable customer context to be useful.
Rich and useful personal data from places like your bank, your travel app and your calendar. Your email, your OpenAI history, your browser information. Your location, your preferences and all the rest.
Otherwise all these digital assistants will become quite dumb. Or at least transactional. Stuck in a single lane, with a narrow view of you and your needs.
So while the regulators get excited about ensuring agents are compliant, we also need to be paying attention to the memory layer.
Who has access to what?
Why do they need it?
What happens once the AI agent has completed its task?
From a data protection point of view, this needs to be as much about access as it does privacy.
When you game it all out, I keep coming back to the same point.With AI, we don’t just have a data privacy problem. We have an intimacy problem. Where your AI - and specifically your AI agent - will know you better than you know yourself.
And that’s harder to see, harder to regulate, and harder to catch and adjust as a consumer.
Ben Evans is right about regulators missing the bigger picture about customer context.
How long until my private, digital context can be portable too?
Your chatbot is building a model of you whether you configure it or not
Paul Worrall is what Marc Andreessen calls an ‘AI vampire’.
Working ridiculously long hours with AI because, as Paul puts it:
“The tools are so compelling you don’t want to stop. Sixteen-hour days. All-nighters. Not because a deadline required it. Because stopping felt like waste.
“I am deep in flow with my agents”
Deep flow indeed. He writes brilliantly about his new AI-driven life, with a small army of always-on AI agents running his business.
And it’s just him as the human in the loop.
He has an AI CFO, an AI CTO, an AI COO and many more AI team members. He wakes up in the morning to find out that they’ve had a bunch of meetings, a fight about some topics, need his final decision, then off they go.
One of his recent posts gave us a glimpse of some of the fascinating impacts of an always-on AI operation:
“I was configuring Claude’s personal preferences — the prompt that shapes how it responds to you across every conversation. Instead of writing it myself, I asked the agent I’ve been working with for months.
“I chose to ask my agent to write it. Because who better to know me?”
The answer surprised him:
“Mild dyslexia. Don’t correct my typos or ask for clarification on them — interpret my intent and continue. I can read the output — don’t recap what you just did at the end of a response. Accuracy over positivity. Once I’ve decided something, don’t re-ask. Execute.”
Interesting because a it came back with a bunch of notes and preferences he never explicitly stated.
“The agent inferred them from how I reacted over weeks of working together. It noticed I get frustrated by summaries. It learned I make fast decisions and don’t revisit them.
“It figured out the dyslexia from patterns in my typing. Not just fat-fingers!“
Just think for a second about the implications.
Social media is sometimes called ‘Baby AI’. Only really used to predict the next post video or photo to put in front of you. Designed to keep you there and paying attention for clicks. But Baby AI only has a fractional view of the things you’re interested in. Who you know. And what you like.
Now deploy that more advanced model of you - a much more intimate model of you, seeing patterns you can’t see - like Paul’s writing habits - and drop it against the global commercial complex.
Against real time bidding for ads. Against government surveillance in the ‘regimes’. Against BigTech infrastructure that wants to ‘own’ the customer, and whose business models depend on your attention and intent… and you get something entirely different.
This is much less about data protection. The AI data security and protection toothpaste is now firmly out the tube. No, it’s much more about data intimacy.
As Paul points out:
“Your AI is building a model of you whether you configure it or not. The question is whether you take control of that model, or leave it to someone else to infer.”
So who are we really empowering. You, or who?
A longer list of things to track
There are SO many interesting and important Customer Futures things happening at the moment. Here are just a few from my open tabs:
Post: Within 20 seconds of taking a covert photograph with these glasses, a deepfake video can be generated locally on the phone in their pocket READ
Article: We keep doing UK digital identity without doing digital identity, and now we’re even stopping that READ
Post: Your sports team’s reach on social isn’t real anymore - you’re renting fake attention READ
News: Brazil takes legal action against Tools for Humanity (World ID) and AWS, because 400k residents were paid to share iris biometrics without informed consent READ
News: myID (was Australia’s MyGovID) is going to charge private sector to use it READ
And that’s a wrap. Stay tuned for more Customer Futures soon, both here and over at LinkedIn.
And if you’re not yet signed up, why not subscribe: