The price of being understood

We use the word "understood" very loosely now. An app understands your taste in music. A platform understands what you'll buy next. A feed understands what will keep you scrolling for another four minutes. None of that is understanding. It's pattern-matching with excellent aim, and we've let it borrow a word that used to mean something closer, and something mutual.

Real understanding runs both ways. If I understand you, you generally know it, and you had some part in it, you told me things, I asked questions, we corrected each other along the way. What a recommendation engine does to you is nothing like that. It infers you from a distance, refines the inference every time you react, and never once asks whether the inference is what you'd actually call true.

The price of that kind of being "understood" is that you start optimising for the version of you the system already believes in. You watch what it expects you to watch. You buy what it expects you to buy. Not because you were persuaded in any honest sense, but because the path of least resistance is the one the system built for the you it already decided you were. Over enough time, the inferred version starts to crowd out the real one.

I don't think the fix is to reject being seen, that's not realistic and it's not even desirable, a certain kind of visibility is how good products and good public services get built in the first place. The fix is making sure the person being modelled has access to the model. If a system has decided who you are for commercial purposes, you should be able to see that decision, question it and correct it. Anything less isn't understanding. It's a guess, made at scale, that you never got to check.

The companies I respect most are the ones willing to make that guess visible. Everyone else is just betting that you'll never ask to see it.