Are We Mystery Shopping the Wrong Things?

I’ve built a pretty significant part of my career and personal brand around mystery shopping apartment communities.

So I recognize the irony in what I’m about to say.

I think we need to challenge mystery shopping.

Not eliminate it. Not declare that it doesn’t work. I still believe there is tremendous value in seeing the experience through the eyes of a prospect.

But lately, I’ve found myself asking a different question:

Are we measuring the things that actually matter?

I recently recorded an episode of Rooms With Ronald entirely about this. I went into the episode thinking about mystery shops. I finished recording thinking about something much bigger.

When the Prospect Changes, Shouldn't the Shop Change Too?

Think about how someone searched for an apartment 10 or 15 years ago compared with today.

Today's prospect may visit an ILS, research your website, interact with AI, text with your community, schedule a tour online, talk to someone on the phone and finally walk through your doors.

Another prospect may drive by, see your sign and walk in without ever interacting with you before.

Those are two very different journeys. Yet we can evaluate them as if they were the same.

I've heard one of the biggest criticisms of mystery shops countless times: "Our teams know when they're being shopped."

My response is becoming: Then maybe it's time to change the shop.

If the scenario is predictable, the questions don't sound like actual prospect questions and the interaction doesn't resemble how people shop for apartments today, what exactly are we testing?

The ability to recognize a mystery shopper? Or the ability to create a great leasing experience?

There still needs to be consistency. There still needs to be a standard. But standardization shouldn't require us to pretend every prospect enters the journey at the same place.

A 95% Doesn't Automatically Mean Someone Can Sell

This might be the part that makes some people uncomfortable.

Can someone score 95% on a mystery shop and still be a mediocre salesperson? I think they can.

We love boxes because boxes are easy to measure.

Asked about move-in date? Check.

Asked about pets? Check.

Used the prospect's name? Check.

Showed the amenities? Check.

Asked for the lease? Check.

But did they listen?

Did they discover why the person was moving?

Did they understand what mattered?

Did they take what they learned and use it to make the apartment relevant?

Did they influence the decision?

Those questions are harder to reduce to a checkbox, but aren't those closer to what we're actually trying to accomplish?

Imagine a prospect tells the leasing professional they don't work out and aren't interested in seeing the fitness center. The leasing professional listens and skips it.

If our shop requires the fitness center to be demonstrated, they just lost points for listening to the customer.

That's where I start to question what we're measuring.

The opposite can happen too. Someone can hit every required talking point, show every required amenity and ask every required question while creating an experience that feels completely robotic.

They passed the shop. But did they sell?

Not Everything We Can Measure Needs to Be Measured

I think we've become very comfortable with the math.

95% is good. 72% is bad. Pass. Fail.

But who decided every behavior deserves the same influence on the outcome?

Forgetting to mention an amenity and never discovering why someone is moving aren't necessarily equal misses.

If the prospect already told us their move-in date, desired floor plan and pet information before arriving, should the leasing professional have to ask those questions again to earn points? Or should they acknowledge what they already know and move the conversation forward?

This doesn't mean throwing out objective standards.

The opposite extreme creates its own problem. "I thought they were great" isn't a performance measurement system either.

We need something defensible and consistent. But consistent doesn't have to mean outdated.

Maybe instead of starting with 40 things we can measure, we should start with a much simpler question:

What behaviors actually contribute to a successful leasing experience?

Then build the evaluation around those.

And Maybe the Score Isn't the Most Important Part

This is where my thinking started changing while I was recording.

Why does a mystery shop automatically need a score? I know. That sounds strange coming from someone who believes in mystery shopping.

But stay with me. Some of the most valuable information I've received from shops hasn't been the percentage. It's been the recording. The comments. The observations. What the shopper saw, heard and felt.

A score can tell me someone earned an 82%.

The experience can tell me why.

That's what I need if the goal is development.

And that's another place where I think we need to challenge ourselves.

What happens after the score?

If an employee fails, receives the report, signs the paperwork, completes a reshop and passes, did we actually change the behavior?

Maybe. Or maybe we taught them how to pass the shop the second time.

If we're serious about development, the shop should be the beginning of the conversation, not the end of it.

Managers need to coach. Training needs to align with the behaviors being evaluated. Employees need to be accountable for improvement. And if someone repeatedly struggles, eventually we need to ask questions about the coaching around them too.

If it's important enough to measure, it should be important enough to teach, practice and coach.

Then I Turned Off the Microphone

This is where the idea got bigger for me.

I finished recording the episode thinking about all the things we measure during the leasing experience.

Then I started thinking about everything we don't.

We put an incredible amount of energy into pressure-testing what happens while we're trying to get someone to lease an apartment.

We shop the phone call. We shop the tour. We measure the greeting. We measure discovery. We measure the close. We measure follow-up. Then the prospect says yes. They apply.

And suddenly, I'm not sure we're nearly as curious.

What happens between application and move-in? What does the first 72 hours as a resident feel like? Does the experience we deliver after someone signs the lease match the experience we worked so hard to sell before they signed it?

That made me realize something. Maybe the bigger opportunity isn't simply redesigning the mystery shop.

Maybe we've been pressure-testing one part of the renter journey and assuming the rest works.

A resident doesn't experience our organizational chart. They don't experience "leasing" and then "operations" and then "maintenance" as separate businesses. They experience one community. One company. One journey.

Everybody has a piece to play in that experience.

So yes, I think it's time to pull out our mystery shops and ask whether every question we're measuring actually helps someone become better at selling apartments.

But once we're done with that, I think there's another question waiting for us:

Why does our pressure-testing stop there?

That's where I'm going next.