Why Benchmarking Is A Bad Idea

July 5, 2017 Carl Ryden


Recently, we’ve received a few inquiries from banks asking us some variation of this question: “Can you give me the average spread on this type of deal, in my area?”

It’s an understandable request, given the amount of data we collect – PL users priced over $100B in loans this past quarter alone. But our answer is always, “No. We don’t provide benchmarking data.”

We’ve explained this position in several podcasts with PrecisionLender CEO Carl Ryden. If you’re interested, you can listen those discussions here (beginning at 7:11) and here (beginning at 2:05).

Or you can read on for excerpts of Carl’s thoughts on these “get to give” services, in which you provide your data and in return you get the averages of all the data they’ve accumulated from the participating banks.

Messy Data

Here’s one of the problems you run into. I was talking to a banker, a user of that (benchmarking) service. And I asked him: “How do you provide say, a line of credit? Because how you price a $100,000 line of credit (LOC) that’s part of a deal with a $15mm commercial real estate deal alongside it is different than how you price a $100,000 LOC that’s stand alone.”

He said, “Absolutely.”

“Okay, is that reflected in the data you’re receiving?”

He answered, “No.”

Then there’s data cleanliness. Their data is often messy. One of the things I hear on that is: ‘We know our data is a mess. We’re hoping everyone else’s is better.’ But I’ve heard this from multiple banks that use these services. It’s one of those things where everybody thinks the data they’re feeding in is flawed, but they’re hoping everyone else’s is better. I don’t know that that works out.

The Average is the Ceiling

But even assuming the data is okay and you can find out the average spread for a particular type of deal, what does that do? Relationship managers who receive that average very seldom start at that number and then move up. The average becomes the place they start because they know they can justify that number. Then, they go down from there. So your pricing of your deals actually moves below the average. Everybody else’s average becomes your ceiling.

Then, you feed all that data back in, dirty as it may be, and it comes back out, and now the ceiling moves lower. It becomes a dynamic system that feeds back on itself in a bad way.

The Door Height Problem

There’s a simpler way of thinking about this, that I call “The Door Height Problem.”

Suppose you measure the average height of every person who walks into your bank. You find that it’s, say, 5’11”. So you decide that – for whatever reason – you can save some money if you move the door height to 6”.

A month later, you again measure the average height of people entering the bank. It’s now down to 5’6″, because you fed that back into the acquisition process that affects the average going forward. You’ve introduced a feedback loop into the system.

You’ve tainted the data. The worst part is when folks think that calculating that average was really cool, and they blindly do it again. You move the door height down to 5’7” … and it keeps moving down. You see that sort of thing happen again and again. It’s a different flavor of the Bridgeton Death Spiral.

Instead of feeding the averages, you need to understand what’s important to the bank. Understand that fully, then build systems that allow you to understand what’s important to each customer. For each customer, handcraft the solution that works best for them, that allows you to elevate each conversation, and earn a better return.

We see that as a huge shift in how you think about things. We’ve seen so many folks go down after feeding in the averages.

One of the things I’ll often ask them is: “How’s that working for you? Has it gone up since you started doing that or has it gone down?” They’ll say, “Oh, the market’s really competitive. It’s gone down.” You create that feeling that it’s the markets, because you continue to tell the RMs that the averages are going down.

A Better Use of Pricing Data

What we want to do is instead take that data and use it as a means of extracting intelligence - about what folks are doing, what they’re not doing, what they do in this particular situation. Then feed that back in. Then have a machine or a system learn the more lenders use it - about how they’re using it, what the best lenders are doing differently from the not so great lenders, and what the ones who are winning are doing differently. It becomes a means of continuous improvement.

I think you’ll see more of this crop up in every aspect of our lives. You have AI assistants scheduling meetings with x.ai and Clara Labs. You also have Siri and Cortana, and even things we use every day like Spotify selecting music that it thinks we’re going to like.

It’s about using intelligence as a means of enhancing that customer experience with a bank, that borrower experience, is really a key thing.



About the Author

Carl Ryden

Carl Ryden has deployed pricing management solutions in hundreds of financial institutions, ranging from banks that span the globe to those that serve local communities. Beginning in 2009 with a blinking cursor and a blank screen, Carl developed PrecisionLender’s loan pricing system. Carl’s breadth of experience and passion for technology, finance, strategy, and software development give him a unique perspective on risk-based pricing. He has an MBA from MIT Sloan School of Management, a Master’s Degree in Electrical and Computer Engineering from MIT and a BS in Electrical Engineering from NC State University. Carl has written a number of articles specifically relevant to bankers, on topics such as relationship pricing, purchasing cloud-based solutions, and the impact artificial intelligence will have on the industry.

Follow on Linkedin More Content by Carl Ryden
Previous Article
BankOnPurpose Speaker Videos: No One Wants A Loan
BankOnPurpose Speaker Videos: No One Wants A Loan

Learn more from Bob Moesta and Chris Spiek, pioneers of the Jobs-to-be-Done Framework. And then watch and l...

Next Article
5 Articles to Make You a Smarter Banker
5 Articles to Make You a Smarter Banker

We've compiled a list of the top 5 articles you need to read to help you become a smarter banker today.