About a month ago, I received a letter saying that I could save a lot of money on my 15-year mortgage. It gave my current rate, the rate I could get if I refinanced, and the amount of the new payment.
This letter came from a nationally known bank in the U.S., which many of my readers would recognize. (I’ve written about this bank before, back in 2018, in Bank Closes Account, Doesn’t Know Why (the account that was closed was mine; I have not disclosed the bank’s name for personal, professional, and purposeful reasons).
Okay, I’ll bite
Anyway, I was curious why I got the letter because I have a pretty good interest rate already on a 15-year fixed mortgage. So I called the banker listed in the letter.
I reached her voicemail, and we exchanged voicemails for a couple days.
Finally, I reached her. I asked her whether I could really save any money. Since my mortgage is with this bank, she reviewed my loan, studied it in silence for a long minute, and then said:
“Sir, I’m not sure why you got this invite. We can’t save you any money. I’m sorry. You’re current loan is pretty good.”
I then replied, “That’s what I thought, but I figured I’d call and find out for sure. At least you got it right the first time,” I joked.
She apologized again (too many people never apologize for mistakes or wasting someone’s time, and I received 2 apologies already), we exchanged a few more words, and we hung up.
Since all the AI classes I have taken recently always explain how supervised machine learning can be used to better target people who are most likely to respond to a marketing campaign, I wondered what happened.
Assuming this bank used machine learning to select me, I thought about why:
- I have been a customer off and on. In the recent past, I’ve had checking, a car loan, a safety deposit box, and this is the second mortgage loan I’ve had with this bank over the years. I always paid off the loans and left the bank in good standing.
- I have excellent credit.
- I’ve done contract work with this bank in the past, and enjoyed my time there.
All of those factors were in my favor, so no surprises there.
- Although I’ve had 2 mortgages with this bank, I’ve never refinanced with the bank. My loan has always been sold to the bank a year or two after I took out the mortgage.
- My current loan was a 15-year loan; the letter was pitching a 30-year loan (okay, the bank makes more on a 30-year loan, but why would the machine learning algorithm select me for a 30-year loan? I assume the bank knows my income and net worth, and that I’m not having financial issues–so why try to suggest I go back to a 30-year?)
- In the end, the banker confirmed that the ban couldn’t save me any money EVEN going with a 30-year fixed, especially when including the closing costs.
So I concluded that the machine learning model* failed, at least in my case. So either the model was wrong in the first place, or due to other factors, the model was no longer correct, at least for group of customers the model assigned me to.
*The best definition of model I’ve found: “any physical, mathematical, or logical representation of an object, property, or process” (Anand Rao). In this case, it’s the process for selecting who is most likely to refinance (and pay off the loan).
Rub salt into my wound
I thought that was the end of it.
Until I received another letter a few weeks later from the same banker with an updated proposal (still 30-year fixed, but the interest rate had changed slightly).
So that means that not only the bank wasted marketing dollars selecting me and sending me a letter in the first place, but in sending me a second letter when I did not refinance after the first letter. And the bank wasted the banker’s time, too.
I can only assume that after my earlier call with the banker, nothing was reported back to marketing that I never should have been selected. That irritated me a bit as I read the second letter, but I threw it away.
I wish I would have kept the 2nd letter, as I could have called the banker back and asked if marketing ever asks to hear about situations like this. If I get a third letter, that’s what I plan to do…
Anyone involved in machine learning should always follow up with those impacted by your model so you can improve your accuracy and reduce expenses the next time around.
Because the banker and the machine learning team/data scientists didn’t talk about my call, no one knew that a customer responded (that’s the marketing goal!) and the banker could not convert the call into a sale because of bad data/bad selection process (result: bad customer service!) Furthermore, the customer got ANOTHER letter with a similar useless pitch and was irritated (more bad customer service #2!).
What if that happened 100 times on this campaign? 1000 times?
If I was told that machine learning was NOT used in this marketing campaign, then not only do they have a poor marketing process, I’d wonder WHY they are NOT using machine learning to improve their process.