Dean Abbott is President of Abbott Analytics and currently is the Bodily Bicentennial Professor in Analytics at UVA Darden School of Business. He is an internationally recognized thought leader and innovator in data science and predictive analytics with more than three decades of experience solving a wide range of private and public sector problems. Mr. Abbott is the author of Applied Predictive Analytics (Wiley, 2014) and coauthor of The IBM SPSS Modeler Cookbook (Packt Publishing, 2013).
Recently, in an exclusive interview with Higher Education Digest, Dean shared his insights on the current trends in analytics, what makes Abbott Analytics unique, his career trajectory, success mantra, pearls of wisdom, and much more. The following excerpts are taken from the interview.
What are some current trends in analytics? What are you most excited about in analytics?
The biggest new trend in analytics is what is sometimes called “Generative AI”, or more technically speaking, a class of learning based on “Large Language Models” (LLMs). The most popular version of this style of analytics is ChatGPT-4 and it has captured the attention of many within analytics and those who don’t know much about analytics at all! I think the reason it has become so intriguing is because the answers these models give are in plain English, giving the illusion that they are intelligent or even sentient (hint: they are neither!). As fascinating as these models have been, to me they are interesting but not the most important thing in analytics.
I’m most excited that analytics, and in particular advanced analytics, machine learning, and data science are becoming more popular, and thousands of business professionals are leveraging these approaches to solve problems in the field. When I started my career more than 35 years ago, very few of us existed! One evidence of this trend is the number of undergraduate and graduate degrees now available in data science and machine learning (or closely related fields). This is good, though has some downsides. I’ve found that many who love the idea of being a data scientist but don’t really have the right mentality for it. But these will work themselves out over time.
Dean, can you tell us about your professional background and areas of interest? How did you get started in analytics?
I started in analytics by accident in some ways and by design in others. The intentional part was my deep interest in computational mathematics. I loved analytics even from my early years, when I was 7, 8, and 9 years old. I loved playing baseball as a child, and part of what I loved was understanding how well I was doing. I was fortunate that my father had an interest in numbers as a data analytics manager for Liberty Mutual Insurance Company. He taught me how to compute my batting average and even my earned run average. I used to come home from games and compute my own statistics for the game and cumulative statistics for my season after every game!
But in college, I had no idea what “analytics” was or what kind of field it might be. I majored in computational mathematics because of my interest in numbers, but it wasn’t until my first job in central Virginia at a company called Barron Associates that I was introduced to analytics. I got the job there because of my interest in graduate school in optimal control; I was an applied mathematics major for my master’s degree and loved numerical methods related to control theory. Barron Associates’ founder and principal, Roger Barron, was experimenting with using optimal control to guide missiles and I was able to help them in this quest. The problem was that the solutions were far too computationally complex to solve in real time. Therefore, we used statistical learning, the precursor to modern machine learning, to predict the optimal guidance commands via these complex machine learning models. The techniques worked very well! This beginning is described in a biographical book chapter I was asked to write for the book, “Journeys to Data Mining: Experiences from 15 Renowned Researchers”.
This was the start, but certainly not the end! I moved on to other companies and finally independent consulting by 1999, focusing on analytics in every position, even though the application changed: missile guidance, sonar, radar, financial portfolio optimization, fraud detection, tax compliance, retail, and much much more.
What sets Abbott Analytics apart from other market competitors?
I really don’t think of myself as competing and have never really had to “market” myself. But if I were to articulate what I think I bring to companies who could use my services, I’d say three things. First, execution. I deliver analyses and models on time. I keep deadlines in mind and make sure I provide a solution that meets the expectations set at the beginning of the project. Sometimes, there is much more one can do, but it’s better to deliver “better” now than to promise “much better” weeks to months down the road. Second, I’ve done this for a long time, so can help organizations think about what’s possible and what’s not possible from experience. Third, I can communicate what the analyses and models are doing in the language of the business, helping them understand what the analysis means and why it’s significant; I won’t hide behind technical “geek speak”!
You are also Bodily Bicentennial Professor of Analytics at University of Virginia Darden School of Business. Tell us about your roles and responsibilities.
This is a new endowed position at the University of Virginia Darden School of Business, named after Dr. Sam Bodily, a distinguished and very popular professor at Darden. The position qualifications are listed on their website (https://www.darden.virginia.edu/bodily-bicentennial-professorship-analytics) and candidates are selected based on their leadership in analytics. The purpose is to give the leader in data science time to write, teach, and speak on topics he or she might not have time to work out due to their daily responsibilities. It is a wonderful idea to expand the understanding of data science and analytics to students and business professionals. I will be doing research and writing related to things I have learned in the data science and machine learning world that I don’t see documented or written about in textbooks.
Can you give us some exciting or unusual examples of the application of predictive analytics?
There are so many examples from the past 35 years, some of which I cannot state in a public setting, unfortunately. I’ve built models for the IRS to find non-compliance in business tax returns for S-Corps, C-Corps and Limited Liability Corporations (LLCs, form 1065, or F1065). For the F1065 corporations, we took an approach that I still consider one of the most innovative approaches I’ve been involved with. F1065 corporations don’t pay tax: they distribute profits and losses to other entities, which could be other F1065 corporations, other corporations in general, or even individuals. In the end, the profits and losses have to end up with a taxpaying corporation or individual, though sometimes we’d find profits in particular passing from F1065 to F1065, never landing with a taxpaying person or business. This is non-compliant tax evasion. But how does one find this? We built a network (like a social network) of money flow from organization to organization. By doing this, we were able to identify corporations that effectively controlled a group of corporations or people: a super F1065 in essence. These super F1065s sometimes had dollars that never left the F1065s. These were passed along to auditors to investigate. The innovation was using analytics to identify these non-compliant patterns automatically.
A second interesting use of analytics was with my company SmarterHQ. We built models that predicted the likelihood that a shopper would purchase something on the website within 7 days. This could be very useful to identify which shoppers were behaving in ways that appeared to be purely researching products on the website compared to shoppers who had an intent to purchase. The companies that used our solutions could then build campaigns that target the likely to purchase shoppers to help them complete the purchase.
One interesting thing we did with this model is less intuitive. We would sometimes use the models in conjunction with audiences of shoppers who were heavily engaged with a category of product, let’s say women’s sandals. The standard approach a retailer might use is that shoppers who were highly engaged with women’s sandals but didn’t compete a purchase should get an offer (a discount) to get them to complete the deal. However, we would also layer in our purchase propensity model. If a shopper was highly engaged with women’s sandals AND highly likely to purchase within a week, don’t give them the offer (yet); save margin. Only send the discount to shoppers who aren’t highly likely to purchase.
This is an example of using the meaning of the analytics to help the business rather than following a simple “do what the model says” protocol.
With the arrival of analytics as a service, do you see the demand for analytics professionals go up or down? Why?
I don’t see consultants going away. As much as we want to automate analytics, there is too much that is custom for every solution that cannot be fully automated to still provide room for analytics professionals to have their role too. Why do I write this? Even within the same vertical market, every organization has its own data sources, customized data collection, and custom analytics needs. It is impossible to always know how to identify and then correct problems when data is wrong (miscollected, mislabelled, or missing). Moreover, each organization has its own goals for the analytics objectives that is very difficult to impossible to know (or guess!) without speaking to business stakeholders. In summary, automation that is the cornerstone of analytics as a service is limited to organizations and problems the organization needs to solve that have already been solved to a large degree and signed off on by domain experts.
What challenges do you see for analytics practitioners in the future?
I’ve been doing essentially the same thing for decades. In some ways, I see this style of analysis continuing into the future indefinitely. I think even if this trend continues, as organizations become more data-driven and collect more data to achieve their business objectives, two things will happen. First, analytics professional will have even higher levels of visibility and therefore pressure to help businesses achieve their objectives. We used to be a niche technology no one knew much about! Second, to process the ever-increasing volumes of data, analytics professionals will need to understand how to scale their analyses commensurate with the business needs. Cloud computing will continue to be a significant player as will big data databases and data marts.
How do you define success? What is your take on the ways to achieve long-term success?
Success for me is defined in several ways. First, it is meeting the objectives of my clients with surprise and delight, meaning they find what I’m doing for them useful!
Second, success means providing insights to the client that goes beyond what they already knew about their data and processes. To me, the ideal analytics solutions do the following: (1) echo back the obvious, so that the client says, “I don’t need analytics to tell me that!”. But if it doesn’t say the obvious, no one will believe the analytics; (2) provides new insights the client didn’t know about or didn’t know were as significant as you are finding they are. This is what we really want from analytics!
Third, success means convincing the skeptic that the analytics is doing something practical and useful.
Ultimately, as they say, “success breeds success”. I know I am being successful if I’m still landing clients and getting referrals from clients and colleagues.
Over the years, you have worn numerous hats and excelled in roles that of an author, keynote speaker, workshop instructor, founder to name a few. What is the secret mantra behind your success?
I have to say that I was surprised that anyone wanted to hear what I had to say! I was a heads-down analyst who spoke on occasion, but then what I discovered was happening was that I would say something in my talks and those in attendance would sometimes come up to me afterwards and say, “I’ve never heard that before!”. So, to answer the question, I think the success I’ve had is largely due to me being an active consultant, working on solving problems for my entire career. But there is a second aspect as well. One of the best pieces of advice I ever received was from a colleague at Barron Associates whom I asked the question, “how do you stay current in the field?”. His answer was “read…read…read….” I therefore read articles, conference papers, and technical notes regularly. Sometimes it is to learn something new, but often it is to refine or expand on concepts I think I know, but when one hears how someone else describes the concept, it can expand how one thinks about that topic.
What are some books that are on your reading list right now?
I’m mostly an audio book person now, though I still like reading paper books as well. Some books are analytics books, but many are not and some books I’ve read that have helped me in the field help me understand how to think rather than what to think. By this I mean they expand the way I think about information rather than tell me how to analyze data.
An analytics book I’m in the middle of include, “Rebooting AI” by Gary Marcus. Up soon is “Algorithms to Live by” by Brian Christian and Tom Griffiths. A non-analytics book I’m listening to now is “A Distant Mirror” by Barbara Tuchman and “The Unexpected Spy” by Tracy Welder. What I’d like it be is this: An analytics book I’m in the middle of “Rebooting AI” by Gary Marcus. Up soon is “Algorithms to Live by” by Brian Christian and Tom Griffiths. A non-analytics book I’m listening to now is “A Distant Mirror” by Barbara Tuchman and “The Unexpected Spy” by Tracy Welder. I listen to non-technical books because they provide a broader sense of life than I get from technical books. They also help with how I can create features for predictive modeling. Features are summaries or different perspectives about data; these books help me think differently about data assumptions. Finally, as a Christian I also read the Bible regularly.
What one advice will you give to aspiring analytics professionals?
Many new and aspiring data scientists ask me this question. I assume they have learned the basics of data science and machine learning when they ask me the question. If they haven’t, I tell them to take courses in data science or complete a certificate program in data science. After this, what I usually tell them is to get a job and do it well! Make your boss look good. Learn how the business can most benefit from analytics and machine learning. Become an expert in this application of analytics. This can take several years before one is comfortable being considered an analytics expert.
I say this because some like to think of data science and machine learning as pure sciences, not wanting to wrestle with the difficulties in data quality, data cleanliness, and how models are used in production by the business. Success comes from the business using these analyses or models in a way that improves the business, not by one building models that are “cool” or really “accurate”. If they aren’t used, they don’t matter.