4 Lessons Learned From 4 Years Of Non-Stop Data Analysis

4 Lessons Learned From 4 Years Of Non-Stop Data Analysis

Analytics is a field that moves fast.

Most changes have happened as a result of the transition from PC computing to mobile. Mobile is still a new space for analytics. Things are happening quickly, and everyone’s looking for the newer, better, and faster solution.

At Amplitude, we’ve now been analyzing data non-stop for four years. We want to share four of the themes that we’ve seen emerge over our time helping companies with analytics. These are bigger-picture ideas and frameworks that sometimes get lost in the rush for the new, but they’re crucial to keep in mind if you want to successfully use your data to its fullest potential.

1. Your Tools Are Different Now

People assume that the tool that they use for regular web analytics should also work on mobile. It doesn’t. Web and mobile aren’t two ways of experiencing your product. They’re completely different mediums with their own structure, organization, incentives, and metrics:

  • Identification: We’re not dealing with cookies in mobile apps. Users are identified by unique IDs that are assigned when they first download and register.
  • Offline usage: Users won’t always be online when they use your app, so you can’t rely on tracking them purely on a live basis.
  • Gestures: A click is simple to track—a click is a click. But users interact with apps in all kinds of ways, whether that’s a tap, a long tap, a swipe, a shake, or one of the many other gestures made possible by mobile.
  • Cohorts: Since a user’s time with your app clearly begins with installation and ends with them becoming inactive, you can perform much more powerful analysis into how certain groups behave and respond to changes.
  • Monetization: Websites have traditionally monetized through advertising and therefore oriented their analytics toward that goal. Mobile apps tend to not, instead trying to directly monetize the end user through additional installs.

Lesson: Emphasize Mobile

These are fundamental differences in how users experience your product, and the tools you use to understand those users have to adjust. Mobile analytics aren’t a nice-to-have: they’re an essential part of building a great mobile product.

Tracking your mobile analytics properly is one of the most powerful tools you have to grow. It’ll give you insights into user behavior that you would never have been able to find with regular analytics tools. What you learn will help you reduce friction, increase retention, and build a stickier, more powerful product. Your company will be stronger as a result.

2. Data Accessibility Matters Much More Now

When the mobile-first app Tinder first started seeing massive usage, their formerly effective analytics platform broke down. Queries that should have taken minutes were taking hours. That’s not acceptable, especially now that we expect almost every decision we make on a product to be backed by some kind of data.

It used to be okay for all data requests to flow through a central node inside your organization. If a product manager needed information on user behavior or conversion funnels, they’d make a request to the data science department and that request would be fulfilled.

Today, smaller screen sizes, more compressed menus, and the greater variety of possible interactions inherent to mobile have combined to generate massive amounts of information about users on a regular basis. And data’s something every company wants to use.


Without a method of making that data accessible to all, the decision-making machinery in an organization grinds to a halt.

Lesson: Think outside the box.

Make sure that your tools serve the data needs of your entire organization. As Fareed Mosavat of Instacart said, “If you say you’re data-driven but everything has to go through an analyst, you’re not actually data-driven.”

As your company starts to scale, this gets more and more crucial. The volume of requests that your data team (should) be getting will slow the decision-making capabilities of your team to a crawl. That’s why it’s essential that you go with an out-of-the-box analytics solution.

Your homebrew tool may have served you well when it was just you and your co-founder, but other people have thought long and hard about the problem of data accessibility. You need to focus on building your business, not hacking together the best possible analytics solution.

3. Please Don’t Mess With The UX

Years ago, startup “growth hackers” enamored with the runaway success of companies like Facebook and Zynga set about “hacking” the growth of their own apps.

But growth hacking is like a game of Telephone. One company does something cool, word gets out about it, and other people start trying to emulate their success. They always do it a little bit wrong, however. The same technique never works twice.

That’s how you get apps that have been so blatantly over-optimized for short-term metrics that they become almost unusable. All those social share buttons, dark patterns, full-screen takeovers and annoying push notifications might get you conversions in the short term, but they will almost always cost you in the long term.

Companies like Facebook didn’t grow because they found some low-level conversion metric to attack. If they’d done early on what some apps do today, they would have basically been MySpace.

The companies that have won and continue to win with data keep the user experience top of mind. Every change that negatively affects their users even in a tiny way needs to be systematically weighed next to the possible benefits of the experiment. In the long-term, it’s making that experience consistently good that will keep users around.

Lesson: Listen to your users.

When most programmers write software, they’re also writing unit tests. These blocks of code mimic the behavior of actual users by sending inputs into the program. By monitoring the output, the developer can see if their application is working as intended.

When you’re running experiments, don’t just make changes and hope for the best—apply the same logic. If the change is dramatic, run it by the people on your team first. If it doesn’t raise any red flags there, expose it to a limited cohort of users.

And don’t do it just to do it: analyze how they respond and then assess whether you want to introduce this change on a larger scale.

4. Look At Behaviors, Not Just Action

The biggest shift in analytics has been the movement from raw and high-level data to a more holistic, behavioral analysis.

Four years ago, you basically had three options for understanding usage in your app:

  • Look at raw data on how individual users navigate and use your app.
  • Look at high-level dashboards that show you conversion funnels and retention charts.
  • Hire a brilliant team of data scientists that can collect, process, and digest not just how groups of people are using your app but why.

The first two were useful but incomplete. The third was too expensive. Today, that kind of “how-and-why” analysis is possible with tools that are more affordable and more user-friendly than anything we had four years ago.

The basic principle is cohorting. Instead of looking at individual users or the entire user base, look at how specific cohorts of users behave. Find the kinds of behavioral patterns you think engender long-term retention in your app and analyze those more closely.


This isn’t about making a button convert 0.04% more visitors by making it green. This is about finding the “Aha!” moment, the inflection point in the user’s experience of the app that makes them want to come back time and time again.

Lesson: keep it simple.

There are a million ways to analyze the data generated by your app. Don’t get tunnel vision and think that the best way to make a measurable impact is to optimize for some random conversion variable you think is important.

What matters in the past, present and future is one thing: the user experience. Your analytics need to serve the user experience, not the other way around. Find the behaviors that set your app apart and double down on them.

Optimize for variables, and you’ll make gains in the short term while destroying your company in the long term. Optimize for the user experience, and it might take longer to see results—but you’ll be moving in the right direction.


The shift to mobile has changed a lot of things about analytics. It’s raised the stakes by multiplying the number of trackable events, changed how we analyze those events, and made it more important than ever to keep the user experience top of mind.

The field’s novelty has led many to come out with their ideas and frameworks for how mobile analytics should be done, and learning new tactics is great. But it’s important to keep the bigger picture in mind while you figure it out:

  • Give your team the right tools
  • Give your team access to the data
  • Don’t hurt the user experience
  • Look at how people actually behave in your app

Organize your experiments and analysis around those four core ideas, and you’ll set yourself up for success in the long run rather than just optimizing for what’s cool today.

Related Posts

Current article:

4 Lessons Learned From 4 Years Of Non-Stop Data Analysis