The Dream11 app hosts over 150 million sports fans, with 308 million RPM (about five million requests per second), and 10.56 million user concurrency. At this scale, it’s all about crunching data to get insights and offer a personalised experience. Dream11 collects over billions of events and Terabytes of data per day

In today’s world when one has access to multiple platforms to solve data-related business requirements, is it possible for a company to solely rely on its in-house analytics tool?

Many would quip ‘It’s unimaginable!’. With each tool fighting to corroborate its unique functionality and stay relevant, companies usually lean towards outsourcing data tools.  However, there is a flip side.

At Dream11, our cultural tenets defined as ‘DO-PUT’ guide us and are the foundation of the company. Read Data-first, Ownership, Perseverance, User-First and Transparency.

In this article, you will see all of these tenets come alive. Be it putting in extra hours to ensure our data is protected, taking ownership to build our very own in-house analytics tool, or dealing with challenges that have come our way during this process. At Dream11, we go the extra mile to ensure the data security and efficiency of our systems. Read more below.

So let’s start with the question that got the ball rolling:  Why do we need to build our very own Inhouse Analytics tool?  

Every start-up reaches a certain inflection point when your data multiples by the second and it is extremely difficult to manage such magnitudes of data inflow. Dream11 went through this phase too; hence, we returned to the drawing board and reflected on the drawbacks of operating company data across multiple tools. That’s how the idea to build our very own ‘Inhouse Analytics’ system was born.

The genesis of ‘Inhouse Analytics’ was based on three factors.

1. More control over our data
2. More flexibility to meet our internal requirements at a faster pace
3. Use the data to get intelligent insights at our discretion, without relying on outsourced tools

The implementation of ‘Inhouse Analytics’ comes with its own set of advantages and challenges.

Initially, we used third party tools to capture interaction data and Redshift to store transaction-led data. Inhouse Analytics enables gathering user interaction events, as well as transaction data on a single platform. This helps us map user actions to a specific transaction, revise the definition of metrics, build user interaction-based audience profiles to send out a promotion, and the list goes on!

Dream11 Data Platform Architecture

Now that we’ve covered the why and the what, let’s get to the next big question: How does ‘Inhouse Analytics’  help us achieve more?

#1 A Centralised Tool For Reporting

With a central data repository, Dream11 is slowly phasing out multiple tools that report different metrics. For instance, in the past, Dream11 used third party tools to analyze interaction data, store user interaction data, check user concurrency, and Looker to build models and report transactional data. Retiring all these tools, we now store all data on a single centralized Inhouse data platform.

With this single platform, Inhouse Analytics consumes data from Amazon Redshift/Athena/Druid and builds models on Looker to report concurrency, track user journeys and build interaction events-based funnels.

#2 User Mapping Across Multiple Platforms

Currently, every tool available in the market used to identify and map a user’s activities across multiple platforms (Web/App) comes with limitations. For eg: a tool could have the functionality to map users across platforms but does not function accurately for user journey funnels. This use case is specifically important if you operate for a parent company with multiple entities or if your business has a presence across the web, android and iOS.

#3 Data Sampling

Due to the high inflow of data, reporting tools sample user data and show the approximate overall counts or drop-offs in a funnel. This might be misleading while taking product-based decisions.

With Inhouse analytics, Dream11 avoids data sampling and therefore, gets an accurate picture of the current product state.

#4 Accelerated Speed of Reporting

Reporting tools are time-consuming. They require significant time to load data to generate unsampled reports. High volumes of data also affect basic reporting. Inhouse Analytics resolves this issue by aggregating data as per business requirements.

#5 Mapping User Interactions and Transaction Data

Deriving or analyzing a correlation between user interactions and transactions is difficult for businesses that don’t require any customer interactions to conduct transactions.

Dream11 used tools to map this, however, it couldn’t solve the use case to map server events data with client events data. Since all the company data sits on a single tool, Inhouse Analytics solves this problem by enabling visibility to unique identifiers for each interaction.

#6 Data Security

Data privacy is a key concern for organizations to maintain competitive advantage. Deploying in-house tools offers an extra layer of security to protect sensitive company data. Additionally, companies should avoid sharing granular data through different tools to maintain high data security standards.

#7 Reduce Cost

Storage and processing charges to use multiple tools is expensive, especially with an increasing number of events. In addition to being a sturdy system, In-house analytics reduces costs to a great extent.

#8 Data Integration with Other Sources

By selectively sharing data,  Dream11’s Inhouse analytics framework enables flexibility to integrate with other sources effectively. For instance, the technology team at Dream11  can build an in-house audience based on the data that they have and feed it to Clevertap to send push notifications to users.

Watch this space to know more about Dream11’s technology stack!