Harvard Business Review, Article #1, Analytics 3.0

HBR  Analytics 3.0
by Thomas H. Davenport Dec 2013
http://hbr.org/2013/12/analytics-30/ar/1

The Evolution of Analytics
Briefly, it is a new resolve to apply:

• Powerful data-gathering 
• Analysis methods 

not just to a company’s operations but also to its offerings to embed data smartness into the products and services customers buy.


History 

• The use of data to make decisions is not a new idea.
• It is as old as decision making itself. 
• But the field of business analytics was born in the mid-1950s

Analytics 1.0 the era of “business intelligence

• For the first time
• data about production processes 
• sales
• customer interactions 
• and more 

         were 

• recorded 
• aggregated
• analyzed
• This was the era of the enterprise data warehouse

• used to capture information 

•  business intelligence software

• used to query and report it

• Readying a data set for inclusion in a warehouse was difficult

• Analysts spent much of their time preparing data for analysis and relatively little time on the analysis itself

Analytics 2.0 the era of big data

• At the mid 2000s, when internet-based and social network firms primarily in Silicon Valley (Google, eBay, ...) began to amass and analyze new kinds of information
• Big data also came to be distinguished from small data because it was not generated purely by a firm’s internal transaction systems
• Big data couldn’t fit or be analyzed fast enough on a single server
•  So it was processed with Hadoop, an open source software framework for fast batch data processing across parallel servers
• To deal with relatively unstructured data companies turned to a new class of databases known as NoSQL
• Machine-learning methods (semi automated model development and testing) were used to rapidly generate models from the fast-moving data

Analytics 3.0 the era of data-enriched offerings

• The pioneering big data firms in Silicon Valley began investing in analytics to support customer-facing products, services, and features
• The common thread in these companies management to compete on analytics 
• not only in the traditional sense (by improving internal business decisions) 
• but also by creating more-valuable products and services
• Any company, in any industry, can develop valuable products and services from their aggregated data
• This strategic change in focus means a new role for analytics within organizations

Ten Requirements for Capitalizing on Analytics 3.0

1. Multiple types of data, often combine
2. A new set of data management options
1. In the 1.0 era, firms used data warehouses as the basis for analysis. 
2. In the 2.0 era, they focused on Hadoop clusters and NoSQL databases. 
3. Today the technology answer is “all of the above”
3. Faster technologies and methods of analysis
4. Embedded analytics
1. Consistent with the increased speed of data processing and analysis, models in Analytics 3.0 are often embedded into operational and decision processes
2. Some firms are embedding analytics into fully automated systems through 
1. Scoring algorithms 
2. Analytics-based rules.
5. Data discovery
6. Cross-disciplinary data teams
1. Companies now employ data hackers, who excel at extracting and structuring information, to work with analysts, who excel at modeling it
7. Chief analytics officers
8. Prescriptive analytics
1. There have always been three types of analytics:
1. Descriptive, which reports on the past
2. Predictive, which uses models based on past data to predict the future 
3. Prescriptive, which uses models to specify optimal behaviors and actions
2. Analytics 3.0 includes all three types, 
3. Analytics 3.0  emphasizes the last
9. Analytics on an industrial scale 
1. Analytics 3.0 provides an opportunity to scale those processes to industrial strength
2. Creating many more models through machine learning can let an organization become much more granular and precise in its predictions
10. New ways of deciding and managing 
1. Some of the changes prompted by the widespread availability of big data will not yield much certainty
2. Big data flows continuously consider the analysis of brand sentiment derived from social media sources and so metrics will inevitably rise and fall over time 
3. Additional uncertainty arises from the nature of big data relationships. Unless they are derived from formal testing, the results from big data generally involve correlation, not causation

Creating Value in the Data Economy

• Analytics 3.0 is the direction of change and the new model for competing on analytics.

Thomas H. Davenport is the President’s Distinguished Professor of IT and Management at Babson College, a fellow of the MIT Center for Digital Business, a senior adviser to Deloitte Analytics, and a cofounder of the International Institute for Analytics (for which the ideas in this article were generated). He is a coauthor of Keeping Up with the Quants (Harvard Business Review Press, 2013) and the author of Big Data at Work (forthcoming from Harvard Business Review Press).

Google Analytics - Reporting

Building Reports with Dimensions & Metrics

The building blocks of every report in Google Analytics are :

• Dimensions : A dimension describes characteristics of your data
• Metrics : Metrics are the quantitative measurements of your data

Reporting API

To use the reporting APIs, you have to build your own application. This application needs to be able to write and send a query to the reporting API. The API uses the query to retrieve data from the aggregate tables, and then sends a response back to your application containing the data that was requested.

Each query sent to the API must contain specific information, including the ID of the view that you would like to retrieve data from, the start and end dates for the report, and the dimensions and metrics you want. Within the query you can also specify how to filter, segment and order the data just like you can with tools in the online reporting interface.

Report Sampling

Report sampling is an analytics practice that generates reports based on a small, random subset of your data instead of using all of your  available data. Sampling lets programs, including Google Analytics, calculate the data for your reports faster than if every single piece of data is included during the generation process.

When does sampling happen?

• During processing
• Modifying one of the standard reports in Google Analytics by adding a segment, secondary dimension, or another customization

Adjusting the sample size

The number of sessions used to calculate the report is called the “sample size.” .If you increase the sample size, you’ll include more sessions in your calculation, but it’ll take longer to generate your report. If you decrease the the sample size, you’ll include fewer sessions in your calculation, but your report will be generated faster.

The sampling limit

Google Analytics sets a maximum number of sessions that can be used to calculate your reports. If you go over that limit, your data gets sampled.

One way to stay below the limit is to shorten the date range in your report, which reduces the number of sessions Google Analytics needs to calculate your request

Google Analytics - Processing & configuration

These two components work together to organize and transform the data that you collect into the information you see in your reports.

Processing data and applying your configuration settings

During processing, there are four major transformations that happen to the data. You can control parts of these transformations using the configuration settings in your Properties and Views.

• First, Google Analytics organizes the hits you’ve collected into users and sessions. There is a standard set of rules that Google Analytics follows to differentiate users and sessions, but you can customize some of these rules through your configuration settings.
• Second, data from other sources can be joined with data collected via the tracking code. For example, you can configure Google Analytics to import data from Google AdWords, Google AdSense or Google Webmaster Tools. You can even configure Google Analytics to import data from other non-Google systems.
• Third, Google Analytics processing will modify your data according to any configuration rules you’ve added. These configurations tell Google Analytics what specific data to include or exclude from your reports, or change the way the data’s formatted.
• Finally, the data goes through a process called “aggregation.” During this step, Google Analytics prepares the data for analysis by organizing it in meaningful ways and storing it in database tables. This way, your reports can be generated quickly from the database tables whenever you need them.

How hits are organized by users

First, let’s talk about how Google Analytics creates users. The first time a device loads your content and a hit is recorded, Google Analytics creates a random, unique ID that is associated with the device. Each unique ID is considered to be a unique user in Google Analytics. This unique ID is sent to Google Analytics in each hit, and every time a new ID is detected, Google Analytics counts a New User. When Google Analytics sees an existing ID in a hit, it counts a Returning User.

How hits are organized into sessions

A session in Google Analytics is a collection of interactions, or hits, from a specific user during a defined period of time. These interactions can include pageviews, events or e-commerce transactions.

A single user can have multiple sessions. Those sessions can occur on the same day, or over several days, weeks, or months. As soon as one session ends, there is then an opportunity to start a new session. But how does Google Analytics know that a session has ended?

By default, a session ends after 30 minutes of inactivity

Importing data into Google Analytics

The most common way to get data into Google Analytics is through your tracking code, but you can also add data from other sources. By adding data into Google Analytics, you can give more context to your analysis.

There are two ways to add data into your Google Analytics account without using the tracking code: through account linking and through Data Import. Both are managed via your Configuration settings in the Admin section of Google Analytics. Any data that you add from these sources will be processed along with all the hits you collect from the tracking code

Account linking

You can link various Google products directly to Google Analytics via your account settings. This includes:

• Google AdWords
• Google AdSense
• Google Webmaster Tools

Data Import

In addition to account linking, you can add data to Google Analytics using the Data Import feature. This might include advertising data, customer data, product data, or any other data.

To import data into Google Analytics there must a “key” that exists both in the data that Google Analytics collects and in the data you want to import.

There are two ways to import data into Google Analytics:

• Dimension Widening : With Dimension Widening, you can import just about any data into Google Analytics
• Cost Data Import : The other kind of data import is called Cost Data Import. You use this feature specifically to add data that shows the amount of money you spent on your non-Google advertising. Importing cost data lets Google Analytics calculate the return-on-investment of your non-Google ads

Transforming & Aggregating Data

An important part of processing is data transformation and aggregation

Common configuration settings: 

• Filters : Filters provide a flexible way you can modify the data within each view. You can use them to exclude data, include data, or actually change how the data looks in your reports. Filters help you transform the data so it’s better aligned with your reporting needs.

• Goals : Another way to transform your data is to set up Goals. When you set up Goals, Google Analytics creates new metrics for your reports

• Channel Grouping and Content Grouping : Grouping is another way you can transform your data. With grouping, you can aggregate certain pieces of data together so you can analyze the collective performance. You can create two types of groups in Google Analytics: Channel groups and Content groups.

Data aggregation

All of your configuration settings, including Filters, Goals, and Grouping, are applied to your data before it goes through aggregation, the final step of data processing.

During aggregation, Google Analytics creates and organizes your report dimensions into tables, called aggregate tables. Google Analytics pre-calculates your reporting metrics for each value of a dimension and stores them in the corresponding table. When you open a Google Analytics report, a query is sent to the aggregate tables that are full of this prepared data, and returns the specific dimensions and metrics for the report. Storing data in these tables makes it faster for your reports to access data when you request it.

Google Analytics - Data collection

Google Analytics uses tracking code to collect data. It doesn’t matter if you are tracking a website, mobile app or other digital environment -- it’s the tracking code that gathers and sends the data back to your account for reporting.

How tracking works

Depending on the environment you want to track -- a website, mobile app, or other digital experience -- Google Analytics uses different tracking technology to create the data hits. For example, there is specific tracking code to create hits for websites and different code to create hits for mobile apps.

In addition to creating hits, the tracking code also performs another critical function. It identifies new users and returning users. We’ll explain how the tracking code does this in later lessons.

Another key function of the tracking code is to connect your data to your Google Analytics account. This is accomplished through a unique identifier embedded in your tracking code.

Google Analytics uses different tracking technology to measure user activity depending on the specific environment you want to track -- websites, mobile apps, or other digital experiences. To track data from a website, Google Analytics provides a standard snippet of JavaScript tracking code. This snippet references a JavaScript library called analytics.js that controls what data is collected.

Adding the Google Analytics JavaScript code to your website

You simply add the standard code snippet before the closing </head> tag in the HTML of every web page you want to track. This snippet generates a pageview hit each time a page is loaded. It’s essential that you place the Google Analytics tracking code on every page of your site

Collecting and sending data with the Measurement Protocol

The Measurement Protocol lets you send data to Google Analytics from any web-connected device. Recall that the Google Analytics JavaScript and mobile SDKs automatically build hits to send data to Google Analytics. However, when you want to collect data from a different device, you must manually build the data collection hits. The Measurement Protocol defines how to construct the hits and how to send them to Google Analytics.

Google Analytics - Platform Principles

There are four main parts of the Google Analytics platform:

• Collection
• Configuration
• Processing
• Reporting

Collection

Collection is all about getting data into your Google Analytics account.

To collect data, you need to add Google Analytics code to your website, mobile app or other digital environment you want to measure. This tracking code provides a set of instructions to Google Analytics, telling it which user interactions it should pay attention to and which data it should collect. The way the data is collected depends on the environment you want to track.

Processing & Configuration

During data processing, Google Analytics transforms the raw data from collection using the settings in your Google Analytics account. These settings, also known as the configuration, help you align the data more closely with your measurement plan and business objectives.  

Reporting

After Google Analytics has finished processing, you can access and analyze your data using the reporting interface, which includes easy-to-use reporting tools and data visualizations. It’s also possible to systematically access your data using the Google Analytics Core Reporting API. Using the API you can build your own reporting tools or extract your data directly into third-party reporting tools.

Conclusion

Throughout the rest of this course, we will dive deeper into key topics about collection, configuration, processing and reporting. Having a comprehensive understanding of each of these platform components will help you better understand the data you see in Google Analytics. It will also prepare you for more advanced topics about how you can customize your data.

Data Model

There are three components to this data model:

• Users : Is a visitor to your website or app,
• Sessions : Is the time they spend there
• Interactions : Is what they do while they’re there

Project 5 Exercise 3

 Exercise 3 - Busiest Hour

import sys
import string
import logging

from util import mapper_logfile
logging.basicConfig(filename=mapper_logfile, format='%(message)s',
                    level=logging.INFO, filemode='w')

def mapper():
    """
    In this exercise, for each turnstile unit, you will determine the date and time 
    (in the span of this data set) at which the most people entered through the unit.
    
    The input to the mapper will be the final Subway-MTA dataset, the same as
    in the previous exercise. You can check out the csv and its structure below:
    https://www.dropbox.com/s/meyki2wl9xfa7yk/turnstile_data_master_with_weather.csv

    For each line, the mapper should return the UNIT, ENTRIESn_hourly, DATEn, and 
    TIMEn columns, separated by tabs. For example:
    'R001\t100000.0\t2011-05-01\t01:00:00'

    Since you are printing the output of your program, printing a debug 
    statement will interfere with the operation of the grader. Instead, 
    use the logging module, which we've configured to log to a file printed 
    when you click "Test Run". For example:
    logging.info("My debugging message")
    """


    for line in sys.stdin:
        data = line.strip().split(",")
        if len(data) == 22 and data[6] == 'ENTRIESn_hourly':
            continue
               
        print "{0}\t{1}\t{2}\t{3}".format(data[1],data[6],data[2],data[3])
        

mapper()

---------------------------------------------------------------------------------------------------------------------

import sys

import logging

from util import reducer_logfile

logging.basicConfig(filename=reducer_logfile, format='%(message)s',

                    level=logging.INFO, filemode='w')

def reducer():

    '''

    Write a reducer that will compute the busiest date and time (that is, the 

    date and time with the most entries) for each turnstile unit. Ties should 

    be broken in favor of datetimes that are later on in the month of May. You 

    may assume that the contents of the reducer will be sorted so that all entries 

    corresponding to a given UNIT will be grouped together.

    

    The reducer should print its output with the UNIT name, the datetime (which 

    is the DATEn followed by the TIMEn column, separated by a single space), and 

    the number of entries at this datetime, separated by tabs.

    For example, the output of the reducer should look like this:

    R001    2011-05-11 17:00:00   31213.0

    R002 2011-05-12 21:00:00   4295.0

    R003 2011-05-05 12:00:00   995.0

    R004 2011-05-12 12:00:00   2318.0

    R005 2011-05-10 12:00:00   2705.0

    R006 2011-05-25 12:00:00   2784.0

    R007 2011-05-10 12:00:00   1763.0

    R008 2011-05-12 12:00:00   1724.0

    R009 2011-05-05 12:00:00   1230.0

    R010 2011-05-09 18:00:00   30916.0

    ...

    ...

    Since you are printing the output of your program, printing a debug 

    statement will interfere with the operation of the grader. Instead, 

    use the logging module, which we've configured to log to a file printed 

    when you click "Test Run". For example:

    logging.info("My debugging message")

    '''

    max_entries = 0.0

    old_key = None

    datetime = ''

    for line in sys.stdin:

        data = line.strip().split('\t')

        if len(data) != 4:

            continue

        this_key, count, date, time  = data

        count = float(count)

        if old_key and old_key != this_key:

            print "{0}\t{1}\t{2}".format(old_key, datetime, max_entries)

            max_entries = 0

            datetime = ''

        old_key = this_key

        if count >= max_entries:

            max_entries = count

            datetime = str(date) + ' ' + str(time)

    if old_key != None:

        print "{0}\t{1}\t{2}".format(old_key, datetime, max_entries)

        logging.info("{0}\t{1}\t{2}".format(old_key, datetime, max_entries))

reducer()

Project 5 Exercise 2

Exercise 2 - Ridership by Weather Type


import sys
import string
import logging

from util import mapper_logfile
logging.basicConfig(filename=mapper_logfile, format='%(message)s',
                    level=logging.INFO, filemode='w')

def mapper():
    '''
    For this exercise, compute the average value of the ENTRIESn_hourly column 
    for different weather types. Weather type will be defined based on the 
    combination of the columns fog and rain (which are boolean values).
    For example, one output of our reducer would be the average hourly entries 
    across all hours when it was raining but not foggy.

    Each line of input will be a row from our final Subway-MTA dataset in csv format.
    You can check out the input csv file and its structure below:
    https://www.dropbox.com/s/meyki2wl9xfa7yk/turnstile_data_master_with_weather.csv
    
    Note that this is a comma-separated file.

    This mapper should PRINT (not return) the weather type as the key (use the 
    given helper function to format the weather type correctly) and the number in 
    the ENTRIESn_hourly column as the value. They should be separated by a tab.
    For example: 'fog-norain\t12345'
    
    Since you are printing the output of your program, printing a debug 
    statement will interfere with the operation of the grader. Instead, 
    use the logging module, which we've configured to log to a file printed 
    when you click "Test Run". For example:
    logging.info("My debugging message")
    '''

    # Takes in variables indicating whether it is foggy and/or rainy and
    # returns a formatted key that you should output.  The variables passed in
    # can be booleans, ints (0 for false and 1 for true) or floats (0.0 for
    # false and 1.0 for true), but the strings '0.0' and '1.0' will not work,
    # so make sure you convert these values to an appropriate type before
    # calling the function.
    def format_key(fog, rain):
        return '{}fog-{}rain'.format(
            '' if fog else 'no',
            '' if rain else 'no'
        )


    for line in sys.stdin:
        data = line.strip().split(',');
       
        if len(data) !=22 or data[6] == "ENTRIESn_hourly":
            continue
        else:
            print "{0}\t{1}".format(format_key(float(data[14]),float(data[15])), data[6])
            logging.info("{0}\t{1}".format(format_key(float(data[14]),float(data[15])), data[6]))
       

mapper()
-------------------------------------------------------------------------------------------------------------------------------------------------- import sys
import logging
import numpy
from util import reducer_logfile
logging.basicConfig(filename=reducer_logfile, format='%(message)s',
                    level=logging.INFO, filemode='w')

def reducer():
    '''
    Given the output of the mapper for this assignment, the reducer should
    print one row per weather type, along with the average value of
    ENTRIESn_hourly for that weather type, separated by a tab. You can assume
    that the input to the reducer will be sorted by weather type, such that all
    entries corresponding to a given weather type will be grouped together.

    In order to compute the average value of ENTRIESn_hourly, you'll need to
    keep track of both the total riders per weather type and the number of
    hours with that weather type. That's why we've initialized the variable 
    riders and num_hours below. Feel free to use a different data structure in 
    your solution, though.

    An example output row might look like this:
    'fog-norain\t1105.32467557'

    Since you are printing the output of your program, printing a debug 
    statement will interfere with the operation of the grader. Instead, 
    use the logging module, which we've configured to log to a file printed 
    when you click "Test Run". For example:
    logging.info("My debugging message")
    '''

    entries = 0.0
    avg = 0.0
    num  = 0
    old_key = None

    for line in sys.stdin:
        data = line.strip().split("\t")
        if len(data) !=2:
            continue
        this_key, count = data
        
        if old_key and old_key != this_key:
            print "{0}\t{1}".format(old_key,avg)
            entries = 0
            num = 0
        old_key = this_key
        entries += float(count)
        num += 1
        avg =  entries / num
        
    if old_key != None:
        print "{0}\t{1}".format(old_key, avg)
        logging.info("{0}\t{1}".format(old_key, avg))


reducer()