Allegany College Data Source Manipulation Database and Big Data Bibliography For each data source listed below, do the following: 1. Write out the steps yo

Allegany College Data Source Manipulation Database and Big Data Bibliography For each data source listed below, do the following:
1. Write out the steps you’d use to clean/wrangle the data for your database. This can be a simple step by step process, pseudocode, sql, python, perl, or other language. These steps can be used to clean the data, remove unnecessary information, or normalize. If it’s just a website, write down what data you’d scrape from it.

2. Find one other dataset public or private that you could use in conjunction with this dataset.

3. What inferences could be made by using these two datasets together.

Datasets:

a. 2017-2014 Candy Hierarchy Data – Data from a survey across 4 years showing peoples preference in halloween candy. https://www.scq.ubc.ca/so-much-candy-data-seriously/ (Links to an external site.)

b. FDA’s National Drug Code Directory – https://www.fda.gov/drugs/drug-approvals-and-databases/national-drug-code-directory (Links to an external site.)

c. The Avengers Death Database – https://github.com/fivethirtyeight/data/tree/master/avengers (Links to an external site.)

d. Bachelor/Bachelorette Dataset – https://github.com/fivethirtyeight/data/blob/master/bachelorette/bachelorette.csv (Links to an external site.)

e. Daily Show Guests – https://github.com/fivethirtyeight/data/blob/master/daily-show-guests/daily_show_guests.csv IT 385
Managing Big Data
Professor Tim Eagle
Module 1:
Course Introduction
Course Description
?
?
?
?
This course introduces the concepts and technology behind big data including technology
to handle the four tenets: volume, veracity, variability and velocity.
The technology is explored that will enable the large volumes of data including tools such
as Hadoop.
Data collection techniques will be examined and students will perform a series of
activities, including data scraping, data wrangling, and data merging. Extraction,
transformation, and loading (ETL) techniques will be developed with a focus on the quality
of data.
Technology to store and analyze structured, unstructured, and text data will be evaluated
and the new technology evolving such as NOSQL databases will be researched in detail.
Course Objectives
?
?
?
?
?
?
?
De?ne “big data”;
Identify the business value of big data in an organization;
Identify where data is, inside and outside the organization, including clickstream data,
government data, and social media data;
Manage data (text and metrics) effectively with Big Data tools;
Develop new knowledge that will support the use of data for data analytics;
Analyze and present data in ways that are meaningful to others using data visualization
techniques; and
Use knowledge gained in a collaborative way to improve both productivity and decision
making.
How to get an ‘A’
?
?
?
?
?
4 Assignments
5 Quizzes
(on Canvas)
Project
Final Exam
Participation Exercises
40%
15%
20%
15%
10%
Class Schedule
See Syllabus
Books and Materials Needed
Yau, Nathan: Visualize This: The FlowingData Guide to Design,
Visualization and Statistics. Wiley Publishing, 2011
Zikopoulos, P. et al: Harness the Power of Big Data, The IBM Big Data
Platform, McGraw-Hill, 2013 (ebook) (uploaded to course documents
already)
Open source software and vendor free downloads will be used in this
course.
Academic Integrity & Class Conduct
1.
Don’t Cheat, you are only cheating yourself
a.
2.
Work with me if you are having any issues
a.
3.
4.
5.
Also, I’ll probably catch you.
Email is best way to contact me.
Try to be to class on-time
When possible, let me know ahead of time if you are going to miss class
Legal Disclaimer to make my employer
happy
All opinions expressed during this class are mine and do not represent the views of the
Department of Labor, Department of Labor, Of?ce of Inspector General, or that of the U.S.
Government.
Fill free to ask me any questions, don’t be scared by the legal disclaimer.
About Me
?
Marymount Alumnus
?
?
?
?
B.S. in Computer Science, Minor in
Web Design
M.S. in Computer Science
SFS Cyber Corp Graduate
Work Experience
?
The George Washington University
?
Certi?cate in Computer Security Info.
Assurance
?
?
?
?
?
?
US Dept. of Labor, OIG, OI-LRF,
Division of Advanced Technology and
Analytics, Branch Chief of Data
Science
US Dept. of ED, OIG, ITACCI,
Technology Crimes Division
US GAO
US Dept. of Justice
Otsuka Pharmaceutical
US Postal Service
Verizon Wireless
Contacting/Stalking Me
Online
?
?
?
?
?
Facebook – www.facebook.com/timeagle
Twitter – www.twitter.com/timeagle
reddit – reddit.com/u/fozzie33
Email – timothy.eagle@marymount.edu
Other sources too, let me know if you locate me in other places.
Of?ine
?
Don’t have an of?ce, but you can ?nd me before or after class
Intro to the Course
?
?
?
?
?
?
?
De?ning “Big Data”
Scope of “Big Data”
Data Scales
Supporting Decision Making
Telling Stories with Data
The Data Scientist
Knowledge and Skills
Big Data
(how you de?ned it)
Big Data (Dictionary.com De?nition)
extremely large data sets that may be analyzed
computationally to reveal patterns, trends, and
associations, especially relating to human behavior
and interactions.
How should we de?ne Big Data?
IBM uses the four Vs
?
?
?
?
Volume
Variety
Velocity
Veracity
The Scope of Big Data
?
?
?
Big Data is changing the way governments, organizations, and academic institutions
conduct business and make discoveries.
Seemingly EVERYTHING we are doing is generating some data point somewhere. There is
the potential to use that data and discover potential connections, and predict outcomes in
way that wasn’t possible before.
Being able to identify the right data is the key.
Data Scales
?
?
?
?
Big Data is less about the data itself and more about what you do with the data.
The application of statistics and statistical principles on the data helps you extract the
information it contains.
Your business data contain a variety of metrics, each representing a particular
characteristic of an object or entity (e.g., customer, transaction). To make sense of these
metrics, you need to understand the types of measurement scales that are used to
represent each metric.
There are four types of measurement scales, nominal, ordinal, interval and ratio.
?
?
?
?
Variables that represent the nominal scale include any type of variable that can be grouped into
mutually exclusive groups: customer name, customer marital status, store location, etc.
Variables that represent the ordinal scale include any type of variable that can be ordered in some
manner.
Variables that are described using the interval measurement scale represent things where the
differences between scale values are equal.
Ratio Scale is similar to Interval, but has a true Zero point.
Supporting Decision Making
Data -> Visualization -> Dashboard of Visualizations
KPI’s
Bleeding into Sports
?
?
?
Sabermetrics
NFL Analytics Departments
Soccer
Telling Stories with Data
?
?
?
Data itself is boring.
Visualizations give the data meaning and tells the story of it.
What to Look For
?
?
?
?
Design
?
?
?
?
?
?
Patterns
Relationships
Questionable Data
Explaining encodings
Label Axes
Keeping Geometry in check
Inclusion of Sources
Consider the Audiences
Always investigate data with a critical eye, and ?gure out the purpose of the graphic.
Other Visuals
https://www.gapminder.org/tools/#$state$time$value=1800;;&chart-type=bubbles
https://informationisbeautiful.net/visualizations/mega-what-the-worlds-biggest-and-most-notable-power-plants/
https://informationisbeautiful.net/visualizations/diversity-in-tech/
https://www.economist.com/graphic-detail/2020/01/03/the-prospect-of-parenthood-makes-people-more-law-abi
ding
Data Science
The Data Scientist
(What you said)
The Data Scientist
?
?
?
It’s a multi-disciplinary job, but typically involves an analytical data expert. They are part
mathematician, part computer scientist, and part trend-spotter.
The term Data Scientist means different things to different people.
It’s constantly evolving it’s meaning.
Knowledge and Skills
Knowledge and Skills
The End
Next Week
Don’t forget readings for
next week.
IT 385
Managing Big Data
Professor Tim Eagle
Module 2:
Business of Big Data
Data-Driven Decision Making
?
DDDM – The process of making organizational decisions based on actual data rather than
intuition or observation alone.
?
Quantitative or/and Qualitative
How to Make Data-Driven Decisions
1. Know Your Mission
Know your Mission – A well-rounded data analyst knows the business well and posses sharp
organizational acumen. Ask yourself what the problems are in your given industry and
competitive market. Identify and understand them thoroughly. Establishing this foundational
knowledge will equip you to make better inferences with your data later on.
Before you begin collecting data, you should start by identifying the business questions that you
want to answer to achieve your organizational goals. By determining the precise questions you
need to know to inform your strategy, you’ll be able to streamline the data collection process
and avoid wasting resources.
How to Make Data-Driven Decisions
2. Identify Data Sources
Put together the sources from which you’ll be extracting your data. You might be coordinating
information from different databases, web-driven feedback forms, and even social media.
Coordinating your various sources seems simple, but ?nding common variables among each
dataset can present a tremendously dif?cult problem. It can be easy to settle for the immediate
goal of utilizing the data for your current purpose alone, but it’s wise to determine whether or
not this data could also be used for additional projects in the future. If so, you should strive to
develop a strategy to present the data in a way that’s accessible in other scenarios as well.
How to Make Data-Driven Decisions
3. Clean and Organize Data
Surprisingly, 80 percent of a data analyst’s time is devoted to cleaning and organizing data, and
only 20 percent is spent actually performing analysis. This so-called “80/20 rule” illustrates the
importance of having clean, orderly information before you can attempt to interpret what it
might mean for your organization.
The term “data cleaning” refers to the process of preparing raw data for analysis by removing or
correcting data that is incorrect, incomplete, or irrelevant. To do so, start by building tables to
organize and catalog what you’ve found. Create a data dictionary—a table that catalogs each of
your variables and translates them into what they mean to you in the context of this particular
project. This information could include data type and other processing factors, as well.
How to Make Data-Driven Decisions
3. Perform Statistical Analysis
Once you’ve thoroughly cleaned the data, you can begin to analyze the information using statistical
models. At this stage, you will start to build models to test your data and answer the business questions
you identi?ed earlier in the process. Testing different models such as linear regressions, decision trees,
random forest modeling, and others can help you determine which method is best suited to your data
set.
Here, you will also need to decide how to present the information in order to answer the question at
hand. There are three different ways to demonstrate your ?ndings:
?
?
?
Descriptive Information: Just the facts.
Inferential Information: The facts, plus an interpretation of what those facts indicate in the
context of a particular project.
Predictive Information: An inference based upon facts and advice for further action based on
your reasoning.
Clarifying how the information will be most effectively presented will help you remain organized when it
comes time to interpret the data.
How to Make Data-Driven Decisions
5. Draw Conclusions
The last step in data-driven decision making is coming to a conclusion. Ask yourself, “What new
information did you learn from the collection of statistics?” Despite pressure to discover
something entirely new, a great place to start is by asking yourself questions to which you
already know—or think you know—the answer.
Many companies make frequent assumptions about their products or market. For example, they
might believe, “A market for this product exists,” or, “This is what our customers want.” But
before seeking out new information, ?rst put existing assumptions to the test. Proving these
assumptions are correct will give you a foundation to work from. Alternatively, disproving these
assumptions will allow you to eliminate any false claims that have, perhaps unknowingly, been
negatively impacting your company. Keep in mind that an exceptional data-driven decision
usually generates more questions than answers.
Characteristics of Big Data
Adoption Status
How Big Data Analytics is adopted
Finding Key Performance Indicators
A Key Performance Indicator is a measurable value that demonstrates how effectively a
company is achieving key business objectives. Organizations use KPIs at multiple levels to
evaluate their success at reaching targets. High-level KPIs may focus on the overall performance
of the business, while low-level KPIs may focus on processes in departments such as sales,
marketing, HR, support and others.
Questions to Ask when Creating KPI’s
?
?
?
?
What does success look like?
How will it be known if performance is trending away from
goals?
What data would the stakeholders and sponsors be most
interested in?
What data is available to the team?
3 M’s for KPI’s
? Meaningful
? Measurable
? Manageable
My actual Organizational KPI’s
Use Cases
Breakout into groups from Canvas (it was done randomly)
Determining Value
?
?
?
Cost Savings Analysis
Return on Investment
Customer Lifetime Value – the dollar value of customer relationship based on the present
value of the projected cash ?ows from that customer. Even though the term “lifetime”
suggest that the calculated value encompasses entire future relationship with the
customer, practical uses are limited to a year and maximum to 3-5 years span.
Total pro?t of an average client over the relationship’s lifetime including all sales LESS
advertising, marketing and incremental product or service ful?llment expenses EQUALS
Lifetime Value of the Customer.
4-V’s Values
?
?
?
?
Volume-based value: The more comprehensive your 360-degree view of customers and the more
historical data you have on them, the more insight you can extract from it all and, all things
considered, the better decisions you can make in the process of acquiring, retaining, growing and
managing those customer relationships.
Velocity-based value: The more customer data you can ingest rapidly into your big-data platform
and the more questions that a user can pose more rapidly against that data (via queries, reports,
dashboards, etc.) within a given time period prior, the more likely you are to make the right
decision at the right time to achieve your customer relationship management objectives.
Variety-based value: The more varied customer data you have – from the CRM system, social
media, call-center logs, etc. – the more nuanced portrait you have on customer pro?les, desires
and so on, hence the better-informed decisions you can make in engaging with them.
Veracity-based value: The more consolidated, conformed, cleansed, consistent current the data
you have on customers, the more likely you are to make the right decisions based on the most
accurate data.
The End
? Next Week
? Don’t forget readings
for next week.
? We will be discussing
Data Sources
? Quiz 1 and Assignment
1 will available next
week.

Purchase answer to see full
attachment