Campbellsville University Uncertainty of Big Data Paper Please read the question and attached PDF throughly As large volumes of raw and complex data, Big

Campbellsville University Uncertainty of Big Data Paper Please read the question and attached PDF throughly

As large volumes of raw and complex data, Big data enables programmers to take better decisions and optimize business processes by understanding customer behavior, latest trends, and changing patterns. An enterprise can collect, store, and analyze these large datasets in a number of ways. An enterprise can even use robust big data tools to store, access, and manage the structured and unstructured data collected from various sources in a faster and more efficient way. But no business can utilize and leverage big data optimally without addressing a number of challenges.

Big data challenges are numerous: Big data projects have become a normal part of doing business — but that doesn’t mean that big data is easy. The most obvious challenge associated with big data is simply storing and analyzing all that information. In its Digital Universe report, IDC estimates that the amount of information stored in the world’s IT systems is doubling about every two years. By 2020, the total amount will be enough to fill a stack of tablets that reaches from the earth to the moon 6.6 times. And enterprises have responsibility or liability for about 85 percent of that information. Much of that data is unstructured, meaning that it doesn’t reside in a database. Documents, photos, audio, videos and other unstructured data can be difficult to search and analyze.

In order for organizations to capitalize on the opportunities offered by big data, they are going to have to do some things differently. And that sort of change can be tremendously difficult for large organizations.

While this weeks topic highlighted the uncertainty of Big Data, the author identified the following as areas for future research. Pick one of the following for your Research paper.:

Additional study must be performed on the interactions between each big data characteristic, as they do not exist separately but naturally interact in the real world.
The scalability and efficacy of existing analytics techniques being applied to big data must be empirically examined.
New techniques and algorithms must be developed in ML and NLP to handle the real-time needs for decisions made based on enormous amounts of data.
More work is necessary on how to efficiently model uncertainty in ML and NLP, as well as how to represent uncertainty resulting from big data analytics.
Since the CI algorithms are able to find an approximate solution within a reasonable time, they have been used to tackle ML problems and uncertainty challenges in data analytics and process in recent years.
To do:
Read the required reading
Complete the Research Paper

Your paper should meet the following requirements:

• Be approximately 5 pages in length, not including the required cover page and reference page.

• Follow APA guidelines. Your paper should include an introduction, a body with fully developed content, and a conclusion.

• Support your response with the readings from the course and at least five peer-reviewed articles or scholarly journals to support your positions, claims, and observations. The UC Library is a great place to find resources.

• Be clear with well-written, concise, using excellent grammar and style techniques. You are being graded in part on the quality of your writing. (2019) 6:44
Hariri et al. J Big Data
https://doi.org/10.1186/s40537-019-0206-3
Open Access
SURVEY PAPER
Uncertainty in big data analytics: survey,
opportunities, and challenges
Reihaneh H. Hariri* , Erik M. Fredericks and Kate M. Bowers
*Correspondence:
rhosseinzadehha@oakland.
edu
Oakland University,
Rochester, MI, USA
Abstract
Big data analytics has gained wide attention from both academia and industry as the
demand for understanding trends in massive datasets increases. Recent developments
in sensor networks, cyber-physical systems, and the ubiquity of the Internet of Things
(IoT) have increased the collection of data (including health care, social media, smart
cities, agriculture, finance, education, and more) to an enormous scale. However, the
data collected from sensors, social media, financial records, etc. is inherently uncertain due to noise, incompleteness, and inconsistency. The analysis of such massive
amounts of data requires advanced analytical techniques for efficiently reviewing and/
or predicting future courses of action with high precision and advanced decisionmaking strategies. As the amount, variety, and speed of data increases, so too does the
uncertainty inherent within, leading to a lack of confidence in the resulting analytics
process and decisions made thereof. In comparison to traditional data techniques and
platforms, artificial intelligence techniques (including machine learning, natural language processing, and computational intelligence) provide more accurate, faster, and
scalable results in big data analytics. Previous research and surveys conducted on big
data analytics tend to focus on one or two techniques or specific application domains.
However, little work has been done in the field of uncertainty when applied to big data
analytics as well as in the artificial intelligence techniques applied to the datasets. This
article reviews previous work in big data analytics and presents a discussion of open
challenges and future directions for recognizing and mitigating uncertainty in this
domain.
Keywords: Big data, Uncertainty, Big data analytics, Artificial intelligence
Introduction
According to the National Security Agency, the Internet processes 1826 petabytes (PB)
of data per day [1]. In 2018, the amount of data produced every day was 2.5 quintillion bytes [2]. Previously, the International Data Corporation (IDC) estimated that the
amount of generated data will double every 2 years [3], however 90% of all data in the
world was generated over the last 2 years, and moreover Google now processes more
than 40,000 searches every second or 3.5 billion searches per day [2]. Facebook users
upload 300 million photos, 510,000 comments, and 293,000 status updates per day [2, 4].
Needless to say, the amount of data generated on a daily basis is staggering. As a result,
techniques are required to analyze and understand this massive amount of data, as it is a
great source from which to derive useful information.
© The Author(s) 2019. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License
(http://creat​iveco​mmons​.org/licen​ses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium,
provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and
indicate if changes were made.
Hariri et al. J Big Data
(2019) 6:44
Advanced data analysis techniques can be used to transform big data into smart data
for the purposes of obtaining critical information regarding large datasets [5, 6]. As such,
smart data provides actionable information and improves decision-making capabilities
for organizations and companies. For example, in the field of health care, analytics performed upon big datasets (provided by applications such as Electronic Health Records
and Clinical Decision Systems) may enable health care practitioners to deliver effective
and affordable solutions for patients by examining trends in the overall history of the
patient, in comparison to relying on evidence provided with strictly localized or current
data. Big data analysis is difficult to perform using traditional data analytics [7] as they
can lose effectiveness due to the five V’s characteristics of big data: high volume, low
veracity, high velocity, high variety, and high value [7–9]. Moreover, many other characteristics exist for big data, such as variability, viscosity, validity, and viability [10]. Several
artificial intelligence (AI) techniques, such as machine learning (ML), natural language
processing (NLP), computational intelligence (CI), and data mining were designed to
provide big data analytic solutions as they can be faster, more accurate, and more precise for massive volumes of data [8]. The aim of these advanced analytic techniques is
to discover information, hidden patterns, and unknown correlations in massive datasets
[7]. For instance, a detailed analysis of historical patient data could lead to the detection
of destructive disease at an early stage, thereby enabling either a cure or more optimal
treatment plan [11, 12]. Additionally, risky business decisions (e.g., entering a new market or launching a new product) can profit from simulations that have better decisionmaking skills [13].
While big data analytics using AI holds a lot of promise, a wide range of challenges
are introduced when such techniques are subjected to uncertainty. For instance, each of
the V characteristics introduce numerous sources of uncertainty, such as unstructured,
incomplete, or noisy data. Furthermore, uncertainty can be embedded in the entire analytics process (e.g., collecting, organizing, and analyzing big data). For example, dealing
with incomplete and imprecise information is a critical challenge for most data mining
and ML techniques. In addition, an ML algorithm may not obtain the optimal result if
the training data is biased in any way [14, 15]. Wang et al. [16] introduced six main challenges in big data analytics, including uncertainty. They focus mainly on how uncertainty
impacts the performance of learning from big data, whereas a separate concern lies in
mitigating uncertainty inherent within a massive dataset. These challenges normally present in data mining and ML techniques. Scaling these concerns up to the big data level
will effectively compound any errors or shortcomings of the entire analytics process.
Therefore, mitigating uncertainty in big data analytics must be at the forefront of any
automated technique, as uncertainty can have a significant influence on the accuracy of
its results.
Based on our examination of existing research, little work has been done in terms of
how uncertainty significantly impacts the confluence of big data and the analytics techniques in use. To address this shortcoming, this article presents an overview of the
existing AI techniques for big data analytics, including ML, NLP, and CI from the perspective of uncertainty challenges, as well as suitable directions for future research in
these domains. The contributions of this work are as follows. First, we consider uncertainty challenges in each of the 5 V’s big data characteristics. Second, we review several
Page 2 of 16
Hariri et al. J Big Data
(2019) 6:44
techniques on big data analytics with impact of uncertainty for each technique, and also
review the impact of uncertainty on several big data analytic techniques. Third, we discuss available strategies to handle each challenge presented by uncertainty.
To the best of our knowledge, this is the first article surveying uncertainty in big data
analytics. The remainder of the paper is organized as follows. “Background” section presents background information on big data, uncertainty, and big data analytics. “Uncertainty perspective of big data analytics” section considers challenges and opportunities
regarding uncertainty in different AI techniques for big data analytics. “Summary of mitigation strategies” section correlates the surveyed works with their respective uncertainties. Lastly, “Discussion” section summarizes this paper and presents future directions of
research.
Background
This section reviews background information on the main characteristics of big data,
uncertainty, and the analytics processes that address the uncertainty inherent in big data.
Big data
In May 2011, big data was announced as the next frontier for productivity, innovation,
and competition [11]. In 2018, the number of Internet users grew 7.5% from 2016 to over
3.7 billion people [2]. In 2010, over 1 zettabyte (ZB) of data was generated worldwide
and rose to 7 ZB by 2014 [17]. In 2001, the emerging characteristics of big data were
defined with three V’s (Volume, Velocity, and Variety) [18]. Similarly, IDC defined big
data using four V’s (Volume, Variety, Velocity, and Value) in 2011 [19]. In 2012, Veracity
was introduced as a fifth characteristic of big data [20–22]. While many other V’s exist
[10], we focus on the five most common characteristics of big data, as next illustrated in
Fig. 1.
Volume refers to the massive amount of data generated every second and applies to the
size and scale of a dataset. It is impractical to define a universal threshold for big data
volume (i.e., what constitutes a ‘big dataset’) because the time and type of data can influence its definition [23]. Currently, datasets that reside in the exabyte (EB) or ZB ranges
are generally considered as big data [8, 24], however challenges still exist for datasets in
smaller size ranges. For example, Walmart collects 2.5 PB from over a million customers every hour [25]. Such huge volumes of data can introduce scalability and uncertainty
problems (e.g., a database tool may not be able to accommodate infinitely large datasets).
Many existing data analysis techniques are not designed for large-scale databases and
can fall short when trying to scan and understand the data at scale [8, 15].
Variety refers to the different forms of data in a dataset including structured data,
semi-structured data, and unstructured data. Structured data (e.g., stored in a relational database) is mostly well-organized and easily sorted, but unstructured data
(e.g., text and multimedia content) is random and difficult to analyze. Semi-structured
data (e.g., NoSQL databases) contains tags to separate data elements [23, 26], but
enforcing this structure is left to the database user. Uncertainty can manifest when
converting between different data types (e.g., from unstructured to structured data),
in representing data of mixed data types, and in changes to the underlying structure of the dataset at run time. From the point of view of variety, traditional big data
Page 3 of 16
Hariri et al. J Big Data
(2019) 6:44
Fig. 1 Common big data characteristics
analytics algorithms face challenges for handling multi-modal, incomplete and noisy
data. Because such techniques (e.g., data mining algorithms) are designed to consider
well-formatted input data, they may not be able to deal with incomplete and/or different formats of input data [7]. This paper focuses on uncertainty with regard to big
data analytics, however uncertainty can impact the dataset itself as well.
Efficiently analysing unstructured and semi-structured data can be challenging,
as the data under observation comes from heterogeneous sources with a variety of
data types and representations. For example, real-world databases are negatively
influenced by inconsistent, incomplete, and noisy data. Therefore, a number of data
preprocessing techniques, including data cleaning, data integrating, and data transforming used to remove noise from data [27]. Data cleaning techniques address data
quality and uncertainty problems resulting from variety in big data (e.g., noise and
inconsistent data). Such techniques for removing noisy objects during the analysis
process can significantly enhance the performance of data analysis. For example, data
cleaning for error detection and correction is facilitated by identifying and eliminating mislabeled training samples, ideally resulting in an improvement in classification
accuracy in ML [28].
Velocity comprises the speed (represented in terms of batch, near-real time, real time,
and streaming) of data processing, emphasizing that the speed with which the data is
processed must meet the speed with which the data is produced [8]. For example, Internet of Things (IoT) devices continuously produce large amounts of sensor data. If the
device monitors medical information, any delays in processing the data and sending the
results to clinicians may result in patient injury or death (e.g., a pacemaker that reports
emergencies to a doctor or facility) [20]. Similarly, devices in the cyber-physical domain
often rely on real-time operating systems enforcing strict timing standards on execution,
Page 4 of 16
Hariri et al. J Big Data
(2019) 6:44
and as such, may encounter problems when data provided from a big data application
fails to be delivered on time.
Veracity represents the quality of the data (e.g., uncertain or imprecise data). For
example, IBM estimates that poor data quality costs the US economy $3.1 trillion per
year [21]. Because data can be inconsistent, noisy, ambiguous, or incomplete, data veracity is categorized as good, bad, and undefined. Due to the increasingly diverse sources
and variety of data, accuracy and trust become more difficult to establish in big data
analytics. For example, an employee may use Twitter to share official corporate information but at other times use the same account to express personal opinions, causing problems with any techniques designed to work on the Twitter dataset. As another example,
when analyzing millions of health care records to determine or detect disease trends,
for instance to mitigate an outbreak that could impact many people, any ambiguities or
inconsistencies in the dataset can interfere or decrease the precision of the analytics process [21].
Value represents the context and usefulness of data for decision making, whereas the
prior V’s focus more on representing challenges in big data. For example, Facebook,
Google, and Amazon have leveraged the value of big data via analytics in their respective
products. Amazon analyzes large datasets of users and their purchases to provide product recommendations, thereby increasing sales and user participation. Google collects
location data from Android users to improve location services in Google Maps. Facebook monitors users’ activities to provide targeted advertising and friend recommendations. These three companies have each become massive by examining large sets of raw
data and drawing and retrieving useful insight to make better business decisions [29].
Uncertainty
Generally, “uncertainty is a situation which involves unknown or imperfect information”
[30]. Uncertainty exists in every phase of big data learning [7] and comes from many different sources, such as data collection (e.g., variance in environmental conditions and
issues related to sampling), concept variance (e.g., the aims of analytics do not present
similarly) and multimodality (e.g., the complexity and noise introduced with patient
health records from multiple sensors include numerical, textual, and image data). For
instance, most of the attribute values relating to the timing of big data (e.g., when events
occur/have occurred) are missing due to noise and incompleteness. Furthermore, the
number of missing links between data points in social networks is approximately 80% to
90% and the number of missing attribute values within patient reports transcribed from
doctor diagnoses are more than 90% [31]. Based on IBM research in 2014, industry analysts believe that, by 2015, 80% of the world’s data will be uncertain [32].
Various forms of uncertainty exist in big data and big data analytics that may negatively impact the effectiveness and accuracy of the results. For example, if training
data is biased in any way, incomplete, or obtained through inaccurate sampling, the
learning algorithm using corrupted training data will likely output inaccurate results.
Therefore, it is critical to augment big data analytic techniques to handle uncertainty.
Recently, meta-analysis studies that integrate uncertainty and learning from data
have seen a sharp increase [33–35]. The handling of the uncertainty embedded in the
entire process of data analytics has a significant effect on the performance of learning
Page 5 of 16
Hariri et al. J Big Data
(2019) 6:44
from big data [16]. Other research also indicates that two more features for big data,
such as multimodality (very complex types of data) and changed-uncertainty (the
modeling and measure of uncertainty for big data) is remarkably different from that of
small-size data. There is also a positive correlation in increasing the size of a dataset
to the uncertainty of data itself and data processing [34]. For example, fuzzy sets may
be applied to model uncertainty in big data to combat vague or incorrect information
[36]. Moreover, and because the data may contain hidden relationships, the uncertainty is further increased.
Therefore, it is not an easy task to evaluate uncertainty in big data, especially when
the data may have been collected in a manner that creates bias. To combat the many
types of uncertainty that exist, many theories and techniques have been developed to
model its various forms. We next describe several common techniques.
Bayesian theory assumes a subjective interpretation of the probability based on past
event/prior knowledge. In this interpretation the probability is defined as an expression of a rational agent’s degrees of belief about uncertain propositions [37]. Belief
function theory is a framework for aggregating imperfect data through an information fusion process when under uncertainty [38]. Probability theory incorporates
randomness and generally deals with the statistical characteristics of the input data
[34]. Classification entropy measures ambiguity between classes to provide an index
of confidence when classifying. Entropy varies on a scale from zero to one, where values closer to zero indicate more complete classification in a single class, while values
closer to one indicate membership among several different classes [39]. Fuzziness is
used to measure uncertainty in classes, notably in human language (e.g., good and
bad) [16, 33, 40]. Fuzzy logic then handles the uncertainty associated with human
perception by creating an approximate reasoning mechanism [41, 42]. The methodology was intended to imitate human reasoning to better handle uncertainty in the
real world [43]. Shannon’s entropy quantifies the amount of information in a variable
to determine the amount of missing information on average in a random source [44,
45]. The concept of entropy in statistics was introduced into the theory of communication and transmission of information by Shannon [46]. Shannon entropy provides
a method of information quantification when it is not possible to measure criteria weights using a decision–maker. Rough set theory provides a mathematical tool
for reasoning on vague, uncertain or incomplete information. With the rough set
approach, concepts are described by two approximations (upper and lower) instead of
one precise concept [47], making such methods invaluable to dealing with uncertain
information systems [48]. Probabilistic theory and Shannon’s entropy are often used
to model imprecise, incomplete, and inaccurate data. Moreover, fuzzy set and rough
theory are used for modeling vague or ambiguous data [49], as shown in Fig. 2.
Evaluating the level of uncertainty is a critical step in big data analytics. Although
a variety of techniques exist to analyze big data, the accuracy of the analysis may be
negatively affected if uncertainty in the data o…
Purchase answer to see full
attachment