IBM Predictive Analytics

About Big Data Analytics

The 5 V’s of Big Data

Too often in the hype and excitement around Big Data, the conversation gets complicated very quickly. Data scientists and technical experts bandy around terms like Hadoop, Pig, Mahout, and Sqoop, making us wonder if we’re talking about information architecture or a Dr. Seuss book. Business executives who want to leverage the value of Big Data analytics in their organisation can get lost amidst this highly-technical and rapidly-emerging ecosystem.

Overview - IBM Big Data Platform


In an effort to simplify Big Data, many experts have referenced the “3 V’s”: Volume, Velocity, and Variety. In other words, is information being generated at a high volume (e.g. terabytes per day), with a rapid rate of change, encompassing a broad range of sources including both structured and unstructured data? If the answer is yes then it falls into the Big Data category along with sensor data from the “internet of things”, log files, and social media streams. The ability to understand and manage these sources, and then integrate them into the larger Business Intelligence ecosystem can provide previously unknown insights from data and this understanding leads to the “4th V” of Big Data – Value.



There is a vast opportunity offered by Big Data technologies to discover new insights that drive significant business value. Industries are seeing data as a market differentiator and have started reinventing themselves as “data companies”, as they realise that information has become their biggest asset. This trend is prevalent in industries such as telecommunications, internet search firms, marketing firms, etc. who see their data as a key driver for monetisation and growth. Insights such as footfall traffic patterns from mobile devices have been used to assist city planners in designing more efficient traffic flows. Customer sentiment analysis through social media and call logs have given new insights into customer satisfaction. Network performance patterns have been analysed to discover new ways to drive efficiencies. Customer usage patterns based on web click-stream data have driven innovation for new products and services to increase revenue. The list goes on.

IBM predictive analytics with Apache Spark: Coding optional, possibilities endless


Key to success in any Big Data analytics initiative is to first identify the business needs and opportunities, and then select the proper fit-for-purpose platform. With the array of new Big Data technologies emerging at a rapid pace, many technologists are eager to be the first to test the latest Dr. Seuss-termed platform. But each technology has a unique specialisation, and might not be aligned to the business priorities. In fact, some identified use cases from the business might be best suited by existing technologies such as a data warehouse while others require a combination of existing technologies and new Big Data systems.



With this integration of disparate data systems comes the 5th V – Veracity, i.e. the correctness and accuracy of information.



Behind any information management practice lies the core doctrines of Data Quality, Data Governance, and Metadata Management, along with considerations for Privacy and Legal concerns.

Big Data & Analytics Architecture


Big Data needs to be integrated into the entire information landscape, not seen as a stand-alone effort or a stealth project done by a handful of Big Data experts.



Big data analytics is the use of advanced analytic techniques against very large, diverse data sets that include different types such as structured/unstructured and streaming/batch, and different sizes from terabytes to zettabytes. Big data is a term applied to data sets whose size or type is beyond the ability of traditional relational databases to capture, manage, and process the data with low-latency. And it has one or more of the following characteristics – high volume, high velocity, or high variety. Big data comes from sensors, devices, video/audio, networks, log files, transactional applications, web, and social media - much of it generated in real time and in a very large scale.

What’s new in predictive analytics: IBM SPSS and IBM decision optimization


Analyzing big data allows analysts, researchers, and business users to make better and faster decisions using data that was previously inaccessible or unusable. Using advanced analytics techniques such as text analytics, machine learning, predictive analytics, data mining, statistics, and natural language processing, businesses can analyze previously untapped data sources independent or together with their existing enterprise data to gain new insights resulting in significantly better and faster decisions.

• Advanced analytics enables you to find deeper insights and drive real-time actions.
• With advanced analytics capabilities, you can understand what happened, what will happen and what should happen.
• Easily engage both business and technical users to uncover opportunities and address big issues. Operationalize analytics into business processes

Prescriptive analytics

What if you could make strategic decisions based not only on what has occurred or is likely to occur in the future, but through targeted recommendations based on why and how things happen? Prescriptive analytics technology recommends actions based on desired outcomes, taking into account specific scenarios, resources and knowledge of past and current events. This insight can help your organization make better decisions and have greater control of business outcomes.

Prescriptive analytics is the next step on the path to insight-based actions. It creates value through synergy with predictive analytics, which analyzes data to predict a future outcome. Prescriptive analytics takes that insight to the next level by suggesting the optimal way to handle that future situation. Organizations that can act fast in dynamic conditions and make superior decisions in uncertain environments gain a strong competitive advantage.

IBM prescriptive analytics solutions provide organizations in commerce, financial services, healthcare, government and other highly data-intensive industries with a way to analyze data and transform it into recommended actions almost instantaneously. These solutions combine predictive models, deployment options, localized rules, scoring and optimization techniques to form a powerful foundation for decision management. For example, you can:

• Automate complex decisions and trade-offs to better manage limited resources.
• Take advantage of a future opportunity or mitigate a future risk.
• Proactively update recommendations based on changing events.
• Meet operational goals, increase customer loyalty, prevent threats and fraud, and optimize business processes.

The information management big data and analytics capabilities include :

Data Management & Warehouse: Gain industry-leading database performance across multiple workloads while lowering administration, storage, development and server costs; Realize extreme speed with capabilities optimized for analytics workloads such as deep analytics, and benefit from workload-optimized systems that can be up and running in hours.

Hadoop System: Bring the power of Apache Hadoop to the enterprise with application accelerators, analytics, visualization, development tools, performance and security features.

Stream Computing: Efficiently deliver real-time analytic processing on constantly changing data in motion and enable descriptive and predictive analytics to support real-time decisions. Capture and analyze all data, all the time, just in time. With stream computing, store less, analyze more and make better decisions faster.

Content Management: Enable comprehensive content lifecycle and document management with cost-effective control of existing and new types of content with scale, security and stability.

Information Integration & Governance: Build confidence in big data with the ability to integrate, understand, manage and govern data appropriately across its lifecycle.

From insight to action: Predictive and prescriptive analytics

The 5 game changing big data use cases

While much of the big data activity in the market up to now has been experimenting and learning about big data technologies, IBM has been focused on also helping organizations understand what problems big data can address.

We’ve identified the top 5 high value use cases that can be your first step into big data:

Big Data Exploration
Find, visualize, understand all big data to improve decision making. Big data exploration addresses the challenge that every large organization faces: information is stored in many different systems and silos and people need access to that data to do their day-to-day work and make important decisions.

What is the Big Data Exploration use case?

Big data exploration addresses the challenge faced by every large organization: business information is spread across multiple systems and silos and people need access to that data to meet their job requirements and make important decisions. Big Data Exploration enables you to explore and mine big data to find, visualize, and understand all your data to improve decision making. By creating a unified view of information across all data sources - both inside and outside of your organization - you gain enhanced value and new insights.

Ask yourself:

• Are you struggling to manage and extract value from the growing volume and variety of data and need to unify information across federated sources?
• Are you unable to relate “raw” data collected from system logs, sensors, or click streams with customer and line-of-business data managed in your enterprise systems?
• Do you risk exposing unsecure personal information and/or privileged data due to lack of information awareness?

Introduction to apache spark v3



Enhanced 360º View of the Customer
Extend existing customer views by incorporating additional internal and external information sources. Gain a full understanding of customers—what makes them tick, why they buy, how they prefer to shop, why they switch, what they’ll buy next, and what factors lead them to recommend a company to others.

IBM Watson Analytics Presentation

What is the Enhanced 360º View of the Customer big data use case?

With the onset of the digital revolution, the touch points between an organization and its customers have increased many times over; organizations now require specialized solutions to effectively manage these connections. An enhanced 360-degree view of the customer is a holistic approach that takes into account all available and meaningful information about the customer to drive better engagement, more revenue and longterm loyalty. It combines data exploration, data governance, data access, data integration and analytics in a solution that harnesses the volume, velocity and variety. IBM provides several important capabilities to help you make effective use of big data and improve the customer experience.

Ask yourself:

• Do you need a deeper understanding of customer sentiment from both internal and external sources?
• Do you want to increase customer loyalty and satisfaction by understanding what meaningful actions are needed?
• Are you challenged to get the right information to the right people to provide customers what they need to solve problems, cross-sell, and up-sell?

If you answered yes to any of the above questions, the enhanced 360 view of the customer use case is the best starting point for your big data journey.

With Enhanced 360º View of the Customer, you can:

Improve campaign effectiveness
Accurate, targeted cross-sell / up-sell
Retain your most profitable customers
Deliver superior customer experience at the point of service

Security Intelligence Extension

Lower risk, detect fraud and monitor cyber security in real time. Augment and enhance cyber security and intelligence analysis platforms with big data technologies to process and analyze new types (e.g. social media, emails, sensors, Telco) and sources of under-leveraged data to significantly improve intelligence, security and law enforcement insight.

What is the Security Intelligence big data use case?

The growing number of high-tech crimes - cyber-based terrorism, espionage, computer intrusions, and major cyber fraud - poses a real threat to every individual and organization. To meet the security challenge, businesses need to augment and enhance cyber security and intelligence analysis platforms with big data technologies to process and analyze new data types (e.g. social media, emails, sensors, Telco) and sources of under-leveraged data. Analyzing data in-motion and at rest can help find new associations or uncover patterns and facts to significantly improve intelligence, security and law enforcement insight.

Ask yourself:

• Do you need to enrich your security or intelligence system with underleveraged or unused data sources (video, audio, smart devices, network, Telco, social media)?
• Are you able to address the need for sub second detection, identification, resolution of physical or cyber threats?
• Are you able to follow activities of criminals, terrorists, or persons in a blacklist and detect criminal activity before it occurs?

If you answered yes to any of the above questions, the security intelligence extension use case is the best starting point for your big data journey.
There are three main areas for Security Intelligence Extension>

Enhanced intelligence and surveillance insight. Analyzing data in-motion and at rest can help find new associations or uncover patterns and facts. This type of real or near real-time insight can be invaluable and even life-saving.

Real-time cyber attack prediction & mitigation. So much of our lives are spent online, and the growing number of high-tech crimes, including cyber-based terrorism, espionage, computer intrusions, and major cyber fraud, pose a real threat to potentially everyone. By analyzing network traffic, organizations can discover new threats early and react in real time.

Crime prediction & protection. The ability to analyze internet (e.g. email, VOIP), smart devices (e.g. location, call detail records) and social media data can help law enforcement organizations better detect criminal threats and gather criminal evidence. Instead of waiting for a crime to be committed, they can prevent them from happening in the first place and proactively apprehend criminals.

With Security Intelligence Extension, organizations can:

• Sift through massive amounts of data - both inside and outside your organization - to uncover hidden relationships, detect patterns, and stamp out security threats
• Uncover fraud by correlating real-time and historical account activity to uncover abnormal user behavior and suspicious transactions
• Examine new sources and varieties of data for evidence of criminal activity, such as internet, mobile devices, transactions, email, and social media

Operations Analysis
Analyze a variety of machine and operational data for improved business results. The abundance and growth of machine data, which can include anything from IT machines to sensors and meters and GPS devices requires complex analysis and correlation across different types of data sets. By using big data for operations analysis, organizations can gain real-time visibility into operations, customer experience, transactions and behavior.

What is the Operations Analysis big data use case?

Operations Analysis focuses on analyzing machine data, which can include anything from IT machines to sensors, meters and GPS devices. It’s growing at exponential rates and comes in large volumes and a variety of formats, including in-motion, or streaming data. Leveraging machine data requires complex analysis and correlation across different types of data sets. By using big data for operations analysis, organizations can gain real-time visibility into operations, customer experience, transactions and behavior.

Ask yourself:

• Do you have real-time visibility into your business operations including customer experience and behavior?
• Are you able to analyze all your machine data and combine it with enterprise data to provide a full view of business operations?
• Are you proactively monitoring end-to-end infrastructure to avoid problems?

If you answered yes to any of the above questions, the Operations Analysis use case is the best starting point for your big data journey.

Through Operations Analysis, organizations can:

• Gain real-time visibility into operations, customer experience and behavior
• Analyze massive volumes of machine data with sub-second latency to identify events of interest as they occur
• Apply predictive models and rules to identify potential anomalies or opportunities
• Optimize service levels in real-time by combining operational and enterprise data

Data Warehouse Modernization
Integrate big data and data warehouse capabilities to increase operational efficiency. Optimize your data warehouse to enable new types of analysis. Use big data technologies to set up a staging area or landing zone for your new data before determining what data should be moved to the data warehouse. Offload infrequently accessed or aged data from warehouse and application databases using information integration software and tools.

IBM Big Data Analytics Concepts and Use Cases


What is the Data Warehouse Modernization big data use case?

Data Warehouse Modernization (formerly known as Data Warehouse Augmentation) is about building on an existing data warehouse infrastructure, leveraging big data technologies to ‘augment’ its capabilities. There are three key types of Data Warehouse Modernizations:

• Pre-Processing - using big data capabilities as a “landing zone” before determining what data should be moved to the data warehouse
• Offloading - moving infrequently accessed data from data warehouses into enterprise-grade Hadoop
• Exploration - using big data capabilities to explore and discover new high value data from massive amounts of raw data and free up the data warehouse for more structured, deep analytics.

Ask yourself:

• Are you integrating big data and data warehouse capabilities to increase operational efficiency?
• Have you taken steps to migrate rarely used data to new technologies like Hadoop to optimize storage, maintenance and licensing costs?
• Are you using stream computing to filter and reduce storage costs? 
• Are you leveraging structured, unstructured, and streaming data sources required for deep analysis?
• Do you have a lot of cold, or low-touch data that is driving up costs or slowing performance?

If you answered yes to any of the above questions, the Data Warehouse Modernization use case is the best starting point for your big data journey.

With Data Warehouse Modernization, organizations can:

• Combine streaming and other unstructured data sources to existing data warehouse investments
• Optimize data warehouse storage and provide query-able archive
• Rationalize the data warehouse for greater simplicity and lower cost
• Provide better query performance to enable complex analytical applications
• Deliver improved business insights to operations for real-time decision-making

Analytics and Big Data are pointless without good and Accurate Data. That is why IBM Launched the IBM DataFirst  http://www.ibmbigdatahub.com/blog/chief-takeaways-ibm-datafirst-launch-event







More Information:

http://enterprisearchitects.com/the-5v-s-of-big-data/

https://www.ibm.com/blogs/watson-health/the-5-vs-of-big-data/

http://www.ibm.com/analytics/us/en/technology/biginsights/

https://developer.ibm.com/predictiveanalytics/2015/11/06/spss-algorithms-optimized-for-apache-spark-spark-algorithms-extending-spss-modeler/

https://www.opendatascience.com

https://aws.amazon.com/blogs/big-data/crunching-statistics-at-scale-with-sparkr-on-amazon-emr/

https://developer.ibm.com/predictiveanalytics/category/predictive-analytics/

http://www.ibm.com/analytics/us/en/events/datafirst/

http://www.ibm.com/analytics/us/en/services/datafirst.html

http://www.ibmbigdatahub.com/blog/chief-takeaways-ibm-datafirst-launch-event

https://www.devrelate.com/ibm-developerworks-datafirst/

https://www.devrelate.com/blog/

https://www.youracclaim.com/org/ibm/badge/the-ibm-datafirst-method

https://www.eventbrite.com/e/ibm-datafirst-launch-event-registration-26725283041

https://www.ibm.com/blogs/bluemix/2016/01/applications-and-microservices-with-docker-and-containers/

https://www.ibm.com/blogs/bluemix/2015/12/bluemix-framework-for-microservices-architecture/

https://www.researchgate.net/figure/281404634_fig1_Figure-1-The-five-V's-of-Big-Data-Adapted-from-IBM-big-data-platform-Bringing-big

http://www.ibmbigdatahub.com/infographic/four-vs-big-data

https://bigdatauniversity.com

https://bigdatauniversity.com/courses/

https://www-01.ibm.com/software/data/bigdata/what-is-big-data.html

https://www-01.ibm.com/software/data/bigdata/use-cases/security-intelligence.html