IBM's Cloud approach and how they will change the Cloud Game

The cloud game

The “cloud” marketplace is diverse and nuanced. In the public cloud, while IaaS is a competitive game, each of the big players has different strengths and strategies. AWS is the clear leader in the space, Google has strong data services driving AI/ML offerings, Microsoft has the most advanced hybrid strategy (Azure public & private cloud options) and strength in business productivity applications. IBM was fighting to be a top five public cloud contender, and while it has a strong portfolio of applications and solid IaaS offering through the Softlayer acquisition, it was failing to compete. Red Hat’s strength is playing across all ecosystems, both Linux (Red Hat Enterprise Linux RHEL is the foundation of its business) and OpenShift (container and application platform including Kubernetes) are available across a wide spectrum of public clouds and infrastructure platforms.



Buying Red Hat does not mean the end of IBM Cloud, rather it is a signal that IBM’s position in a multi-cloud world is one of partnership. I first heard the term “co-opetition” in relation to IBM many years ago; Big Blue has decades of managing the inherent tension of working with partners where there are also competitive angles. Cloud computing is only gaining in relevance, and open source is increasingly important to end-users. IBM’s relevance in multi-cloud is greatly enhanced with the Red Hat acquisition, and IBM’s positioning with the C-suite should help Red Hat into more strategic relationships. This is not a seismic shift in the cloud landscape.

Changing perceptions

It is very difficult for a 100 year old company to change how people think of it. It is not fair for people to think that IBM is a navy suit selling mainframes – while of course some people still wear suits, and IBM zSeries has been doing well, the company has gone through tons of changes. IBM is one of the few companies that has avoided being destroyed from disruptions in the technology industry.

IBM Cloud Innovation Day 

While many (including myself) are a sad to see the “leader in open source” acquired, there are few companies outside of Red Hat that have as good of a record with the communities as IBM. Microsoft rejuvenated its image with a new CEO who focuses on not only cloud and AI, but diversity and openness (Microsoft was on stage this year at Red Hat Summit). IBM says that they will keep the Red Hat brand and products; will they be able to leverage Red Hat’s culture and community strength to bolster both IBM sales and relevance in the marketplace? There is an opportunity for IBM and Red Hat to both rebrand and reposition how they are considered by customers and partners in the new digital world.

The Wikibon and SiliconANGLE teams will continue to examine all of the angles of this acquisition. Here’s a 20 minute video with Dave Vellante and me sharing our initial thoughts:

IBM $34B Red Hat Acquisition: Pivot To Growth But Questions Remain

Most companies are just getting started on their cloud journey.

They’ve maybe completed 10 to 20 percent of the trek, with a focus on cost and productivity efficiency, as well as scaling compute power. There’s a lot more though to unlock in that remaining 80 percent: shifting business applications to the cloud and optimizing supply chains and sales, which will require moving and managing data across multiple clouds.
To accomplish those things easily and securely, businesses need an open, hybrid multicloud approach. While most companies acknowledge they are embracing hybrid multicloud environments, well over half attest to not having the right tools, processes or strategy in place to gain control of them.
Here are four of the recent steps IBM has taken to help our clients embrace just that type of approach.

The Future of Data Warehousing, Data Science and Machine Learning


1. IBM to acquire Red Hat.
The IBM and Red Hat partnership has spanned 20 years. IBM was an early supporter of Linux, collaborating with Red Hat to help develop and grow enterprise-grade Linux and more recently to bring enterprise Kubernetes and hybrid multicloud solutions to customers. By joining together, we will be positioned to help companies create cloud-native business applications faster and drive greater portability and security of data and applications across multiple public and private clouds, all with consistent cloud management. Read the Q&A with Arvind Krishna, Senior Vice President, IBM Hybrid Cloud.

2. The launch of IBM Multicloud Manager.
When applications and data are distributed across multiple environments, it can be a challenge for enterprises to keep tabs on all their workloads and make sure they’re all in the right place. The new IBM Multicloud Manager solution helps organizations tackle that challenge by helping them improve visibility across all their Kubernetes environments using a single dashboard to maintain security and governance and automate capabilities. Learn why multicloud management is becoming critical for enterprises.

3. AI OpenScale improves business AI.
AI OpenScale will be available on IBM Cloud and IBM Cloud Private with the goal of helping businesses operate and automate artificial intelligence (AI) at scale, no matter where the AI was built or how it runs. AI OpenScale heightens visibility, detects bias and makes AI recommendations and decisions fully traceable. Neural Network Synthesis (NeuNetS), a beta feature of the solution, configures to business data, helping organizations scale AI across their workflows more quickly.

4. New IBM Security Connect community platform.
IBM Security Connect is a new cloud-based community platform for cyber security applications. With the support of more than a dozen other technology companies, it is the first cloud security platform built on open federated technologies. Built using open standards, IBM Security Connect can help companies develop microservices, create new security applications, integrate existing security solutions, and make use of data from open, shared services. It also enables organizations to apply machine learning and AI, including Watson for Cyber Security, to analyze and identify threats or risks.

Successful enterprises innovate. They listen, learn and experiment, and after doing so, they either lead or adapt. Or, for those who do not, they risk failure.

Building a Kubernetes cluster in the IBM Cloud


Possibly nowhere is this more evident than in cloud computing, an environment driven by user demand and innovation. Today the innovation focuses squarely on accelerating the creation and deployment of integrated, hybrid clouds. Whether on public, private or on-premises systems, more companies are demanding interoperability to enable scalability, agility, choice, performance – and no vendor lock-in. Simultaneously, they want to integrate all of it with their existing massive technology investments.

Paul Cormier, Executive Vice President, and President Products and Technologies, Red Hat, and Arvind Krishna, Senior Vice President, IBM Hybrid Cloud, at Red Hat Summit 2018 in San Francisco on May 7, 2018.

One of the fundamental building blocks of this burgeoning integrated, hybrid cloud environment is the software container – a package of software that includes everything needed to run it. Through containers, which are lightweight, easily portable and OS-independent, organizations can create and manage applications that can run across clouds with incredible speed.

This leads to a truly flexible environment that is optimized for automation, security, and easy scalability. IBM recognized the value of containers several years ago, and has been aggressively building out easy-to-use cloud services and capabilities with Docker, Kubernetes, and our own technologies.



In addition, IBM has moved to containerize all of its portfolio middleware, including everything from WebSphere to Db2 to API Connect to Netcool.

This strategy fits with our view that the core of the cloud is flexibility. There is no “one size fits all.” This idea is what led to the creation of the IBM Cloud Private platform, which delivers containers and services that help accelerate clients’ journey to the cloud.

And that’s why today IBM and Red Hat are coming together to enable the easy integration of IBM’s extensive middleware and data management with Red Hat’s well-entrenched Red Hat OpenShift open source container application platform. This is a major extension of our long-standing relationship with Red Hat and provides enterprises access to a complete and integrated hybrid cloud environment – from the operating system, to open source components, to Red Hat Enterprise Linux containers, to IBM middleware that has been re-engineered for the cloud era.

IBM Cloud App Management - Manage Kubernetes environments with speed and precision

Both of our companies believe in a hybrid cloud future and, with this partnership, we are ensuring seamless connectivity between private and public clouds so that clients get all the benefits of the cloud in a controlled environment.

With this news, we will be certifying our private cloud platform – IBM Cloud Private – as well as IBM middleware including WebSphere, MQ and Db2, and other key IBM software, to run on Red Hat Enterprise Linux via Red Hat OpenShift Container Platform. Red Hat OpenShift will also be available on IBM public cloud, as well as IBM Power Systems. This allows enterprises to get the best hybrid experience, including on the IBM Cloud, with Red Hat OpenShift and IBM middleware that they have trusted for years.

IBM Cloud Series - 002 - Multi Node Kubernetes Cluster on IBM Cloud


It’s about assisting our clients in their digital transformation journey – a journey that, for many, includes the execution of multiple approaches at the same time: A shift to a cloud-only strategy, the aggressive utilization of public cloud, adding more workloads, and simplification of hybrid data.
Armed with these capabilities, clients can make smarter decisions more quickly, engage with customers more intimately and manage their businesses more profitably.

IBM Cloud Private: The Next-Generation Application Server

Istio 

IBM and Google announced the launch of Istio, an open technology that provides a way for developers to seamlessly connect, manage and secure networks of different microservices—regardless of platform, source or vendor.

Istio is the result of a joint collaboration between IBM, Google and Lyft as a means to support traffic flow management, access policy enforcement and the telemetry data aggregation between microservices. It does all this without requiring developers to make changes to application code by building on earlier work from IBM, Google and Lyft.
developerWorksTV report by Scott Laningham.

Istio currently runs on Kubernetes platforms, such as the IBM Bluemix Container Service. Its design, however, is not platform specific. The Istio open source project plan includes support for additional platforms, including CloudFoundry, VMs.

IBM, Google, and Lyft launch Istio

Why IBM built Istio

We continue to see an increasing number of developers turning to microservices when building their applications. This strategy allows developers to decompose a large application into smaller, more manageable pieces. Although decomposing big applications into smaller pieces is a practice we’ve seen in the field for as long as software has been written, the microservices approach is particularly well suited to developing large scale, continuously available software in the cloud.

We have personally witnessed this trend with our large enterprise clients as they move to the cloud. As microservices scale dynamically, problems such as service discovery, load balancing and failure recovery become increasingly important to solve uniformly. The individual development teams manage and make changes to their microservices independently, making it difficult to keep all of the pieces working together as a single unified application. Often, we see customers build custom solutions to these challenges that are unable to scale even outside of their own teams.

Before combining forces, IBM, Google, and Lyft had been addressing separate, but complementary, pieces of the problem.

IBM’s Amalgam8 project, a unified service mesh that was created and open sourced last year, provided a traffic routing fabric with a programmable control plane to help its internal and enterprise customers with A/B testing, canary releases, and to systematically test the resilience of their services against failures.

Google’s Service Control provided a service mesh with a control plane that focused on enforcing policies such as ACLs, rate limits and authentication, in addition to gathering telemetry data from various services and proxies.

Lyft developed the Envoy proxy to aid their microservices journey, which brought them from a monolithic app to a production system spanning 10,000+ VMs handling 100+ microservices. IBM and Google were impressed by Envoy’s capabilities, performance, and the willingness of Envoy’s developers to work with the community.

It became clear to all of us that it would be extremely beneficial to combine our efforts by creating a first-class abstraction for routing and policy management in Envoy, and expose management plane APIs to control Envoys in a manner that can be easily integrated with CI/CD pipelines. In addition to developing the Istio control plane, IBM also contributed several features to Envoy such as traffic splitting across service versions, distributed request tracing with Zipkin and fault injection. Google hardened Envoy on several aspects related to security, performance, and scalability.

How does Istio work?
Improved visibility into the data flowing in and out of apps, without requiring extensive configuration and reprogramming.

Istio converts disparate microservices into an integrated service mesh by introducing programmable routing and a shared management layer. By injecting Envoy proxy servers into the network path between services, Istio provides sophisticated traffic management controls such as load-balancing and fine-grained routing. This routing mesh also enables the extraction of a wealth of metrics about traffic behavior, which can be used to enforce policy decisions such as fine-grained access control and rate limits that operators can configure. Those same metrics are also sent to monitoring systems. This way, it offers improved visibility into the data flowing in and out of apps, without requiring extensive configuration and reprogramming to ensure all parts of an app work together smoothly and securely.

Once we have control of the communication between services, we can enforce authentication and authorization between any pair of communicating services. Today, the communication is automatically secured via mutual TLS authentication with automatic certificate management. We are working on adding support for common authorization mechanisms as well.

Key partnerships driving open collaboration
We have been working with Tigera, the Kubernetes networking folks who maintain projects like CNI, Calico and flannel, for several months now to integrate advanced networking policies into the IBM Bluemix offerings. As we now look to integrate Istio and Envoy, we are extending that collaboration to include these projects and how we can enable a common policy language for layers 3 through 7.

“It takes more than just open sourcing technology to drive innovation,” said Andy Randall, Tigera co-founder and CEO. “There has to be an open, active multi-vendor community, and as a true believer in the power of open collaboration, IBM is playing an essential role in fostering that community around Kubernetes and related projects including Calico and Istio. We have been thrilled with our partnership and look forward to ongoing collaboration for the benefit of all users of these technologies.”

Key Istio features
Automatic zone-aware load balancing and failover for HTTP/1.1, HTTP/2, gRPC, and TCP traffic.
Fine-grained control of traffic behavior with rich routing rules, fault tolerance, and fault injection.
A pluggable policy layer and configuration API supporting access controls, rate limits and quotas.
Automatic metrics, logs and traces for all traffic within a cluster, including cluster ingress and egress.
Secure service-to-service authentication with strong identity assertions between services in a cluster.

How to use it today
You can get started with Istio here. We also have a sample application composed of four separate microservices that can be easily deployed and used to demonstrate various features of the Istio service mesh.

Project and collaboration
Istio is an open source project developed by IBM, Google and Lyft. The current version works with Kubernetes clusters, but we will have major releases every few months as we add support for more platforms. If you have any questions or feedback, feel free to contact us on istio-users@googlegroups.com mailing list.

We are excited to see early commitment and support for the project from many companies in the community: Red Hat with Red Hat Openshift and OpenShift Application Runtimes, Pivotal with Pivotal Cloud Foundry, Weaveworks with Weave Cloud and Weave Net 2.0, Tigera with the Project Calico Network Policy Engine. If you are also interested in participating in further development of this open source project, please join us at GitHub. If you are an IBM partner/vendor, we encourage you to build solutions on top of Istio to serve your client’s unique needs. As your clients move from monolithic applications to microservices, they can easily manage complex enterprise level microservices running on Bluemix infrastructure using Istio.

Please feel free to reach out to us at istio-users@googlegroups.com if you have any questions.

Data is the new natural resource — abundant, often untamed, and fueling Artificial Intelligence (AI). It has the potential not only to transform business, but to enable the creation of new business models.

But where and how are you expected to begin your data journey? These are two of the most common questions we get asked. For us, it has everything to do with making things like data science and machine learning capabilities, accessible and easy to use across platforms — providing solutions that handle the analytics where the data resides, rather than bringing the data to the analytics.

The Real World with OpenShift - Red Hat DevOps & Microservices


By taking this approach, IBM has been a leader in helping clients around the globe more easily collect, organize, and analyze their growing data volumes, all with the end goal of ascending the AI Ladder. To be clear, that’s not as easy as it sounds. For example, according to a report from MIT Sloan, Reshaping Business with Artificial Intelligence, an estimated 85% of 3,000 business leaders surveyed believed artificial intelligence (AI) would enable competitive advantage, however, only about 20% have done anything about it. For many organizations, the task of understanding, organizing, and managing their data at the enterprise level was too complex.



So earlier this year we set out to change all that and make it easier for enterprises to gain control of their data, to make their data simple, and then to put that data to work to unearth insights into their organizations. We launched IBM Cloud Private for Data, the first true data platform of its kind that integrates data science, data engineering, and app building under one containerized roof that can be run on premises or across clouds.



IBM has been busy adding to the platform ever since. Since launch we’ve added support for MongoDB Enterprise and EDB Postgres; we’ve integrated IBM Data Risk Manager; and we’ve announced support for Red Hat Openshift, to name a few. This week we’re keeping the momentum going, announcing a variety of new updates, from premium add-on services and modular install options, to the availability of the first-of-a-kind Data Virtualization technology.



With these updates, the design criterion was to help organizations modernize their environments even further to take advantage of cloud benefits — flexibility, agility, scalability, cost-efficiency — while keeping their data where it is. Leveraging multi-cloud elasticity and the portability of a microservices-based containerized architecture lets enterprises place their data and process where it most benefits the business.



Here’s how the new capabilities line up:

Premium Add-On Services
We continue to enrich IBM Cloud Private for Data’s service catalog with premium add-on services:
An advanced data science add-on featuring IBM SPSS Modeler, IBM Watson Explorer, and IBM Decision Optimization to help organizations turn data into game-changing insights and actions with powerful ML and data science technologies

Key databases— Mongo DB and Db2 on Cloud

IBM AI OpenScale will soon be available as a single convenient package with IBM Cloud Private for Data, helping businesses operate and automate AI at scale, with trust and transparency capabilities to eliminate bias and explain outcomes.

DB2 family and v11.1.4.4

Data Virtualization

IBM Cloud Private for Data’s data virtualization (announced in September) can help organizations leverage distributed data at the source, eliminating the need to move or centralize their data. Some of the key highlights are:

• Query anything, anywhere — across data silos (heterogeneous data assets)
• Help reduce operational costs with distributed parallel processing (vs centralized processing) — free of data movement, ETL, duplication, etc.
• Auto-discovery of source and metadata, for ease of viewing information across your organization
• Self-discovering, self-organizing cluster
• Unify disparate data assets with simple automation, providing seamless access to data as one virtualized source
• Governance, security, and scalability built in

DB2 12 overview


In essence the service appears as a single Db2 database to all applications.

IBM Cloud Private for Data update

Other Capabilities in this Release

• FISMA Compliance — FIPS Level 1
• Modular Installation — Reduced footprint for base installer by almost 50%. Customers can deploy add-ons as optional features.
• Support for Microsoft Azure by end of year — Adding to existing IBM Cloud Private, Red Hat OpenShift, and OpenStack and Amazon Web Services support

CNCF Reference Architecture


Next steps
As organizations take this journey with us, these new capabilities of IBM Cloud Private for Data can help further modernize and simplify their data estates for multicloud, leverage the best of the open source ecosystem, and infuse their applications and business processes with data science and AI capabilities. We remain committed to helping our clients unlock the value of their data in innovative smarter ways for better, more timely business outcomes. IBM Cloud Private for Data can be that place to start.


IBM® and Red Hat have partnered to provide a joint solution that uses IBM Cloud Private and OpenShift. You can now deploy IBM certified software containers running on IBM Cloud Private onto Red Hat OpenShift.
Similar to IBM Cloud Private, OpenShift is a container platform built on top of Kubernetes. You can install IBM Cloud Private on OpenShift by using the IBM Cloud Private installer for OpenShift.

Integration capabilities

• Supports Linux® 64-bit platform in offline only installation mode
• Single-master configuration
• Integrated IBM Cloud Private cluster management console and Catalog
• Integrated core Platform services, such as monitoring, metering, and logging
• IBM Cloud Private uses the OpenShift image registry

This integration defaults to using the Open Service Broker in OpenShift. Brokers that are registered in OpenShift are still recognized and can contribute to the IBM Cloud Private Catalog. IBM Cloud Private is also configured to use the OpenShift Kube API Server.

Notes:
IBM Cloud Private Platform images are not Red Hat OpenShift certified
IBM Cloud Private Vulnerability Advisor (VA) and audit logging are not available on OpenShift
Not all CLI command options, for example all cloudctl cm commands, are supported

Security

Authentication and authorization administration happens from only IBM Cloud Private to OpenShift. If a user is created in OpenShift, the user is not available in IBM Cloud Private. Authorization is handled by IBM Cloud Private IAM services that integrate with OpenShift RBAC.
The IBM Cloud Private cluster administrator is created in OpenShift during installation. All other users and user-groups from IBM Cloud Private LDAP are dynamically created in OpenShift when the users invoke any Kube API for the first time. The roles for all IBM Cloud Private users and user-groups are mapped to equivalent OpenShift roles. The tokens that are generated by IBM Cloud Private are accepted by the OpenShift Kube API server, OpenShift UI and OpenShift CLI.

What is IBM Cloud Private and OpenShift?

Before getting into the details of the partnership, a little refresher on IBM Cloud Private and Red Hat OpenShift.

OpenShift 4.0 - Features, Functions, Future at OpenShift Commons Gathering Seattle 2018


Cloud Private is IBM’s private cloud platform that enables enterprise IT to employ a hybrid cloud environment on both x86 and POWER platforms.  IBM sees Cloud Private as addressing three enterprise IT needs:


MOOR INSIGHTS & STRATEGY
IBM’s value prop is essentially save money on legacy app support, securely integrate with third parties for implementations such as Blockchain  and simply develop twelve-factor cloud-native applications in a microservices architecture. Important to note in Cloud Private is its ability to run in both POWER and x86 environments.


OpenShift is Red Hat’s enterprise container platform. OpenShift is based on Docker and Kubernetes and manages the hosting, deployment, scaling, and security of containers in the enterprise cloud.

What this partnership enables

As previously mentioned, the partnership extends IBM Cloud Private to Red Hat OpenShift. So, enterprise IT organizations familiar with the Red Hat tools can more simply deploy a cloud environment that brings all its data and apps together in a single console. Legacy line of business (LoB) applications can be deployed and managed alongside native cloud applications. IBM middleware can be deployed in any OpenShift environment.

This partnership also allows a simpler, more secure interface to the power of IBM Cloud Services. The seamless integration from IBM Cloud Private should allow IT organizations to quickly enable services that would normally take months to deploy such as Artificial Intelligence (AI), Blockchain and other platforms.

OpenShift on OpenStack and Bare Metal


PowerAI and RHEL brings Deep Learning to the enterprise
Somewhat hidden in the news of the IBM – Red Hat announcement is what may be the most interesting bit of news. That is, the availability of PowerAI for RHEL 7.5 on the recently updated Power System AC922 server platform.

PowerAI is IBM’s packaging and delivery of performance-tuned frameworks for deep learning such as Tensorflow, Kerras, and Caffe. This should lead to simplified deployment of frameworks, quicker development time and shorter training times. This is the beginning of democratizing deep learning for the enterprise. You can find more on PowerAI here by Patrick Moorhead.

OpenShift Commons Briefing: New Marketplace and Catalog UX for OpenShift 4 Serena Nichols, Red Hat


The IBM POWER System AC922 is the building block of PowerAI. As previously mentioned, this is based on the IBM POWER9 architecture. Why does this matter? In an acronym, I/O. POWER9 has native support for both PCIe Gen4, NVLink 2.0 and CAPI 2.0. Both of these allow for greater I/O capacity and bandwidth. Moreover, what that means to a workload like deep learning is the ability to move more data between (more) storage and compute much faster. This leads to a big decrease in learning time. To an enterprise IT organization, that means faster customer insights, greater efficiencies in manufacturing and a lot of other benefits that drive differentiation from competitors.

What this means for Enterprise IT

There are a few ways this partnership benefits the enterprise IT organization. One of the more obvious benefits is the tighter integration of applications and data, both legacy and cloud-native. Enterprise IT organizations that have gone through the pains of trying to integrate legacy data with newer applications can more easily take advantage of IBMs (and open source) middleware to achieve greater efficiencies.

This partnership also allows enterprise IT to more quickly enable a greater catalog of services to business units looking to gain competitive advantages in the marketplace through IBM Cloud Services.

Perhaps the biggest benefit to enterprise IT is the availability of OpenAI on RHEL. I believe this begins the democratization of AI for the enterprise. This partnership attempts to remove the biggest barriers to adoption by simplifying the deployment and tuning of Deep Learning frameworks.

OpenShift roadmap: You won't believe what's next

How this benefits IBM and Red Hat

IBM can extend the reach of its cloud services to enterprise IT organizations running Red Hat OpenShift. I believe those organizations will quickly be able to understand the real benefits associated with Cloud Private and Cloud Services.

The benefit to Red Hat is maybe a little less obvious, but equally significant. Red Hat’s support for IBM Cloud Private and (by extension) Cloud Services opens the addressable market for OpenShift and enables a new set of differentiated capabilities. In an ever-increasing competitive hybrid cloud management space, this sets Red Hat apart.

On the AI front, I believe this partnership further sets IBM apart as the leader and introduces Red Hat into the discussion for good measure.  This could be a partnership that many try to catch for some time.

Transform the Enterprise with IBM Cloud Private on OpenShift

Closing thoughts

The partnership between IBM and Red Hat has always been strong, and in many ways these solutions offerings only make sense. Red Hat has a strong offering in the developing, deploying and managing cloud-native applications with OpenShift. IBM has a best-of-breed solution in Cloud Private and PowerAI. Marrying the two can empower the enterprise IT organization and extend the datacenter footprint of both Red Hat and IBM.

However, many great technical partnerships never reach their potential because the partnerships end at technical enablement. Red Hat and IBM would be wise to develop a comprehensive go-to-market campaign that focuses on education and awareness. Cross-selling and account seeding is the first step in enabling this partnership, followed by a series of focused campaigns in targeted vertical industries and market segments.

Cloud Native, Event Drive, Serverless, Microservices Framework - OpenWhisk - Daniel Krook, IBM


Finally, the joint consulting services between the IBM Garage and Red Hat Consulting organizations will need to work closely in ensuring early customer success with PowerAI.  Real enterprises realizing real benefits is the difference between a science project and a solution. Moreover, these organizations are going to be critical to helping enterprise IT stand up these deep learning frameworks.

I will be following this partnership closely and look forward to watching how Red Hat and IBM jointly attack the market. Look for a follow up on this as the partnership evolves.

IBM Cloud SQL Query Introduction

More Information:

https://developer.ibm.com/dwblog/2017/istio/

https://medium.com/ibm-analytics/ibm-advances-ibm-cloud-private-for-data-further-with-modularity-premium-add-ons-and-more-cb1f57ce0cfb

https://www.ibm.com/blogs/think/2018/05/ibm-red-hat-containers/

https://www.thecube.net/red-hat-summit-2018

Final Christmas Thought:

Bach - Aria mit 30 Veränderungen Goldberg Variations BWV 988 - Rondeau | Netherlands Bach Society

Oracle Autonomous Database

Oracle Autonomous Database combines the flexibility of cloud with the
power of machine learning to deliver data management as a service. It
enables businesses to:

• Safely run mission-critical workloads using the most secure, available,
performant, and proven platform - Oracle Database on Exadata

• Migrate both new and existing OLTP or Analytics applications

• Deploy in both the Oracle Public Cloud and on Cloud at Customer in their
own data centers, providing the easiest and safest cloud migration and
hybrid cloud enablement

• Cut administration costs up to 80% with full automation of operations and
tuning

• Cut runtime costs up to 90% by billing only for resources needed at any
given time

• Protect themselves from cyber-attacks and rogue employees by
automatically encrypting all data and automatically applying any needed
security updates online

• Guarantee 99.995%1 uptime to ensure mission-critical applications are
always available. Downtime is limited to under 2.5 minutes per month,
including maintenance

Oracle Database Support Best Practices
THE ORACLE AUTONOMOUS DATABASE

Like an autonomous car, the Oracle Autonomous Database (Autonomous Database) provides a level of performance and reliability manually managed databases can’t deliver. Compared to a manually managed database, the Autonomous Database costs less to run, performs better, is more available, and eliminates human error.

Larry Ellison Introduces Oracle Autonomous Database Cloud

Self-Driving

You tell the Autonomous Database the service level to achieve, and it handles the rest. The Autonomous Database eliminates human labor to provision, secure, monitor, backup, recover, troubleshoot, and tune databases. This greatly reduces database maintenance tasks, reducing costs and freeing scarce administrator resources to work on higher value tasks.

Oracle Cloud Infrastructure Weninars:


Since the Autonomous Database is based on the extremely feature rich and proven Oracle Database, on the Exadata platform, it is able to run both OLTP and analytic workloads up to 100X faster. It includes many performance enhancing Exadata features such as smart flash cache, automatic columnar format in flash cache, smart scan, Exafusion communication over the super-fast InfiniBand network, and automatic storage indexes.

Oracle PaaS: Data Integration Plataform Cloud: Product Overview Animated Video


In addition, when it comes time to upgrade or patch, the Autonomous Database can replay the real production workload on a test database to make sure the upgrade does not have any unexpected side effects on a mission-critical system.

Autonomous database automatically tunes itself using Machine Learning algorithms including automatically creating any indexes needed to accelerate applications. Users get the ultimate simplicity of a “load and go” architecture in which they can simply load their data and run SQL without worrying about creating and tuning their database access structures.

Self-Securing

The Autonomous Database is more secure than a manually operated database because it protects itself rather than having to wait for an available administrator. This applies to defenses against both external and internal attacks.

Security patches are automatically applied every quarter. This is much sooner than most manually operated Oracle databases, narrowing an unnecessary window of vulnerability. Patching can also occur off-cycle if a zero-day exploit is discovered. By applying patches in a rolling fashion across the nodes of a cluster, the Autonomous Database secures itself without application downtime.

Oracle Multitenant: New Features in Oracle Database 18c


Patching is just part of the picture. The database also protects itself with always-on encryption. Customers can control their own keys to further improve security.

In the future, Oracle’s Data Masking and Redaction technologies will be used to safeguard sensitive data by concealing it for some users or workloads and masking it on test databases.

Create Autonomous Data Warehouse Cloud Connection with Data Integration Platform Cloud

Self-Repairing

The Autonomous Database is more reliable than a manually operated database. At startup, it automatically establishes a triple-mirrored scale-out configuration in one regional cloud datacenter, with an optional full standby copy in another region. The Autonomous Database automatically recovers from any physical failures whether at the server or datacenter level. It has the ability to rewind data to a point in time in the past to back out user errors. By applying software updates in a rolling fashion across nodes of the cluster, it keeps the application online during updates of the database, clusterware, OS, VM, hypervisor, or firmware.

If the database detects an impending error, it gathers statistics and feeds them to AI diagnostics to determine the root cause. As a final safety net, the Autonomous Database runs nightly backups for you.


Oracle Database Release Uptake and Patching Strategies
In the future, when it is time to update the Autonomous Database, it will be possible to replay the full
production workload on a parallel testing environment to verify the safety of the update before it is applied to a mission-critical environment.

Oracle will offer a 99.995% uptime guarantee for the Autonomous Database. Oracle understands that
mission-critical systems run 24x7. Unlike other cloud vendors, Oracle provides an uptime guarantee that includes planned maintenance and all other common sources of downtime in its calculations.

Optimized for Different Workloads

Modern automobiles are specialized by workload: family car, van, pickup truck, sports car, etc. In the same way, the Autonomous Database consists of a single set of technologies available in multiple products, each tailored to a different workload:

Data Warehousing. The Oracle Autonomous Database for Data Warehousing is the simplest and most
efficient database for data marts, reporting databases, and data warehousing. Available January 2018.

OCI Level 100 Autonomous Database


OLTP and mixed workloads. 

The Oracle Autonomous Database for OLTP is designed to run mission-critical
enterprise applications, including mixed workloads and real-time analytics, with no compromise on app performance. Coming in 2018.
In the future, Oracle will also bring the autonomous principles of self-driving, self-securing, self-repairing to other kinds of databases:

• NoSQL. Delivers transactional operations on JSON documents and key-value data. Available in 2018.
• Graph. Automatically creates graph representations from tabular and JSON data for discovery of new connections through network analysis. Coming in 2018.

Melbourne Groundbreakers Tour - Hints and Tips




In addition, the Autonomous Database provides IT leaders with a cloud-native enterprise-class foundation for new app and data science development.

• Increase app developer productivity. The Autonomous Database instantly provides app developers
with a platform that offers the variety of data management methods their apps require with the simplicity of a self-managing database. App developers simply push a button to provision a mission critical capable database.
• Simplify data science experimentation. Data science, like all science, boils down to experimentation.

The Autonomous Database’s built-in machine learning capabilities along with its self-driving and selfsecuring capabilities, makes it easy for data science teams to experiment with datasets that are
otherwise locked away in operational silos for performance or security reasons.

Oracle Autonomous Database - Fear Not.... Embrace it - AutonomousDB-01


EASY AND FAST TRANSITION TO THE CLOUD

For IT leaders who want to move enterprise IT to a cloud foundation, the Autonomous Database offers the smoothest and easiest transition.

• Oracle Public Cloud, Cloud at Customer, or both. The Autonomous Database runs in both the Oracle Public Cloud and Cloud at Customer environments. This means IT leaders can have the management ease and subscription pricing of cloud for all enterprise workloads, including those that must stay inhouse for regulatory, data sovereignty, or network latency reasons.

• Go cloud-native without app changes. Because the Autonomous Database is still an Oracle database, existing apps can be quickly and easily moved to this new cloud-native data management platform with no app changes.

With Autonomous Database, major cost savings and agility improvements come quickly, not after
years to decades of application rewrites.


Oracle Autonomous Data Warehouse Cloud Service



SAFEST TRANSITION TO THE CLOUD

The transition to the cloud must improve the availability of mission-critical workloads, not put them at risk.

The Autonomous Database is built on top of the most widely proven and sophisticated database in the world:

Oracle Database. The Oracle Database is capable of running any type of workload in a highly secure,
available, and scalable fashion.

The Autonomous Database runs on the best database platform in the world: Exadata. Exadata is a cloudarchitected scale-out platform that uses the latest technologies including NVMe flash and InfiniBand networking, together with unique database optimizations in storage, compute, and networking to deliver leading performance, scaling, and availability, at the lowest cost.

Oracle on Serverless Computing: Developing FaaS with Oracle


Oracle’s long experience and track record ensures that the transition to the cloud is safe and smooth. The largest enterprises and governments in the world already run all types of mission-critical workloads with Oracle Database on Exadata including:

• Multi-petabyte warehouses
• Ultra-critical applications like financial trading of trillions of dollars daily
• Highly sophisticated and complex business applications like SAP, Oracle Fusion Apps, Salesforce, etc.
• Massive enterprise database consolidations to reduce the cost of fragmented database deployments

DO MUCH MORE, WITH FAR LESS

Administering a mission-critical database is traditionally very expensive because it requires manual
provisioning, securing, monitoring, patching, backing-up, upgrading, recovering, troubleshooting, testing, andtuning of a complex highly available scale-out deployment with disaster recovery protection. The extensiveautomation provided by Autonomous Database dramatically simplifies these tasks, reducing administration costs up to 80%.

Traditional database deployments need to provision for the peak possible workload and add a substantial margin of safety on top of that. But peak workloads tend to occur infrequently, leaving most of this costly capacity idle the majority of the time. Oracle’s Universal Credits subscription model for cloud deployments allows customers to pay for just the resources they use. Autonomous Database allows elastic adjustment of compute and storage resources so that only the required resources are provisioned at any given time, decreasing runtime costs by up to 90%.

Under the Hood of the Smartest Availability Features in Oracle's Autonomous



New application development often suffers from many months of delays waiting for database provisioning, testing, and tuning. With Autonomous Database, new applications don’t wait at all, saving tens of thousands of dollars per application and enabling much faster innovation.

The Autonomous Database subscription includes many management, testing, and security capabilities that previously had to be licensed separately, including:

• Data Encryption
• Diagnostics Pack
• Tuning Pack
• Real Application Testing
• Data Masking, Redaction and Subsetting
• Hybrid Columnar Compression
• Database Vault
• Database In-Memory (subset) – in Autonomous Data Warehouse
• Advanced Analytics (subset) - in Autonomous Data Warehouse

To implement full data management workflows, other clouds use a combination of multiple specialized databases such as a queuing database, OLTP database, JSON data store, reporting database, analytics database, etc. Each database is independently developed and therefore has its own data model, security model, execution model, monitoring model, tuning model, consistency model, query language, analytics, etc. Data needs to be transformed and copied between these specialized
databases. While moving data between specialized databases can make sense for some extreme
high-end applications, it adds enormous unnecessary cost and complexity to the large majority of
applications.

Oracle RAC - Roadmap for New Features



Furthermore, it severely compromises security since protection is limited by the worst
system in the workflow. The Autonomous Database handles all these functions in a single database with no need for complex data movement and provides integrated analytics across all data types.

Behavior Changes, Deprecated and Desupported Features for Oracle Database 18c

Review for information about Oracle Database 18c changes, deprecations, and desupports.

About Deprecated and Desupported Status
In addition to new features, Oracle Database release can modify, deprecate or desupport features, and introduce upgrade behavior changes for your database

Simplified Image-Based Oracle Database Installation
Starting with Oracle Database 18c, installation and configuration of Oracle Database software is simplified with image-based installation.

Initialization Parameter Changes in Oracle Database 18c
Review to see the list of new, deprecated, and desupported initialization parameters in this release.

Deprecated Features in Oracle Database 18c
Review the deprecated features listed in this section to prepare to use alternatives after you upgrade.

Desupported Features in Oracle Database 18c
Review this list of desupported features as part of your upgrade planning.

Terminal Release of Oracle Streams
Oracle Database 18c is the terminal release for Oracle Streams support. Oracle Streams will be desupported from Oracle Database 19c onwards.

Feature Changes for Oracle Database 18c Upgrade Planning
Use these feature changes to help prepare for changes that you include as part of your planning for Oracle Database 18c upgrades.

MAA - Best Practices for the Cloud

Oracle Database 18c – Some important changes

Posted on March 7, 2018 by Mike.Dietrich

I know that Oracle Database 18c is available in the Oracle Cloud and on Exadata engineered systems only right now. But actually I’ve had conversations with some customers who downloaded Oracle 18c on-prem software for Exadata and installed it on their systems. Therefore it may be useful to talk about Oracle Database 18c – Some important changes.

Oracle Database 18c – Some important changes
I will highlight some important changes but of course won’t cover all of them here.

Installation
You may recognize the first change after downloading the image: The installation and configuration of Oracle Database software is simplified with image-based installation. You’ll extract a zip file (db_home.zip) into the directory where you’d like your Oracle installation to be located in – and then call the runInstaller. Be aware: it sits now directly in the $ORACLE_HOME, not in the $ORACLE_HOME/oui subdirectory:

cd $ORACLE_HOME
./runInstaller
runInstaller is a script in this case. The name is kept for persistence.

Melbourne Groundbreakers Tour - Upgrading without risk



Furthermore there are two other new features:

RPM based installation in Oracle 18c
It performs preinstall checks, extracts the database software, reassigns ownership to the preconfigured user and groups, maintains the inventory, and executes all root operations required. Works for a single-instance and client software.

Read-Only Oracle Home in Oracle 18c
In Oracle 18c you can configure read-only homes. In this case all the configuration data and log files reside outside the Oracle home. You can deploy it as a software image across multiple servers.Apart from the traditional ORACLE_BASE and ORACLE_HOME directories, the following directories contain files that used to be in ORACLE_HOME: ORACLE_BASE_HOME and ORACLE_BASE_CONFIG

And interesting thing happens if you call the installer kept for internal purposes in $ORACLE_HOME/oui/bin:
it will start with a different versioning, i.e. as an Oracle 12.2 OUI. The install script runInstaller is in the $ORACLE_HOME directory. And it will greet you with Oracle 18c – and not Oracle 12.2.

Oracle Streams

Yes, the deprecation of Oracle Streams has been announced in the Oracle Database 12.1.0.2 Upgrade Guide a while ago. And Oracle 18c is now the terminal release for Oracle Streams. Beginning with Oracle 19c the feature Oracle Streams won’t be supported anymore. Please note that Oracle Multitenant, regardless of Single- or Multitenant, never implemented Oracle Streams functionality.

Oracle Multimedia

Beginning with Oracle 18c Oracle Multimedia is deprecated now. In case you’d like to remove Oracle Multimedia from your database please see this blog post: Remove Oracle Multimedia. In addition, Multimedia DICOM gets desupported with Oracle 18c as well.

Please note (and thanks Felipe for asking):
The Locator will become a top level component once Oracle Multimedia gets removed and therefore not depend on Multimedia anymore. This will happen in the first release of Oracle where Multimedia does not get installed anymore by default or even removed as part of an upgrade.

Deprecated and Desupported Features  in Oracle Database 18c
Please find the full list of deprecated features in Oracle Database 18c in the Database 18c Upgrade Guide. Furthermore you’ll find a list of desupported features and parameters in Oracle Database 18c within the same book.

The Oracle Autonomous Database #12c #18c




Cool New Features for Developers in 18c and 12c

I normally write an article for each conference presentation I do, but my presentation by this name is a series of live demos, so an article isn't really appropriate. Instead here is a links page to all the articles referenced by the conference presentation of the same name.

This is not supposed to be an exhaustive list of new features, just some that stand out for me, and some that others may not have noticed.

• JSON Data Guide (12.2)
• SQL/JSON (12.2)
• PL/SQL Objects for JSON (12.2)
• Real-Time Materialized Views (12.2)
• Row Limiting Clause (12.1)
• Qualified Expressions (18)
• Polymorphic Table Functions (18)
• Approximate Query Processing (12.1, 12.2, 18)
• Private Temporary Tables (18)
• DBMS_SESSION.SLEEP (18)
• External Table Enhancements (12.2 & 18)
• Case Insensitive Queries (12.2)

Hope this helps. Regards Tim...  ( https://oracle-base.com/articles/misc/cool-new-features-for-developers-in-18c-and-12c )

Faster Insights with Cloudera Enterprise on Oracle Cloud Infrastructure

Qualified Expressions in PL/SQL in Oracle Database 18c

Qualified expressions provide and alternative way to define the value of complex objects, which in some cases can make the code look neater.

Syntax

• Qualified Expressions with Record Types
• Qualified Expressions with Associative Arrays

The basic syntax for a qualified expression is as follows

typemark(aggregate)
The typemark is the type name. The aggregate is the data associated with this instance of the type. The data can specified using positional or the named association syntax. That all sounds a bit complicated, but it's similar to using a constructor for a object and will be obvious once you see some examples.

Qualified Expressions with Record Types
Records with large numbers of columns can be a little clumsy to work with. Qualified expressions can simplify code in some circumstances.

The following example shows three ways to populate a record in Oracle 18c. The first method, available in previous releases, involves a direct assignment to each column in the record variable. The second method uses a qualified expression where the aggregate uses positional notation. The third example uses a qualified expression where the aggregate uses the named association syntax.

DECLARE
  TYPE t_rec IS RECORD (
    id   NUMBER,
    val1 VARCHAR2(10),
    val2 VARCHAR2(10),
    val3 VARCHAR2(10),
    val4 VARCHAR2(10),
    val5 VARCHAR2(10),
    val6 VARCHAR2(10),
    val7 VARCHAR2(10),
    val8 VARCHAR2(10),
    val9 VARCHAR2(10)
  );

  l_rec t_rec;
BEGIN
  -- Pre-18c - Direct assignment to record columns.
  l_rec.id   := 1;
  l_rec.val1 := 'ONE';
  l_rec.val2 := 'TWO';
  l_rec.val3 := 'THREE';
  l_rec.val4 := 'FOUR';
  l_rec.val5 := 'FIVE';
  l_rec.val6 := 'SIX';
  l_rec.val7 := 'SEVEN';
  l_rec.val8 := 'EIGHT';
  l_rec.val9 := 'NINE';

  -- 18c - Qualified expression using position notation.
  l_rec := t_rec(1, 'ONE', 'TWO', 'THREE', 'FOUR', 'FIVE', 'SIX', 'SEVEN', 'EIGHT', 'NINE');

  -- 18c - Qualified expression using named association.
  l_rec := t_rec(id   => 1,
                 val1 => 'ONE',
                 val2 => 'TWO',
                 val3 => 'THREE',
                 val4 => 'FOUR',
                 val5 => 'FIVE',
                 val6 => 'SIX',
                 val7 => 'SEVEN',
                 val8 => 'EIGHT',
                 val9 => 'NINE');
END;
/
The first and last examples show clearly which columns gets which values, but they take a bit more space. The qualified expression using position notation is more compact, but relies on you knowing the order of the columns. In this case it's easy as the type if declared directly above. It would be less obvious if the type were defines in a package specification.

Things look a little different if we are only dealing with a subset of the columns. In the following example the qualified expression using named association looks neater, but still similar to the direct assignment to the record columns.

DECLARE
  TYPE t_rec IS RECORD (
    id   NUMBER,
    val1 VARCHAR2(10),
    val2 VARCHAR2(10),
    val3 VARCHAR2(10),
    val4 VARCHAR2(10),
    val5 VARCHAR2(10),
    val6 VARCHAR2(10),
    val7 VARCHAR2(10),
    val8 VARCHAR2(10),
    val9 VARCHAR2(10)
  );

  l_rec t_rec;
BEGIN
  -- Pre-18c - Direct assignment to record columns.
  l_rec.id   := 1;
  l_rec.val1 := 'ONE';
  l_rec.val9 := 'NINE';

  -- 18c - Qualified expression using position notation.
  l_rec := t_rec(1, 'ONE', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'NINE');

  -- 18c - Qualified expression using named association.
  l_rec := t_rec(id => 1, val1 => 'ONE', val9 => 'NINE');
END;
/
The difference becomes more apparent when the same variable is used for multiple sparse records, each referencing different columns in the record. In the following example the same record variable is used twice for each method. In the first pass the val1 and val9 columns are set. In the second pass the val2 and val8 columns are set. After each assignment the values of the val1 and val9 columns are displayed. The qualified expressions represent a new instance of the record, so all the unused columns are blanked explicitly or implicitly. Without the qualified expression it is up to the developer to blank the previous values manually.

SET SERVEROUTPUT ON
DECLARE
  TYPE t_rec IS RECORD (
    id   NUMBER,
    val1 VARCHAR2(10),
    val2 VARCHAR2(10),
    val3 VARCHAR2(10),
    val4 VARCHAR2(10),
    val5 VARCHAR2(10),
    val6 VARCHAR2(10),
    val7 VARCHAR2(10),
    val8 VARCHAR2(10),
    val9 VARCHAR2(10)
  );

  l_rec t_rec;
BEGIN
  -- Pre-18c - Direct assignment to record columns.
  l_rec.id   := 1;
  l_rec.val1 := 'ONE';
  l_rec.val9 := 'NINE';
  DBMS_OUTPUT.put_line('(1) Record1 val1 = ' || l_rec.val1 || '  val9 = ' || l_rec.val9);

  l_rec.id   := 2;
  l_rec.val2 := 'TWO';
  l_rec.val8 := 'EIGHT';
  DBMS_OUTPUT.put_line('(1) Record2 val1 = ' || l_rec.val1 || '  val9 = ' || l_rec.val9);

  -- 18c - Qualified expression using position notation.
  l_rec := t_rec(1, 'ONE', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'NINE');
  DBMS_OUTPUT.put_line('(2) Record1 val1 = ' || l_rec.val1 || '  val9 = ' || l_rec.val9);

  l_rec := t_rec(1, NULL, 'TWO', NULL, NULL, NULL, NULL, NULL, 'EIGHT', NULL);
  DBMS_OUTPUT.put_line('(2) Record2 val1 = ' || l_rec.val1 || '  val9 = ' || l_rec.val9);

  -- 18c - Qualified expression using named association.
  l_rec := t_rec(id => 1, val1 => 'ONE', val9 => 'NINE');
  DBMS_OUTPUT.put_line('(3) Record1 val1 = ' || l_rec.val1 || '  val9 = ' || l_rec.val9);

  l_rec := t_rec(id => 1, val2 => 'TWO', val8 => 'EIGHT');
  DBMS_OUTPUT.put_line('(3) Record2 val1 = ' || l_rec.val1 || '  val9 = ' || l_rec.val9);
END;
/
(1) Record1 val1 = ONE  val9 = NINE
(1) Record2 val1 = ONE  val9 = NINE
(2) Record1 val1 = ONE  val9 = NINE
(2) Record2 val1 =   val9 =
(3) Record1 val1 = ONE  val9 = NINE
(3) Record2 val1 =   val9 =


PL/SQL procedure successfully completed.

SQL>
We can even use a qualified expression in the definition of a default value. In the following example a procedure accepts a record type as a parameter, which has a default value specified using a qualified expression.

DECLARE
  TYPE t_rec IS RECORD (
    id   NUMBER,
    val1 VARCHAR2(10),
    val2 VARCHAR2(10)
  );

  PROCEDURE dummy (p_rec IN t_rec DEFAULT t_rec(id => 1, val1 => 'ONE')) AS
  BEGIN
    NULL;
  END;
BEGIN
  NULL;
END;
/


Perth APAC Groundbreakers tour - SQL Techniques




Qualified Expressions with Associative Arrays
When dealing with associative arrays we have the option of assigning values to the individual elements of the associative array, or creating a new associative array using a qualified expression. The following example uses a PLS_INTEGER as the index of the associative array.

DECLARE
  TYPE t_tab IS TABLE OF VARCHAR2(10) INDEX BY PLS_INTEGER;

  l_tab t_tab;
BEGIN
  -- Pre-18c - Direct assignment to elements of the collection.
  l_tab(1) := 'ONE';
  l_tab(2) := 'TWO';
  l_tab(3) := 'THREE';

  -- 18c - Qualified expression using named association.
  l_tab := t_tab(1 => 'ONE',
                 2 => 'TWO',
                 3 => 'THREE');
END;
/
This example uses a VARCHAR2 as the index of the associative array.

DECLARE
  TYPE t_tab IS TABLE OF VARCHAR2(10) INDEX BY VARCHAR2(10);

  l_tab t_tab;
BEGIN
  -- Pre-18c - Direct assignment to record columns.
  l_tab('IND1') := 'ONE';
  l_tab('IND2') := 'TWO';
  l_tab('IND3') := 'THREE';

  -- 18c - Qualified expression using named association.
  l_tab := t_tab('IND1' => 'ONE',
                 'IND2' => 'TWO',
                 'IND3' => 'THREE');
END;
/

Remember, the qualified expression creates a new instance of the associative array, so any previously defined elements are removed. In this example we create an associative array with three elements, then immediately assign a two element associative array. If we try to reference the element with index 2 we get a NO_DATA_FOUND exception.

SET SERVEROUTPUT ON
DECLARE
  TYPE t_tab IS TABLE OF VARCHAR2(10) INDEX BY VARCHAR2(10);

  l_tab t_tab;
BEGIN
  -- 18c - Qualified expression using named association.
  l_tab := t_tab(1 => 'ONE',
                 2 => 'TWO',
                 3 => 'THREE');

  l_tab := t_tab(1 => 'ONE',
                 3 => 'THREE');

  DBMS_OUTPUT.put_line('2=' || l_tab(2));
EXCEPTION
  WHEN NO_DATA_FOUND THEN
    DBMS_OUTPUT.put_line('I knew this would cause a NDF error!');
END;
/
I knew this would cause a NDF error!

PL/SQL procedure successfully completed.

SQL>
In the following example a procedure accepts an associative array as a parameter, which has a default value specified using a qualified expression.

DECLARE
  TYPE t_tab IS TABLE OF VARCHAR2(10) INDEX BY VARCHAR2(10);

  PROCEDURE dummy (p_tab IN t_tab DEFAULT t_tab(1 => 'ONE', 2 => 'TWO',3 => 'THREE')) AS
  BEGIN
    NULL;
  END;
BEGIN
  NULL;
END;
/

Sangam 18 - Database Development: Return of the SQL Jedi

The best upcoming features in Oracle Database 19c

By DBA RJ in Events, Oracle Database General

In the Oracle Open World 2018 event that happened in San Francisco last week, from October 22nd to 25th, much has been said about the trends and strategy paths that Oracle is taking in both OCI and in Oracle Database.

Melbourne Groundbreakers Tour - Hints and Tips


As we DBA's are always excited about the upcoming features, I will share below some of the main things that I've spotted on OOW. Please note that this can change, and we don't even have a beta release yet.

1 - Stability
First of all, it was very clear that Oracle's main focus for the 19c database will be stability. This will be the final release for the "12cR2" family. So it was repeated multiple times: "don't expect to see many new features in this release", what in my opinion is really great.

Since 12.1.0.1, Oracle has been implementing a lot of core changes in Oracle Database (like multi-tenancy, unified audit, etc) and it's still very hard nowadays to find a stable 12 release to recommend. 12.1.0.2 is my favorite one, however many bugs are unfixed and it lacks a secure PDB layout (PDB escape techniques are pretty easy to explore). 18c will probably be ignored by all as it was a "transition" release, so I hope that 19c becomes the real stable one, as 11.2.0.4 was for 11g release family. Let's see...

Perth APAC Groundbreakers tour - 18c features



Now comes the real features...
2 - Automatic Indexing
This is indeed the most important and one of the coolest features I've even seen in Oracle DB. Once this kind of automation is implemented and released, it will open doors to many other product automations (like automatic table reorganization, automatic table encryption or anything you can imagine).

The automatic indexing methodology will be based on a common approach to manual SQL Tuning. Oracle will capture the SQL statements, identify the candidate indexes and evaluates the ones that will benefit those statements. The whole process is not something simple.

Basically, Oracle will first create those indexes as unusable and invisible (metadata only). Then, outside the application workflow, oracle will ask the optimizer to test if those candidate indexes improve the SQL performance. In case the performance is better for all statements when indexed is used, it will become visible. If performance is worse, it remains invisible. And if it only performs better for some statements, the index is only marked visible for those SQLs (via SQL Patch, maybe).



The automation will also drop the indexes that become obsoleted by the newly created indexes (logical merge) and also remove the indexes that were created automatically but have not been used in a long time. Everything is customizable. For more details, we need to wait for the Beta Release!

3 - Real-time Stats + Stats Only Queries
With those features, you will be able to turn on the real-time stats for some of your database objects and it will be possible to run some SQLs that will query only the object stats, not doing a single logical read! Cool, isn't it?

4 - Data-guard DML Redirect
When you have a physical standby, opened in read-only mode, and plug in it some type of report tool that needs to create an underlying table or insert some log lines to operate, you have a problem. With this feature, you can define some tables (or maybe schema, it's not clear yet) and you will be able to run DMLs on them. So Oracle will redirect that DML to your primary and reflect the changes on your standby, not impacting those tools. This can be dangerous if not configured properly but will also allow us to do many new things.

5 - Partial JSON Update support
When you update a JSON data column, currently Oracle needs to upload the whole new column value to the database and validates. With this, we will now be able to update just a part (like a tag) of a json data.

6 - Schema-only Oracle accounts
With Oracle 18c, it was introduced the passwordless accounts. This means that you could connect to your schema using only a sort of external authentication like Active Directory. Now oracle has gone further, creating the true concept of Schema Only account (meaning there will be no way to authenticate).



7 - DB REST API
Oracle is trying to make the whole Oracle Database "rest aware", meaning that in a very soon future you will be able to perform ALL kinds of DB operations using REST API (like creating a database, creating a user, granting a privilege or adding a new listener port).

8 - Partitioned Hybrid Tables
Remember in very old times when we didn't have partitioned table, and had to implement partition manually using views + UNION ALL of many tables? Thanks god since 8 (released in 1997) we don't need it. Now Oracle finally did one step further, and you can have a partitioned hybrid table, meaning each partition can be of a different type or source (like one partition is external table and other is a traditional data table).

With Oracle 18 XE limited to 12GB data, this feature will be cool as we will probably be able to offload some of the data externally.

9 - EZConnect Improvements
EZConnect is something very useful to make fast connection calls without requiring a TNS. Problem is that, until now, if you want to use some value pairs likes SDU, RETRY_COUNT, CONNECT_TIMEOUT, this wasn't possible and you would end-up using TNS. Now in 19c you will be able to write something like:

sqlplus soe/soe@//salesserver1:1521/sales.us.example.com?connect_timeout=60&transport_connect_timeout=30&retry_count=3

It will also allow to enable multiples hosts/ports in the connection string (typically used in load-balancing client connections).



10 - Some other cool features
There are many other features that we have to wait for the Beta release to understand better. Below are some of them:

Improvements for count distinct and group by queries
Sharding now supports multiple PDB shards in a CDB
SQL JSON Enhancements
RAT and ADDM at PDB level
Data Dictionary Encryption
Database Vault Operations Control
Web SQLDeveloper

Sangam 18 - The New Optimizer in Oracle 12c




Final thoughts...
As I said, compared to Oracle 12R1, 12R2 or 18c, Oracle reduced a lot the total number of features introduced. That's why I'm excited for 19c, this is the first time in a while that I hear Oracle saying that will invest in stability. Hope this is true.

If you want to test 19c before the others, subscribe in Oracle Beta Program at https://pdpm.oracle.com/.

Sangam 18 - The Groundbreaker Community

More Information:

http://www.oracle.com/us/products/database/autonomous-database-strategy-wp-4124741.pdf

https://go.oracle.com/LP=78880?elqCampaignId=166416&src1=:ow:o:s:feb:AutonomousDB&intcmp=WWMK180723P00010:ow:o:s:feb:AutonomousDB

https://www.oracle.com/database/autonomous-database.html

https://blogs.oracle.com/database/oracle-database-18c-:-now-available-on-the-oracle-cloud-and-oracle-engineered-systems

https://blogs.oracle.com/database/

Oracle Database 18c : Now available on the Oracle Cloud and Oracle Engineered Systems

https://blogs.oracle.com/database/oracle-database-18c-:-now-available-on-the-oracle-cloud-and-oracle-engineered-systems

We’ve covered some of the bigger changes in Oracle Database 18c but there are many more that we don’t have space to cover here. If you want a more comprehensive list take a look at the new features guide here.

https://docs.oracle.com/en/database/oracle/oracle-database/18/newft/new-features.html

You can also find more information on the application development tools here

http://www.oracle.com/technetwork/developer-tools/sql-developer/overview/index.html

http://www.oracle.com/technetwork/developer-tools/rest-data-services/overview/index.html

http://www.oracle.com/technetwork/developer-tools/sqlcl/overview/sqlcl-index-2994757.html

http://www.oracle.com/technetwork/developer-tools/apex/overview/index.html

If you’d like to try out Oracle Database 18c you can do it here with LiveSQL

https://livesql.oracle.com/apex/livesql/file/index.html

For More information on when Oracle Database 18c will be available on other platforms please refer to Oracle Support Document 742060.1

Quantum Information and Computation for Dummies

Quantum computers are devices capable of performing computations using quantum bits, or qubits.

The first thing we need to understand is what a qubit actually is. A “classical computer,” like the one you’re reading this on (either desktop, laptop, tablet, or phone), is also referred to as a “binary computer” because all of the functions it performs are based on either ones or zeros.

D-Wave seminar at Nagoya University: "An Introduction to Quantum Computing"



On a binary computer the processor uses transistors to perform calculations. Each transistor can be on or off, which indicates the one or zero used to compute the next step in a program or algorithm.

There’s more to it than that, but the important thing to know about binary computers is the ones and zeros they use as the basis for computations are called “bits.”



Quantum computers don’t use bits; they use qubits. Qubits, aside from sounding way cooler, have extra functions that bits don’t. Instead of only being represented as a one or zero, qubits can actually be both at the same time. Often qubits, when unobserved, are considered to be “spinning.” Instead of referring to these types of “spin qubits” using ones or zeros, they’re measured in states of “up,” “down,” and “both.”

Superposition

Qubits can be more than one thing at a time because of a strange phenomenon called superposition. Quantum superposition in qubits can be explained by flipping a coin. We know that the coin will land in one of two states: heads or tails. This is how binary computers think. While the coin is still spinning in the air, assuming your eye isn’t quick enough to ‘observe’ the actual state it’s in, the coin is actually in both states at the same time. Essentially until the coin lands it has to be considered both heads and tails simultaneously.

A clever scientist by the name of Schrodinger explained this phenomenon using a cat which he demonstrated as being both alive and dead at the same time.

Quantum Computing: Untangling the Hype

Observation theory

Qubits work on the same principle. An area of related study called “observation theory” dictates that when a quantum particle is being watched it can act like a wave. Basically the universe acts one way when we’re looking, another way when we aren’t. This means quantum computers, using their qubits, can simulate the subatomic particles of the universe in a way that’s actually natural: they speak the same language as an electron or proton, basically.

Different companies are approaching qubits in different ways because, as of right now, working with them is incredibly difficult. Since observing them changes their state, and using them creates noise – the more qubits you have the more errors you get – measuring them is challenging to say the least.

This challenge is exacerbated by the fact that most quantum processors have to be kept at near perfect-zero temperatures (colder than space) and require an amount power that is unsustainably high for the quality of computations. Right now, quantum computers aren’t worth the trouble and money they take to build and operate.

In the future, however, they’ll change our entire understanding of biology, chemistry, and physics. Simulations at the molecular level could be conducted that actually imitate physical concepts in the universe we’ve never been able to reproduce or study.

Automatski - RSA-2048 Cryptography Cracked using Shor's Algorithm on a Quantum Computer

Quantum Supremacy

For quantum computers to become useful to society we’ll have to achieve certain milestones first. The point at which a quantum computer can process information and perform calculations that a binary computer can’t is called quantum supremacy.

Quantum supremacy isn’t all fun and games though, it presents another set of problems. When quantum computers are fully functional even modest systems in the 100 qubit range may be able to bypass binary security like a hot knife through butter.

This is because those qubits, which can be two things at once, figure out multiple solutions to a problem at once. They don’t have to follow binary logic like “if one thing happens do this but if another thing happens do something else.” Individual qubits can do both at the same time while spinning, for example, and then produce the optimum result when properly observed.

Currently there’s a lot of buzz about quantum computers, and rightfully so. Google is pretty sure its new Bristlecone processor will achieve quantum supremacy this year. And it’s hard to bet against Google or one of the other big tech companies. Especially when Intel has already put a quantum processor on a silicon chip and you can access IBM’s in the cloud right now.

No matter your feelings on quantum computers, qubits, or half-dead/half-alive cats, the odds are pretty good that quantum computers will follow the same path that IBM’s mainframes did. They’ll get smaller, faster, more powerful, and eventually we’ll all be using them, even if we don’t understand the science behind them.

Overview

Quantum algorithms are usually described, in the commonly used circuit model of quantum computation, by a quantum circuit which acts on some input qubits and terminates with a measurement. A quantum circuit consists of simple quantum gates which act on at most a fixed number of qubits[why?]. Quantum algorithms may also be stated in other models of quantum computation, such as the Hamiltonian oracle model.[5]

Quantum algorithms can be categorized by the main techniques used by the algorithm. Some commonly used techniques/ideas in quantum algorithms include phase kick-back, phase estimation, the quantum Fourier transform, quantum walks, amplitude amplification and topological quantum field theory. Quantum algorithms may also be grouped by the type of problem solved, for instance see the survey on quantum algorithms for algebraic problems.[6]

Precision atom qubits achieve major quantum computing milestone


Algorithms based on the quantum Fourier transform
The quantum Fourier transform is the quantum analogue of the discrete Fourier transform, and is used in several quantum algorithms. The Hadamard transform is also an example of a quantum Fourier transform over an n-dimensional vector space over the field F2. The quantum Fourier transform can be efficiently implemented on a quantum computer using only a polynomial number of quantum gates.[citation needed]

Deutsch–Jozsa algorithm
Main article: Deutsch–Jozsa algorithm
The Deutsch–Jozsa algorithm solves a black-box problem which probably requires exponentially many queries to the black box for any deterministic classical computer, but can be done with exactly one query by a quantum computer. If we allow both bounded-error quantum and classical algorithms, then there is no speedup since a classical probabilistic algorithm can solve the problem with a constant number of queries with small probability of error. The algorithm determines whether a function f is either constant (0 on all inputs or 1 on all inputs) or balanced (returns 1 for half of the input domain and 0 for the other half).

Simon's algorithm
Main article: Simon's algorithm
Simon's algorithm solves a black-box problem exponentially faster than any classical algorithm, including bounded-error probabilistic algorithms. This algorithm, which achieves an exponential speedup over all classical algorithms that we consider efficient, was the motivation for Shor's factoring algorithm.



Quantum phase estimation algorithm
Main article: Quantum phase estimation algorithm
The quantum phase estimation algorithm is used to determine the eigenphase of an eigenvector of a unitary gate given a quantum state proportional to the eigenvector and access to the gate. The algorithm is frequently used as a subroutine in other algorithms.

Shor's algorithm
Main article: Shor's algorithm
Shor's algorithm solves the discrete logarithm problem and the integer factorization problem in polynomial time,[7] whereas the best known classical algorithms take super-polynomial time. These problems are not known to be in P or NP-complete. It is also one of the few quantum algorithms that solves a non–black-box problem in polynomial time where the best known classical algorithms run in super-polynomial time.

Hidden subgroup problem
The abelian hidden subgroup problem is a generalization of many problems that can be solved by a quantum computer, such as Simon's problem, solving Pell's equation, testing the principal ideal of a ring R and factoring. There are efficient quantum algorithms known for the Abelian hidden subgroup problem.[8] The more general hidden subgroup problem, where the group isn't necessarily abelian, is a generalization of the previously mentioned problems and graph isomorphism and certain lattice problems. Efficient quantum algorithms are known for certain non-abelian groups. However, no efficient algorithms are known for the symmetric group, which would give an efficient algorithm for graph isomorphism[9] and the dihedral group, which would solve certain lattice problems.[10]



Boson sampling problem
Main article: Boson sampling
The Boson Sampling Problem in an experimental configuration assumes[11] an input of bosons (ex. photons of light) of moderate number getting randomly scattered into a large number of output modes constrained by a defined unitarity. The problem is then to produce a fair sample of the probability distribution of the output which is dependent on the input arrangement of bosons and the Unitarity.[12] Solving this problem with a classical computer algorithm requires computing the permanent[clarification needed] of the unitary transform matrix, which may be either impossible or take a prohibitively long time. In 2014, it was proposed[13] that existing technology and standard probabilistic methods of generating single photon states could be used as input into a suitable quantum computable linear optical network and that sampling of the output probability distribution would be demonstrably superior using quantum algorithms. In 2015, investigation predicted[14] the sampling problem had similar complexity for inputs other than Fock state photons and identified a transition in computational complexity from classically simulatable to just as hard as the Boson Sampling Problem, dependent on the size of coherent amplitude inputs.

Estimating Gauss sums
A Gauss sum is a type of exponential sum. The best known classical algorithm for estimating these sums takes exponential time. Since the discrete logarithm problem reduces to Gauss sum estimation, an efficient classical algorithm for estimating Gauss sums would imply an efficient classical algorithm for computing discrete logarithms, which is considered unlikely. However, quantum computers can estimate Gauss sums to polynomial precision in polynomial time.[15]

Fourier fishing and Fourier checking
We have an oracle consisting of n random Boolean functions mapping n-bit strings to a Boolean value. We are required to find n n-bit strings z1,..., zn such that for the Hadamard-Fourier transform, at least 3/4 of the strings satisfy

{\displaystyle \left|{\tilde {f}}\left(z_{i}\right)\right|\geqslant 1} \left|{\tilde  {f}}\left(z_{i}\right)\right|\geqslant 1
and at least 1/4 satisfies

{\displaystyle \left|{\tilde {f}}\left(z_{i}\right)\right|\geqslant 2} \left|{\tilde  {f}}\left(z_{i}\right)\right|\geqslant 2.
This can be done in Bounded-error Quantum Polynomial time (BQP).[16]

Bob Sutor demonstrates the IBM Q quantum computer


Algorithms based on amplitude amplification
Amplitude amplification is a technique that allows the amplification of a chosen subspace of a quantum state. Applications of amplitude amplification usually lead to quadratic speedups over the corresponding classical algorithms. It can be considered to be a generalization of Grover's algorithm.

Grover's algorithm
Main article: Grover's algorithm
Grover's algorithm searches an unstructured database (or an unordered list) with N entries, for a marked entry, using only {\displaystyle O({\sqrt {N}})} O({\sqrt  {N}}) queries instead of the {\displaystyle O({N})} {\displaystyle O({N})} queries required classically.[17] Classically, {\displaystyle O({N})} {\displaystyle O({N})} queries are required, even if we allow bounded-error probabilistic algorithms.

Bohmian Mechanics is a non-local hidden variable interpretation of quantum mechanics. It has been shown that a non-local hidden variable quantum computer could implement a search of an N-item database at most in {\displaystyle O({\sqrt[{3}]{N}})} {\displaystyle O({\sqrt[{3}]{N}})} steps. This is slightly faster than the {\displaystyle O({\sqrt {N}})} O({\sqrt  {N}}) steps taken by Grover's algorithm. Neither search method will allow quantum computers to solve NP-Complete problems in polynomial time.[18]

Quantum counting
Quantum counting solves a generalization of the search problem. It solves the problem of counting the number of marked entries in an unordered list, instead of just detecting if one exists. Specifically, it counts the number of marked entries in an {\displaystyle N} N-element list, with error {\displaystyle \epsilon } \epsilon  making only {\displaystyle \Theta \left({\frac {1}{\epsilon }}{\sqrt {\frac {N}{k}}}\right)} \Theta \left({\frac  {1}{\epsilon }}{\sqrt  {{\frac  {N}{k}}}}\right) queries, where {\displaystyle k} k is the number of marked elements in the list.[19][20] More precisely, the algorithm outputs an estimate {\displaystyle k'} k' for {\displaystyle k} k, the number of marked entries, with the following accuracy: {\displaystyle |k-k'|\leq \epsilon k} |k-k'|\leq \epsilon k.

Solving a linear systems of equations
Main article: Quantum algorithm for linear systems of equations
In 2009 Aram Harrow, Avinatan Hassidim, and Seth Lloyd, formulated a quantum algorithm for solving linear systems. The algorithm estimates the result of a scalar measurement on the solution vector to a given linear system of equations.[21]



Provided the linear system is a sparse and has a low condition number {\displaystyle \kappa } \kappa , and that the user is interested in the result of a scalar measurement on the solution vector, instead of the values of the solution vector itself, then the algorithm has a runtime of {\displaystyle O(\log(N)\kappa ^{2})} O(\log(N)\kappa ^{2}), where {\displaystyle N} N is the number of variables in the linear system. This offers an exponential speedup over the fastest classical algorithm, which runs in {\displaystyle O(N\kappa )} O(N\kappa ) (or {\displaystyle O(N{\sqrt {\kappa }})} O(N{\sqrt {\kappa }}) for positive semidefinite matrices).

Algorithms based on quantum walks
Main article: Quantum walk
A quantum walk is the quantum analogue of a classical random walk, which can be described by a probability distribution over some states. A quantum walk can be described by a quantum superposition over states. Quantum walks are known to give exponential speedups for some black-box problems.[22][23] They also provide polynomial speedups for many problems. A framework for the creation of quantum walk algorithms exists and is quite a versatile tool.[24]

Element distinctness problem
Main article: Element distinctness problem
The element distinctness problem is the problem of determining whether all the elements of a list are distinct. Classically, Ω(N) queries are required for a list of size N, since this problem is harder than the search problem which requires Ω(N) queries. However, it can be solved in {\displaystyle \Theta (N^{2/3})} \Theta (N^{{2/3}}) queries on a quantum computer. The optimal algorithm is by Andris Ambainis.[25] Yaoyun Shi first proved a tight lower bound when the size of the range is sufficiently large.[26] Ambainis[27] and Kutin[28] independently (and via different proofs) extended his work to obtain the lower bound for all functions.

Triangle-finding problem
Main article: Triangle finding problem
The triangle-finding problem is the problem of determining whether a given graph contains a triangle (a clique of size 3). The best-known lower bound for quantum algorithms is Ω(N), but the best algorithm known requires O(N1.297) queries,[29] an improvement over the previous best O(N1.3) queries.[24][30]

Quantum Algorithms for Evaluating MIN-MAX Trees



Formula evaluation
A formula is a tree with a gate at each internal node and an input bit at each leaf node. The problem is to evaluate the formula, which is the output of the root node, given oracle access to the input.

A well studied formula is the balanced binary tree with only NAND gates.[31] This type of formula requires Θ(Nc) queries using randomness,[32] where {\displaystyle c=\log _{2}(1+{\sqrt {33}})/4\approx 0.754} c=\log _{2}(1+{\sqrt  {33}})/4\approx 0.754. With a quantum algorithm however, it can be solved in Θ(N0.5) queries. No better quantum algorithm for this case was known until one was found for the unconventional Hamiltonian oracle model.[5] The same result for the standard setting soon followed.[33]

Quantum Computing deep dive



Fast quantum algorithms for more complicated formulas are also known.[34]

Group commutativity
The problem is to determine if a black box group, given by k generators, is commutative. A black box group is a group with an oracle function, which must be used to perform the group operations (multiplication, inversion, and comparison with identity). We are interested in the query complexity, which is the number of oracle calls needed to solve the problem. The deterministic and randomized query complexities are {\displaystyle \Theta (k^{2})} \Theta (k^{2}) and {\displaystyle \Theta (k)} \Theta (k) respectively.[35] A quantum algorithm requires {\displaystyle \Omega (k^{2/3})} \Omega (k^{{2/3}}) queries but the best known algorithm uses {\displaystyle O(k^{2/3}\log k)} O(k^{{2/3}}\log k) queries.[36]

BQP-complete problems
Computing knot invariants
Witten had shown that the Chern-Simons topological quantum field theory (TQFT) can be solved in terms of Jones polynomials. A quantum computer can simulate a TQFT, and thereby approximate the Jones polynomial,[37] which as far as we know, is hard to compute classically in the worst-case scenario.[citation needed]



Quantum simulation
The idea that quantum computers might be more powerful than classical computers originated in Richard Feynman's observation that classical computers seem to require exponential time to simulate many-particle quantum systems.[38] Since then, the idea that quantum computers can simulate quantum physical processes exponentially faster than classical computers has been greatly fleshed out and elaborated. Efficient (that is, polynomial-time) quantum algorithms have been developed for simulating both Bosonic and Fermionic systems[39] and in particular, the simulation of chemical reactions beyond the capabilities of current classical supercomputers requires only a few hundred qubits.[40] Quantum computers can also efficiently simulate topological quantum field theories.[41] In addition to its intrinsic interest, this result has led to efficient quantum algorithms for estimating quantum topological invariants such as Jones[42] and HOMFLY polynomials,[43] and the Turaev-Viro invariant of three-dimensional manifolds.[44]

Hybrid quantum/classical algorithms
Hybrid Quantum/Classical Algorithms combine quantum state preparation and measurement with classical optimization.[45] These algorithms generally aim to determine the ground state eigenvector and eigenvalue of a Hermitian Operator.

QAOA
The quantum approximate optimization algorithm is a toy model of quantum annealing which can be used to solve problems in graph theory.[46] The algorithm makes use of classical optimization of quantum operations to maximize an objective function.

Variational Quantum Eigensolver
The VQE algorithm applies classical optimization to minimize the energy expectation of an ansatz state to find the ground state energy of a molecule.[47] This can also be extended to find excited energies of molecules.[48]



BIG THINGS HAPPEN when computers get smaller. Or faster. And quantum computing is about chasing perhaps the biggest performance boost in the history of technology. The basic idea is to smash some barriers that limit the speed of existing computers by harnessing the counterintuitive physics of subatomic scales.

Quantum Computing for Computer Scientists


If the tech industry pulls off that, ahem, quantum leap, you won’t be getting a quantum computer for your pocket. Don’t start saving for an iPhone Q. We could, however, see significant improvements in many areas of science and technology, such as longer-lasting batteries for electric cars or advances in chemistry that reshape industries or enable new medical treatments. Quantum computers won’t be able to do everything faster than conventional computers, but on some tricky problems they have advantages that would enable astounding progress.

It’s not productive (or polite) to ask people working on quantum computing when exactly those dreamy applications will become real. The only thing for sure is that they are still many years away. Prototype quantum computing hardware is still embryonic. But powerful—and, for tech companies, profit-increasing—computers powered by quantum physics have recently started to feel less hypothetical.

The Mathematics of Quantum Computers | Infinite Series

That’s because Google, IBM, and others have decided it’s time to invest heavily in the technology, which, in turn, has helped quantum computing earn a bullet point on the corporate strategy PowerPoint slides of big companies in areas such as finance, like JPMorgan, and aerospace, like Airbus. In 2017, venture investors plowed $241 million into startups working on quantum computing hardware or software worldwide, according to CB Insights. That’s triple the amount in the previous year.

Like the befuddling math underpinning quantum computing, some of the expectations building around this still-impractical technology can make you lightheaded. If you squint out the window of a flight into SFO right now, you can see a haze of quantum hype drifting over Silicon Valley. But the enormous potential of quantum computing is undeniable, and the hardware needed to harness it is advancing fast. If there were ever a perfect time to bend your brain around quantum computing, it’s now. Say “Schrodinger’s superposition” three times fast, and we can dive in.



The History of Quantum Computing Explained
The prehistory of quantum computing begins early in the 20th century, when physicists began to sense they had lost their grip on reality.

First, accepted explanations of the subatomic world turned out to be incomplete. Electrons and other particles didn’t just neatly carom around like Newtonian billiard balls, for example. Sometimes they acted like waves instead. Quantum mechanics emerged to explain such quirks, but introduced troubling questions of its own. To take just one brow-wrinkling example, this new math implied that physical properties of the subatomic world, like the position of an electron, didn’t really exist until they were observed.



If you find that baffling, you’re in good company. A year before winning a Nobel for his contributions to quantum theory, Caltech’s Richard Feynman remarked that “nobody understands quantum mechanics.” The way we experience the world just isn’t compatible. But some people grasped it well enough to redefine our understanding of the universe. And in the 1980s a few of them—including Feynman—began to wonder if quantum phenomena like subatomic particles'“don’t look and I don’t exist” trick could be used to process information. The basic theory or blueprint for quantum computers that took shape in the 80s and 90s still guides Google and others working on the technology.

Mathematics for Machine Learning full Course || Linear Algebra || Part-1


Before we belly flop into the murky shallows of quantum computing 0.101, we should refresh our understanding of regular old computers. As you know, smartwatches, iPhones, and the world’s fastest supercomputer all basically do the same thing: they perform calculations by encoding information as digital bits, aka 0s and 1s. A computer might flip the voltage in a circuit on and off to represent 1s and 0s for example.

Quantum computers do calculations using bits, too. After all, we want them to plug into our existing data and computers. But quantum bits, or qubits, have unique and powerful properties that allow a group of them to do much more than an equivalent number of conventional bits.

Qubits can be built in various ways, but they all represent digital 0s and 1s using the quantum properties of something that can be controlled electronically. Popular examples—at least among a very select slice of humanity—include superconducting circuits, or individual atoms levitated inside electromagnetic fields. The magic power of quantum computing is that this arrangement lets qubits do more than just flip between 0 and 1. Treat them right and they can flip into a mysterious extra mode called a superposition.

QUANTUM LEAPS

1980
Physicist Paul Benioff suggests quantum mechanics could be used for computation.

1981
Nobel-winning physicist Richard Feynman, at Caltech, coins the term quantum computer.

1985
Physicist David Deutsch, at Oxford, maps out how a quantum computer would operate, a blueprint that underpins the nascent industry of today.

1994
Mathematician Peter Shor, at Bell Labs, writes an algorithm that could tap a quantum computer’s power to break widely used forms of encryption.

2007
D-Wave, a Canadian startup, announces a quantum computing chip it says can solve Sudoku puzzles, triggering years of debate over whether the company’s technology really works.

2013
Google teams up with NASA to fund a lab to try out D-Wave’s hardware.

2014
Google hires the professor behind some of the best quantum computer hardware yet to lead its new quantum hardware lab.

2016
IBM puts some of its prototype quantum processors on the internet for anyone to experiment with, saying programmers need to get ready to write quantum code.

2017
Startup Rigetti opens its own quantum computer fabrication facility to build prototype hardware and compete with Google and IBM.

You may have heard that a qubit in superposition is both 0 and 1 at the same time. That’s not quite true and also not quite false—there’s just no equivalent in Homo sapiens’ humdrum classical reality. If you have a yearning to truly grok it, you must make a mathematical odyssey WIRED cannot equip you for. But in the simplified and dare we say perfect world of this explainer, the important thing to know is that the math of a superposition describes the probability of discovering either a 0 or 1 when a qubit is read out—an operation that crashes it out of a quantum superposition into classical reality. A quantum computer can use a collection of qubits in superpositions to play with different possible paths through a calculation. If done correctly, the pointers to incorrect paths cancel out, leaving the correct answer when the qubits are read out as 0s and 1s.



For some problems that are very time consuming for conventional computers, this allows a quantum computer to find a solution in far fewer steps than a conventional computer would need. Grover’s algorithm, a famous quantum search algorithm, could find you in a phone book with 100 million names with just 10,000 operations. If a classical search algorithm just spooled through all the listings to find you, it would require 50 million operations, on average. For Grover’s and some other quantum algorithms, the bigger the initial problem—or phonebook—the further behind a conventional computer is left in the digital dust.

Leading the Evolution of Compute: Quantum Computing


The reason we don’t have useful quantum computers today is that qubits are extremely finicky. The quantum effects they must control are very delicate, and stray heat or noise can flip 0s and 1s, or wipe out a crucial superposition. Qubits have to be carefully shielded, and operated at very cold temperatures, sometimes only fractions of a degree above absolute zero. Most plans for quantum computing depend on using a sizable chunk of a quantum processor’s power to correct its own errors, caused by misfiring qubits.

Recent excitement about quantum computing stems from progress in making qubits less flaky. That’s giving researchers the confidence to start bundling the devices into larger groups. Startup Rigetti Computing recently announced it has built a processor with 128 qubits made with aluminum circuits that are super-cooled to make them superconducting. Google and IBM have announced their own chips with 72 and 50 qubits, respectively. That’s still far fewer than would be needed to do useful work with a quantum computer—it would probably require at least thousands—but as recently as 2016 those companies’ best chips had qubits only in the single digits. After tantalizing computer scientists for 30 years, practical quantum computing may not exactly be close, but it has begun to feel a lot closer.


What the Future Holds for Quantum Computing
Some large companies and governments have started treating quantum computing research like a race—perhaps fittingly it’s one where both the distance to the finish line and the prize for getting there are unknown.

Google, IBM, Intel, and Microsoft have all expanded their teams working on the technology, with a growing swarm of startups such as Rigetti in hot pursuit. China and the European Union have each launched new programs measured in the billions of dollars to stimulate quantum R&D. And in the US, the Trump White House has created a new committee to coordinate government work on quantum information science. Several bills were introduced to Congress in 2018 proposing new funding for quantum research, totalling upwards of $1.3 billion. It’s not quite clear what the first killer apps of quantum computing will be, or when they will appear. But there’s a sense that whoever is first make these machines useful will gain big economic and national security advantages.

JARGON FOR THE QUANTUM QURIOUS

What's a qubit?
A device that uses quantum mechanical effects to represent 0s and 1s of digital data, similar to the bits in a conventional computer.

What's a superposition?
It's the trick that makes quantum computers tick, and makes qubits more powerful than ordinary bits. A superposition is in an intuition-defying mathematical combination of both 0 and 1. Quantum algorithms can use a group of qubits in a superposition to shortcut through calculations.

What's quantum entanglement?
A quantum effect so unintuitive that Einstein dubbed it “spooky action at a distance.” When two qubits in a superposition are entangled, certain operations on one have instant effects on the other, a process that helps quantum algorithms be more powerful than conventional ones.

What's quantum speedup?
The holy grail of quantum computing—a measure of how much faster a quantum computer could crack a problem than a conventional computer could. Quantum computers aren’t well-suited to all kinds of problems, but for some they offer an exponential speedup, meaning their advantage over a conventional computer grows explosively with the size of the input problem.

Back in the world of right now, though, quantum processors are too simple to do practical work. Google is working to stage a demonstration known as quantum supremacy, in which a quantum processor would solve a carefully designed math problem beyond existing supercomputers. But that would be an historic scientific milestone, not proof quantum computing is ready to do real work.

As quantum computer prototypes get larger, the first practical use for them will probably be for chemistry simulations. Computer models of molecules and atoms are vital to the hunt for new drugs or materials. Yet conventional computers can’t accurately simulate the behavior of atoms and electrons during chemical reactions. Why? Because that behavior is driven by quantum mechanics, the full complexity of which is too great for conventional machines. Daimler and Volkswagen have both started investigating quantum computing as a way to improve battery chemistry for electric vehicles. Microsoft says other uses could include designing new catalysts to make industrial processes less energy intensive, or even to pull carbon dioxide out of the atmosphere to mitigate climate change.

Quantum computers would also be a natural fit for code-breaking. We’ve known since the 90s that they could zip through the math underpinning the encryption that secures online banking, flirting, and shopping. Quantum processors would need to be much more advanced to do this, but governments and companies are taking the threat seriously. The National Institute of Standards and Technology is in the process of evaluating new encryption systems that could be rolled out to quantum-proof the internet.

Special Purpose Quantum Annealing Quantum Computer v1.0



Tech companies such as Google are also betting that quantum computers can make artificial intelligence more powerful. That’s further in the future and less well mapped out than chemistry or code-breaking applications, but researchers argue they can figure out the details down the line as they play around with larger and larger quantum processors. One hope is that quantum computers could help machine-learning algorithms pick up complex tasks using many fewer than the millions of examples typically used to train AI systems today.

Despite all the superposition-like uncertainty about when the quantum computing era will really begin, big tech companies argue that programmers need to get ready now. Google, IBM, and Microsoft have all released open source tools to help coders familiarize themselves with writing programs for quantum hardware. IBM has even begun to offer online access to some of its quantum processors, so anyone can experiment with them. Long term, the big computing companies see themselves making money by charging corporations to access data centers packed with supercooled quantum processors.

What’s in it for the rest of us? Despite some definite drawbacks, the age of conventional computers has helped make life safer, richer, and more convenient—many of us are never more than five seconds away from a kitten video. The era of quantum computers should have similarly broad reaching, beneficial, and impossible to predict consequences. Bring on the qubits.

More Information:

http://quantumalgorithmzoo.org

https://thierry-breton.com/en/the-quantum-challenge/

https://www.digitaltrends.com/computing/microsoft-provides-free-quantum-computing-lessons/

https://www.wired.com/story/wired-guide-to-quantum-computing/

https://www.forbes.com/sites/chadorzel/2017/04/17/what-sorts-of-problems-are-quantum-computers-good-for/

https://cloudblogs.microsoft.com/quantum/2018/06/06/the-microsoft-approach-to-quantum-computing/

https://blogs.microsoft.com/ai/microsoft-doubles-quantum-computing-bet/

https://thenextweb.com/microsoft/2018/07/24/microsoft-debuts-free-quantum-computer-programming-katas/

https://thenextweb.com/artificial-intelligence/2018/03/15/understanding-quantum-computers-the-basics/

Azure Big Data Architecture

Building the Data Lake with Azure Data Factory and Data Lake Analytics


The cloud is changing the way applications are designed, including how data is processed and stored. Instead of a single general-purpose database that handles all of a solution's data, polyglot persistence solutions use multiple, specialized data stores, each optimized to provide specific capabilities. The perspective on data in the solution changes as a result. There are no longer multiple layers of business logic that read and write to a single data layer. Instead, solutions are designed around a data pipeline that describes how data flows through a solution, where it is processed, where it is stored, and how it is consumed by the next component in the pipeline.

Control and protect your data through privileged access management capabilities

Traditional RDBMS workloads. These workloads include online transaction processing (OLTP) and online analytical processing (OLAP). Data in OLTP systems is typically relational data with a pre-defined schema and a set of constraints to maintain referential integrity. Often, data from multiple sources in the organization may be consolidated into a data warehouse, using an ETL process to move and transform the source data.

Big data architectures and the data lake

Big data solutions. A big data architecture is designed to handle the ingestion, processing, and analysis of data that is too large or complex for traditional database systems. The data may be processed in batch or in real time. Big data solutions typically involve a large amount of non-relational data, such as key-value data, JSON documents, or time series data. Often traditional RDBMS systems are not well-suited to store this type of data. The term NoSQL refers to a family of databases designed to hold non-relational data. (The term isn't quite accurate, because many non-relational data stores support SQL compatible queries.)

Azure SQL Database Managed Instance

These two categories are not mutually exclusive, and there is overlap between them, but we feel that it's a useful way to frame the discussion. Within each category, the guide discusses common scenarios, including relevant Azure services and the appropriate architecture for the scenario. In addition, the guide compares technology choices for data solutions in Azure, including open source options. Within each category, we describe the key selection criteria and a capability matrix, to help you choose the right technology for your scenario.

Differentiate Big Data vs Data Warehouse use cases for a cloud solution

A big data architecture is designed to handle the ingestion, processing, and analysis of data that is too large or complex for traditional database systems. The threshold at which organizations enter into the big data realm differs, depending on the capabilities of the users and their tools. For some, it can mean hundreds of gigabytes of data, while for others it means hundreds of terabytes. As tools for working with big data sets advance, so does the meaning of big data. More and more, this term relates to the value you can extract from your data sets through advanced analytics, rather than strictly the size of the data, although in these cases they tend to be quite large.

Building a modern data warehouse

Over the years, the data landscape has changed. What you can do, or are expected to do, with data has changed. The cost of storage has fallen dramatically, while the means by which data is collected keeps growing. Some data arrives at a rapid pace, constantly demanding to be collected and observed. Other data arrives more slowly, but in very large chunks, often in the form of decades of historical data. You might be facing an advanced analytics problem, or one that requires machine learning. These are challenges that big data architectures seek to solve.

Big data solutions typically involve one or more of the following types of workload:

• Batch processing of big data sources at rest.
• Real-time processing of big data in motion.
• Interactive exploration of big data.
• Predictive analytics and machine learning.

Consider big data architectures when you need to:

• Store and process data in volumes too large for a traditional database.
• Transform unstructured data for analysis and reporting.
• Capture, process, and analyze unbounded streams of data in real time, or with low latency.

Components of a big data architecture

The following diagram shows the logical components that fit into a big data architecture. Individual solutions may not contain every item in this diagram.

Most big data architectures include some or all of the following components:

Data sources. All big data solutions start with one or more data sources. Examples include:

Application data stores, such as relational databases.
Static files produced by applications, such as web server log files.
Real-time data sources, such as IoT devices.
Data storage. Data for batch processing operations is typically stored in a distributed file store that can hold high volumes of large files in various formats. This kind of store is often called a data lake. Options for implementing this storage include Azure Data Lake Store or blob containers in Azure Storage.

Batch processing. Because the data sets are so large, often a big data solution must process data files using long-running batch jobs to filter, aggregate, and otherwise prepare the data for analysis. Usually these jobs involve reading source files, processing them, and writing the output to new files. Options include running U-SQL jobs in Azure Data Lake Analytics, using Hive, Pig, or custom Map/Reduce jobs in an HDInsight Hadoop cluster, or using Java, Scala, or Python programs in an HDInsight Spark cluster.

Build hybrid data platform with Azure SQL Database and SQL Server


Real-time message ingestion. If the solution includes real-time sources, the architecture must include a way to capture and store real-time messages for stream processing. This might be a simple data store, where incoming messages are dropped into a folder for processing. However, many solutions need a message ingestion store to act as a buffer for messages, and to support scale-out processing, reliable delivery, and other message queuing semantics. This portion of a streaming architecture is often referred to as stream buffering. Options include Azure Event Hubs, Azure IoT Hub, and Kafka.

Stream processing. After capturing real-time messages, the solution must process them by filtering, aggregating, and otherwise preparing the data for analysis. The processed stream data is then written to an output sink. Azure Stream Analytics provides a managed stream processing service based on perpetually running SQL queries that operate on unbounded streams. You can also use open source Apache streaming technologies like Storm and Spark Streaming in an HDInsight cluster.

Analytical data store. Many big data solutions prepare data for analysis and then serve the processed data in a structured format that can be queried using analytical tools. The analytical data store used to serve these queries can be a Kimball-style relational data warehouse, as seen in most traditional business intelligence (BI) solutions. Alternatively, the data could be presented through a low-latency NoSQL technology such as HBase, or an interactive Hive database that provides a metadata abstraction over data files in the distributed data store. Azure SQL Data Warehouse provides a managed service for large-scale, cloud-based data warehousing. HDInsight supports Interactive Hive, HBase, and Spark SQL, which can also be used to serve data for analysis.

Microservices architecture with .NET Core Docker containers and Azure 

Analysis and reporting. The goal of most big data solutions is to provide insights into the data through analysis and reporting. To empower users to analyze the data, the architecture may include a data modeling layer, such as a multidimensional OLAP cube or tabular data model in Azure Analysis Services. It might also support self-service BI, using the modeling and visualization technologies in Microsoft Power BI or Microsoft Excel. Analysis and reporting can also take the form of interactive data exploration by data scientists or data analysts. For these scenarios, many Azure services support analytical notebooks, such as Jupyter, enabling these users to leverage their existing skills with Python or R. For large-scale data exploration, you can use Microsoft R Server, either standalone or with Spark.

Azure IoT Edge with examples | Laurent Ellerbach | Microsoft


Orchestration. Most big data solutions consist of repeated data processing operations, encapsulated in workflows, that transform source data, move data between multiple sources and sinks, load the processed data into an analytical data store, or push the results straight to a report or dashboard. To automate these workflows, you can use an orchestration technology such Azure Data Factory or Apache Oozie and Sqoop.

Lambda architecture

When working with very large data sets, it can take a long time to run the sort of queries that clients need. These queries can't be performed in real time, and often require algorithms such as MapReduce that operate in parallel across the entire data set. The results are then stored separately from the raw data and used for querying.

One drawback to this approach is that it introduces latency — if processing takes a few hours, a query may return results that are several hours old. Ideally, you would like to get some results in real time (perhaps with some loss of accuracy), and combine these results with the results from the batch analytics.

The lambda architecture, first proposed by Nathan Marz, addresses this problem by creating two paths for data flow. All data coming into the system goes through these two paths:

A batch layer (cold path) stores all of the incoming data in its raw form and performs batch processing on the data. The result of this processing is stored as a batch view.

A speed layer (hot path) analyzes data in real time. This layer is designed for low latency, at the expense of accuracy.

The batch layer feeds into a serving layer that indexes the batch view for efficient querying. The speed layer updates the serving layer with incremental updates based on the most recent data.



Data that flows into the hot path is constrained by latency requirements imposed by the speed layer, so that it can be processed as quickly as possible. Often, this requires a tradeoff of some level of accuracy in favor of data that is ready as quickly as possible. For example, consider an IoT scenario where a large number of temperature sensors are sending telemetry data. The speed layer may be used to process a sliding time window of the incoming data.

Data flowing into the cold path, on the other hand, is not subject to the same low latency requirements. This allows for high accuracy computation across large data sets, which can be very time intensive.

Building resilient microservices with .NET Core and Azure Kubernetes Service


Eventually, the hot and cold paths converge at the analytics client application. If the client needs to display timely, yet potentially less accurate data in real time, it will acquire its result from the hot path. Otherwise, it will select results from the cold path to display less timely but more accurate data. In other words, the hot path has data for a relatively small window of time, after which the results can be updated with more accurate data from the cold path.

The raw data stored at the batch layer is immutable. Incoming data is always appended to the existing data, and the previous data is never overwritten. Any changes to the value of a particular datum are stored as a new timestamped event record. This allows for recomputation at any point in time across the history of the data collected. The ability to recompute the batch view from the original raw data is important, because it allows for new views to be created as the system evolves.

Kappa architecture

A drawback to the lambda architecture is its complexity. Processing logic appears in two different places — the cold and hot paths — using different frameworks. This leads to duplicate computation logic and the complexity of managing the architecture for both paths.

The kappa architecture was proposed by Jay Kreps as an alternative to the lambda architecture. It has the same basic goals as the lambda architecture, but with an important distinction: All data flows through a single path, using a stream processing system.



There are some similarities to the lambda architecture's batch layer, in that the event data is immutable and all of it is collected, instead of a subset. The data is ingested as a stream of events into a distributed and fault tolerant unified log. These events are ordered, and the current state of an event is changed only by a new event being appended. Similar to a lambda architecture's speed layer, all event processing is performed on the input stream and persisted as a real-time view.

If you need to recompute the entire data set (equivalent to what the batch layer does in lambda), you simply replay the stream, typically using parallelism to complete the computation in a timely fashion.

Internet of Things (IoT)

From a practical viewpoint, Internet of Things (IoT) represents any device that is connected to the Internet. This includes your PC, mobile phone, smart watch, smart thermostat, smart refrigerator, connected automobile, heart monitoring implants, and anything else that connects to the Internet and sends or receives data. The number of connected devices grows every day, as does the amount of data collected from them. Often this data is being collected in highly constrained, sometimes high-latency environments. In other cases, data is sent from low-latency environments by thousands or millions of devices, requiring the ability to rapidly ingest the data and process accordingly. Therefore, proper planning is required to handle these constraints and unique requirements.

AI for an intelligent cloud and intelligent edge: Discover, deploy, and manage with Azure ML services

Event-driven architectures are central to IoT solutions. The following diagram shows a possible logical architecture for IoT. The diagram emphasizes the event-streaming components of the architecture.




The cloud gateway ingests device events at the cloud boundary, using a reliable, low latency messaging system.

Devices might send events directly to the cloud gateway, or through a field gateway. A field gateway is a specialized device or software, usually collocated with the devices, that receives events and forwards them to the cloud gateway. The field gateway might also preprocess the raw device events, performing functions such as filtering, aggregation, or protocol transformation.

After ingestion, events go through one or more stream processors that can route the data (for example, to storage) or perform analytics and other processing.

The following are some common types of processing. (This list is certainly not exhaustive.)

Writing event data to cold storage, for archiving or batch analytics.

Hot path analytics, analyzing the event stream in (near) real time, to detect anomalies, recognize patterns over rolling time windows, or trigger alerts when a specific condition occurs in the stream.

Handling special types of nontelemetry messages from devices, such as notifications and alarms.

Machine learning.

The boxes that are shaded gray show components of an IoT system that are not directly related to event streaming, but are included here for completeness.

The device registry is a database of the provisioned devices, including the device IDs and usually device metadata, such as location.

The provisioning API is a common external interface for provisioning and registering new devices.

Some IoT solutions allow command and control messages to be sent to devices.

Relevant Azure services:

• Azure IoT Hub
• Azure Event Hubs
• Azure Stream Analytics

Learn more about IoT on Azure by reading the Azure IoT reference architecture https://azure.microsoft.com/updates/microsoft-azure-iot-reference-architecture-available/.

Microsoft and Hortonworks Delivers the Modern Data Architecture for Big Data


Advanced analytics goes beyond the historical reporting and data aggregation of traditional business intelligence (BI), and uses mathematical, probabilistic, and statistical modeling techniques to enable predictive processing and automated decision making.

Advanced analytics solutions typically involve the following workloads:

• Interactive data exploration and visualization
• Machine Learning model training
• Real-time or batch predictive processing

Choosing technologies for a big data solution in the cloud

Most advanced analytics architectures include some or all of the following components:

Data storage. Advanced analytics solutions require data to train machine learning models. Data scientists typically need to explore the data to identify its predictive features and the statistical relationships between them and the values they predict (known as a label). The predicted label can be a quantitative value, like the financial value of something in the future or the duration of a flight delay in minutes. Or it might represent a categorical class, like "true" or "false,""flight delay" or "no flight delay," or categories like "low risk,""medium risk," or "high risk."

Batch processing. To train a machine learning model, you typically need to process a large volume of training data. Training the model can take some time (on the order of minutes to hours). This training can be performed using scripts written in languages such as Python or R, and can be scaled out to reduce training time using distributed processing platforms like Apache Spark hosted in HDInsight or a Docker container.

Real-time message ingestion. In production, many advanced analytics feed real-time data streams to a predictive model that has been published as a web service. The incoming data stream is typically captured in some form of queue and a stream processing engine pulls the data from this queue and applies the prediction to the input data in near real time.

Stream processing. Once you have a trained model, prediction (or scoring) is typically a very fast operation (on the order of milliseconds) for a given set of features. After capturing real-time messages, the relevant feature values can be passed to the predictive service to generate a predicted label.

Analytical data store. In some cases, the predicted label values are written to the analytical data store for reporting and future analysis.

Analysis and reporting. As the name suggests, advanced analytics solutions usually produce some sort of report or analytical feed that includes predicted data values. Often, predicted label values are used to populate real-time dashboards.

Orchestration. Although the initial data exploration and modeling is performed interactively by data scientists, many advanced analytics solutions periodically re-train models with new data — continually refining the accuracy of the models. This retraining can be automated using an orchestrated workflow.

Fundamentals of Kubernetes on Microsoft Azure

Machine learning

Machine learning is a mathematical modeling technique used to train a predictive model. The general principle is to apply a statistical algorithm to a large dataset of historical data to uncover relationships between the fields it contains.

Machine learning modeling is usually performed by data scientists, who need to thoroughly explore and prepare the data before training a model. This exploration and preparation typically involves a great deal of interactive data analysis and visualization — usually using languages such as Python and R in interactive tools and environments that are specifically designed for this task.

Red Hat Openshift on Microsoft Azure

In some cases, you may be able to use pretrained models that come with training data obtained and developed by Microsoft. The advantage of pretrained models is that you can score and classify new content right away, even if you don't have the necessary training data, the resources to manage large datasets or to train complex models.

There are two broad categories of machine learning:

Supervised learning. Supervised learning is the most common approach taken by machine learning. In a supervised learning model, the source data consists of a set of feature data fields that have a mathematical relationship with one or more label data fields. During the training phase of the machine learning process, the data set includes both features and known labels, and an algorithm is applied to fit a function that operates on the features to calculate the corresponding label predictions. Typically, a subset of the training dataset is held back and used to validate the performance of the trained model. Once the model has been trained, it can be deployed into production, and used to predict unknown values.

Unsupervised learning. In an unsupervised learning model, the training data does not include known label values. Instead, the algorithm makes its predictions based on its first exposure to the data. The most common form of unsupervised learning is clustering, where the algorithm determines the best way to split the data into a specified number of clusters based on statistical similarities in the features. In clustering, the predicted outcome is the cluster number to which the input features belong. While they can sometimes be used directly to generate useful predictions, such as using clustering to identify groups of users in a database of customers, unsupervised learning approaches are more often used to identify which data is most useful to provide to a supervised learning algorithm in training a model.

Relevant Azure services:

• Azure Machine Learning
• Machine Learning Server (R Server) on HDInsight
• Deep learning

Machine learning models based on mathematical techniques like linear or logistic regression have been available for some time. More recently, the use of deep learning techniques based on neural networks has increased. This is driven partly by the availability of highly scalable processing systems that reduce how long it takes to train complex models. Also, the increased prevalence of big data makes it easier to train deep learning models in a variety of domains.

Openshift 3.10 & Container solutions for Blockchain, IoT and Data Science

When designing a cloud architecture for advanced analytics, you should consider the need for large-scale processing of deep learning models. These can be provided through distributed processing platforms like Apache Spark and the latest generation of virtual machines that include access to GPU hardware.

Relevant Azure services:

• Deep Learning Virtual Machine
• Apache Spark on HDInsight
• Artificial intelligence

Artificial intelligence (AI) refers to scenarios where a machine mimics the cognitive functions associated with human minds, such as learning and problem solving. Because AI leverages machine learning algorithms, it is viewed as an umbrella term. Most AI solutions rely on a combination of predictive services, often implemented as web services, and natural language interfaces, such as chatbots that interact via text or speech, that are presented by AI apps running on mobile devices or other clients. In some cases, the machine learning model is embedded with the AI app.

Democratizing Data Science on Kubernetes


Model deployment
The predictive services that support AI applications may leverage custom machine learning models, or off-the-shelf cognitive services that provide access to pretrained models. The process of deploying custom models into production is known as operationalization, where the same AI models that are trained and tested within the processing environment are serialized and made available to external applications and services for batch or self-service predictions. To use the predictive capability of the model, it is deserialized and loaded using the same machine learning library that contains the algorithm that was used to train the model in the first place. This library provides predictive functions (often called score or predict) that take the model and features as input and return the prediction. This logic is then wrapped in a function that an application can call directly or can be exposed as a web service.

Relevant Azure services:

• Azure Machine Learning
• Machine Learning Server (R Server) on HDInsight

,also

• Choosing a cognitive services technology
• Choosing a machine learning technology

Power BI for Big Data and the New Look of Big Data Solutions

More Information:

https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/

https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/data/enterprise-bi-sqldw

https://hyperonomy.com/category/microsoft-azure/

https://www.pluralsight.com/courses/azure-architecture-getting-started

https://docs.microsoft.com/en-us/azure/architecture/

https://wwt.com/all-blog/kubernetes-101/

https://docs.microsoft.com/en-us/learn/browse/?roles=solution-architect&products=azure

https://azure.microsoft.com/en-us/solutions/?cdn=disable

https://nagarajbhairaji.blogspot.com/2017/12/SmartHotel360-demoapp-Microsoft-Connect2017.html

https://mountainss.wordpress.com/2018/06/24/a-great-microservices-e-book-about-architecture-for-containerized-dotnet-apps-docker-kubernetes-containers/

https://github.com/dotnet-architecture/eShopModernizing/wiki/04.-How-to-deploy-your-Windows-Containers-based-apps-into-Kubernetes-in-Azure-Container-Service-(Including-CI-CD)

https://blogs.vmware.com/euc/2017/10/deploying-using-vmware-horizon-cloud-microsoft-azure-new-video-series.html

https://techzone.vmware.com/quick-start-tutorial-vmware-horizon-cloud-microsoft-azure/components-architecture

https://azurestack.blog/2016/10/dive-into-microsoft-azure-stack-architecture-part-1/

https://azurestack.blog/2016/10/dive-into-microsoft-azure-stack-architecture-part-2/

https://azurestack.blog/2016/10/adding-and-using-os-gallery-items-to-azure-stack-tp2/

Run your largest SAP HANA workloads on Azure. Handle transactions and analytics in-memory on a single data copy to accelerate your business processes, gain business intelligence and simplify your IT environment.

Advance to the Cloud and Beyond with SAP S/4HANA


SAP HANA on Azure offers:

On-demand M-series virtual machines certified for SAP HANA with scale up to 4 TB.
Purpose-built SAP HANA instances that scale up to 20 TB on a single node.
Scale out SAP HANA capabilities up to 60 TB.
A 99.99 per cent service-level agreement for large instances in a high-availability pair, and a 99.9 per cent SLA for a single SAP HANA large instance.



The SAP HCM Evolution from On-Premise to the Cloud

Provide self-service visualisation for data in your SAP ERP Central Component (ECC), SAP Business Warehouse (BW) and S/4HANA using the SAP HANA and SAP BW connector in Power BI Desktop. Access tools and services that help you:

Copy data from SAP HANA into Azure data services using the Azure Data Factory connector for SAP HANA, SAP ECC and SAP BW.
Store data inexpensively using Azure Data Lake Storage.
Learn from your data using Azure Databricks and Azure Machine Learning service.

Explore the Real Advantages of a Public Cloud ERP with SAP S/4HANA Cloud

Discover how to integrate SAP application data with Azure, including guidance on Azure Data Factory and SAP, Power BI and SAP, Azure Analysis Services and SAP, Azure Data Catalog and SAP, and more.

Focus will be on the following areas:

• Using Azure Data Services for Business Insights
• SAP and Azure: Drivers for the data-driven enterprise
• Azure and SAP Integration Architecture
• Azure and SAP Integrations

SAP Analytics 2019 Strategy and Roadmap

You’ve known it for some time—all SAP NetWeaver landscapes will eventually migrate to SAP HANA and the cloud. But many things still need to work side by side until that journey is complete. How many S/4HANA systems will you need to deploy? How many short-lived S/4HANA systems will be needed for project purposes? This guide provides valuable tools and processes for successfully migrating your mission-critical SAP landscapes to the cloud.

SAP S/4HANA 1709 - Highlights with Rudolf Hois


Download this white paper and learn how Azure makes it possible to:
Migrate your existing SAP systems running on all SAP supported databases.
Deploy S/4HANA systems on demand and pay only for active infrastructure usage.
Scale infrastructure of older NetWeaver systems as your processes move into S/4HANA.
Protect your data on all SAP supported databases, such as Oracle, SQL Server, or IBM DB/2.
Ensure high availability for your production SAP instances with support for SQL Server AlwaysOn, HANA System Replication (HSR) and Oracle Dataguard.

Migrating SAP Applications to Azure

As organizations shift from simply extracting and storing data to gaining valuable insights, SAP customers continue to benefit from end-to-end data and analytics capabilities that work together.

SAP offers complete data and analytics solutions that provide everything you need to help your organization make faster, more intelligent decisions. We’re excited to share more product innovation details that will help bring data and analytics even closer together for our customers in 2019.

Gartner recently recognized SAP as a leader in the January 2019 Magic Quadrant for Data Management Solutions for Analytics for the seventh consecutive year. We see this position as supported by a strong multi-cloud strategy, multi-model data processing engines, and machine learning and artificial intelligence (AI) capabilities. The ability to process transactions and analytics on a single platform has given many organizations, including Asian Paints, information at their fingertips to better serve their customers.

The SAP Cloud Platform Open Connectors service by Cloud Elements


SAP is also positioned as a visionary in this year’s Magic Quadrant for Analytics and Business Intelligence Platforms. This is the first time a cloud-based analytics product from SAP has been included as the only SAP product in this Magic Quadrant. We’re enthusiastic for the cloud analytics momentum and wanted to provide a look ahead to some of our development priorities in 2019.

SAP S/4 on HANA

The shift from Data to Intelligence

Now in its fourth year of development, product use and adoption continue to rise as more customers take advantage of new capabilities and SAP business applications embed analytics functionality within their offerings to provide customers with automated insights, supporting our customers’ strategy to deliver an intelligent enterprise. The vision of SAP Analytics Cloud is clearly resonating, with SAP’s viability rated higher than any other customer group in the survey.

SAP’s strategy to bring all analytics together in one cloud-based platform for business intelligence (BI), enterprise planning, and augmented analytics is a clear market differentiator. SAP is investing heavily in its smart analytics capabilities and is positioned for where we believe Gartner anticipates the market is heading with augmented analytics. Enhancements in smart search help users find the right content, and algorithms guide data discovery and pattern detection with no data scientist required.

SAP’s business domain expertise for lines of business and industries is unmatched. This includes data models, stories, visualization, templates, agendas, and guidance on using data sources – all packaged with SAP Analytics Cloud. This pre-built content helps customers develop use cases and quickly realize value.

SAP’s strategy to support hybrid customer scenarios for cloud and on-premise analytics with SAP Analytics Hub, part of SAP Analytics Cloud, is an advantage for customers. SAP Analytics Hub offers a single point of access to all analytics content – be it SAP or non-SAP content – no matter where it resides.

SAP Cloud Platform – Data & Storage - Overview

Quality, performance, and customer support are all continued areas of focus for the SAP Analytics Cloud team. SAP has recently added a quarterly product release cycle for SAP Analytics Cloud in addition to the two-week release cycles. This gives customers the choice of receiving new innovations with SAP Analytics Cloud at a pace that works best for their organization. You can find out about all the latest features and enhancements with the product updates for SAP Analytics Cloud.

Our Customer Success team supports customers in their adoption and use of the product. The team continues to expand its operations, helping companies around the world with dedicated support. Customers can meet the team and get started with the Welcome Guide.

Our customers’ feedback is a top priority for us and plays a decisive role in our planning. In 2019, we are also focusing on further modelling improvements, with enhancements coming in the Q1 2019 time frame, and other developments to continue improving the overall user experience of data modelling.

On the topic of data connections, we have recently added 100 data sources via the SAP Analytics Cloud agent, providing the same level of data connectivity SAP BusinessObjects customers have had for years. These enhancements complement recent investment made in significantly increasing data source connectivity for SAP Analytics Cloud customers. In addition to this, the team has enhanced connectivity to the trusted business data sources of SAP S/4HANA, SAP Business Warehouse, and SAP BusinessObjects. Learn more about data connections in SAP Analytics Cloud.

SAP S/4HANA Finance and the Digital Core


We listened when our customers and partners asked us to provide more capabilities to embed and extend analytics. SAP Analytics Cloud is in the early stages of extending application programming interfaces (APIs) and also recently added APIs for SAP Analytics Cloud on the SAP API Business Hub, offering easier access to stories. Furthermore, SAP Analytics Cloud plans to include application design capabilities to further enhance this in the Q2 time frame. This functionality will allow developers and partners to build, embed, and extend their intelligent analytics applications, further supporting SAP’s customer strategy for the Intelligent Enterprise.

Azure Data Factory offers SAP HANA and Business Warehouse data integration

Azure Data Factory is a cloud-based data integration service that orchestrates and automates the movement and transformation of data, which support copying data from 25+ data stores on-premises and in the cloud easily and performantly. Today, we are excited to announce that Azure Data Factory newly enables loading data from SAP HANA and SAP Business Warehouse (BW) into various Azure data stores for advanced analytics and reporting, including Azure Blob, Azure Data Lake, Azure SQL DW, etc.

What’s new

SAP is one of the most widely-used enterprise software in the world. We hear you that it’s crucial for Microsoft to empower customers to integrate their existing SAP system with Azure to unblock business insights. Azure Data Factory start the SAP data integration support with SAP HANA and SAP BW, which are the most popular ones in the SAP stack used by enterprise customers.

SAP HANA Academy - Replicating SAP System data in SAP HANA with SLT


With this release, you can easily ingest data from the existing SAP HANA and SAP BW to Azure, so as to build your own intelligent solutions by leveraging Azure’s first-class information management services, big data stores, advanced analytics tools, and intelligence toolkits to transform data into intelligent action. More specifically:

The SAP HANA connector supports copying data from HANA information models (such as Analytic and Calculation views) as well as Row and Column tables using SQL queries. To establish the connectivity, you need to install the latest Data Management Gateway (version 2.8) and the SAP HANA ODBC driver. Refer to SAP HANA supported versions and installation on more details.
The SAP BW connector supports copying data from SAP Business Warehouse version 7.x InfoCubes and QueryCubes (including BEx queries) using MDX queries. To establish the connectivity, you need to install the latest Data Management Gateway (version 2.8) and the SAP NetWeaver library. Refer to SAP BW supported versions and installation on more details.

99 Facts on the Future of Business in the Digital Economy

Azure Data Services for Business Insights

Azure provides a number of data services ranging from cloud storage to SQL/NoSQL databases to big data, along with various services to process the data. In this whitepaper, we are going to introduce how various Azure services can help organizations discover, make sense, process and use the data to gain competitive advantage and better ROI. The organizational data may be stored in Azure data services, SAP technologies or other third-party services and it is critical to bring them together to realize business value.

SAP Cloud Platform - Big Data Services


The Azure services described in this whitepaper such as Azure Data Catalog, Azure Data Factory, Azure Logic Apps, Azure Analysis Services and Power BI play an important role in extracting the raw data to enable intelligent data driven decision making process. A typical workflow involving Azure services is as follows:

• Azure Data Catalog helps in data source discovery and in adding context to the data.
• Azure Data Factory through automated workflows can bring together all the data into a single version of the truth by combining it from various sources. When paired with solutions such as Azure SQL/HDInsight/Spark, the insights from big data can be made consumable using other tools.
• Azure Logic Apps can help develop workflows that work with Microsoft Business Applications and third-party data. Azure Functions, the serverless architecture for applications, can help in developing custom logic into the integration process.
• Using Azure Analysis Services, users can develop complex models on top of the data. When coupled with Microsoft Power BI, a self-service visualization tool, semantic models on top of data from various sources can add efficiencies to the process.
• Power BI can provide powerful visualization to gain insights including predictive and decision insights. These can also be made available through APIs for developers to build applications or integrate with other applications.

Even though these tools can work independently support the entire lifecycle of data flows as they traverse from raw data to intelligent insights and solutions.

SAP PartnerEdge: ISV Partnership for Growth

SAP on Azure: Drivers for the data-driven enterprise

Microsoft and SAP have been working together to make SAP products deeply integrated and certified on Azure, making SAP products first class citizens on Azure. This partnership helps enterprise customers have a more seamless experience developing applications that integrates data from Azure’s Data Services along with the data stored in SAP systems. With SAP on Azure, enterprises can use machine learning and artificial intelligence on data stored in mission-critical systems and other data sources without high operational costs as deeper integration leads to cost savings.

Why SAP on Azure?

As enterprises pursue their data driven transformation, they find value in leveraging data in various Azure data services along with SAP data. A typical data migration may involve exporting SAP data onto an excel spreadsheet and then moving the data to Power BI service to visualize the data to gain insights. Apart from the operational and cost efficiencies in this process, this also leads to data silos and missed opportunities. To streamline the process, customers can use SAP on Azure with SAP HANA certified on-demand virtual machines such as Azure M-series or purpose built SAP HANA on Azure Large Instances. They can also tap into various Azure data services to make the process efficient and remove any data silos. Azure provides a robust enterprise grade platform that offers agility and scale needed for integrating with SAP technologies along with security.

SAP Cloud Platform Integration Suite Q3/Q4 Webinar


Some benefits of running SAP applications on Azure are:

• On-Demand and scalable: Azure provides compute resources on-demand with a pay per use model. Enterprises can scale the resources up and down based on their usage needs. Additionally, Azure also offers purpose-built HANA large instances that can support large memory implementations. 
• Intelligent solutions: By taking advantage of Azure data services, analytics service and tools like Power BI, users gain insights from their data. 
• Security and Compliance: Azure makes it easy to protect your data and solutions using encryption, Azure Security Center and Azure Active Directory, which provides single sign on across multiple services, both on-premises and cloud. With a variety of industry compliance and trust certifications, Azure provides a complete compliance solution. 
• Global scale: Azure has the largest footprint of any cloud providers with 54 regions to support any scale, enabling organizations to optimize for the best user experience and to meet local data residency requirements. 
• Enterprise-grade Resilience: Azure provides enterprise grade resilience with redundancy and geo-replication with 99.99% SLA. 
• Lower TCO: Azure also provides for a 60% cost reduction when compared to traditional, on-premise storage systems. By using SAP technologies on Azure, customers can realize 40-75% TCO cost savings in Dev and Test environments.

Azure and SAP Integration Architecture

The following diagram gives a high-level architecture for running SAP and third-party data within the Azure data platform and Azure Analytics Services to build intelligent solutions.

Azure Data Platform combines the business operational data with streaming and analytics data:

• To provide a data science service which data scientists, developers and other enterprise applications can use
• An easy interface to access analyze, and visualize data to gain valuable insights

Whether it is about SAP applications or Azure Big Data services or Azure Analytics services or Azure Data Lake or external data sources, Azure makes it easy to manage the data and enforce policy and governance while providing developers with a seamless interface. With an easy extensibility of the data platform to bring together data from many sources and IoT Hub to collect data from various IoT devices, Azure is empowering customers to derive maximum value from their data.

We will go in depth about integration of SAP data with Azure’s data services and how customers can derive business value from these integrations.

Azure and SAP Integrations

For various Azure services:
1. Azure Data Factory and SAP
2. Power BI and SAP
3. Azure Analysis Services and SAP
4. Azure Data Catalog and SAP
5. Using iPaaS to integrate Microsoft Business Apps With SAP
6. Azure Active Directory and SAP

iPaaS: Bringing Microsoft Business Apps Workflows With SAP

Azure’s Integration Platform (iPaaS) offers an easy way to integrate Microsoft business applications with SAP technologies and other third party applications. Azure Logic Apps and other Azure services like Azure Functions and Azure Service Bus Messaging works together to provide this integration layer. Azure Logic Apps helps to implement and orchestrate visual workflows to leverage business processes using 100+ connections across different protocols, applications and systems running on Azure and on-premises. Azure Functions Service can then be tapped to bring custom logic, either as functions or Microservices, that is then leveraged by Azure Logic Apps. Azure Service Bus Messaging layer can then help decouple various steps in the integration process. If necessary, the API Management service can be used to handle http triggers.

SAP Cloud Platform in the Garage Virtual Event | Internet of Things (IoT) in the Cloud


Azure Logic Apps also connects Office 365 and Dynamics 365 applications with other enterprise applications, SaaS applications, SAP technologies and other third party applications. This helps bring business data stored in Office 365 and Dynamics 365 to work along with other data sources, making it easy to derive critical business workflows.

Azure Logic Apps Connector to SAP

Azure Logic Apps is the cloud based iPaaS offering that helps organization connect disparate data sources including SaaS and enterprise applications. By offering many out of the box integrations, Logic Apps lets you seamlessly connect data, applications and devices across cloud and on-premises to develop complex business workflows.

Key Benefits

• Simplify and implement complex, scalable integrations and workflows for enterprise applications on the cloud, on-premises and Office 365
• Brings speed and scalability into the enterprise integration space, Logic Apps scale up and down based on demand
• Easy user interface with designer
• Powerful management tools to tame the complexity
• Easy to automate EAI, B2B/EDI, and business processes

In this section, we will briefly discuss various integration that can help Azure Logic App work with SAP technologies. SAP ERP Central Component (ECC) connector allows Azure Logic Apps to connect to on-premises or cloud SAP resources from inside a logic app. The connector supports message or data integration to and from SAP NetWeaver-based systems through Intermediate Document (IDoc) or Business Application Programming Interface (BAPI) or Remote Function Call (RFC).

The connector allows following three operations:

• Send to SAP: Send IDoc or call BAPI functions over tRFC in SAP systems.
• Receive from SAP: Receive IDoc or BAPI function calls over tRFC from SAP systems.
• Generate schemas: Generate schemas for the SAP artifacts for IDoc or BAPI or RFC.

More information about this you can find use these resources:

• Azure Logic Apps documentation
• Connectors for Azure Logic Apps
• Connect to SAP systems from Azure Logic Apps

SAP Cloud Platform Training | SAP HCP Training | SAP SCP Training

Azure is the right cloud platform to meet the demands of such data-driven enterprises. With the wide variety of services available in the Azure portfolio, organizations can easily bring all the data sources together including data residing in systems such as SAP, analyze the data, author, build models, deliver it for consumption by developer through an API and visualize using powerful and easy to use tools.

Introducing SAP HANA Spatial Services Application (2019 Edition)

Whether it is data commercialization using a Data Science as a Service platform or building a powerful predictive or decision analytics platform, Azure’s diverse set of data and integration services such as Azure Data Catalog, Azure Data Factory, Azure Logic Apps, Azure Analysis Service and Power BI with its tight integration with SAP technologies, are well positioned to help enterprises in their data driven journey. Filly, Azure Active Directory provides the enterprise grade identity management and security that is critical for protecting valuable organizational data. This whitepaper has given you an overview on how these technologies can be used to gain useful insights or build intelligent applications in your journey to get the most value for your SAP data on Azure.

More Information:

https://azure.microsoft.com/en-gb/resources/integrating-sap-application-data-with-azure/

https://docs.microsoft.com/en-gb/azure/virtual-machines/workloads/sap/hana-overview-architecture

https://azure.microsoft.com/en-us/blog/azure-data-factory-offer-sap-hana-and-business-warehouse-data-integration/

https://www.sapanalytics.cloud/learning/data-connections/

https://www.epiuselabs.com/lets-talk-hcm/successconnect-2018-summary-erp-sac-payroll

https://www.epiuselabs.com/news

https://azure.microsoft.com/en-us/blog/azure-data-factory-offer-sap-hana-and-business-warehouse-data-integration/

https://www.forbes.com/sites/gilpress/2016/08/05/iot-mid-year-update-from-idc-and-other-research-firms/#515551655c59

https://www.gartner.com/en/newsroom/press-releases/2017-10-02-gartner-survey-of-more-than-3000-cios-confirms-the-changing-role-of-the-chief-information-officer

https://customers.microsoft.com/en-us/story/ab-inbev-consumer-goods-azure

https://customers.microsoft.com/en-us/story/newell-brands-consumer-goods-azure

https://customers.microsoft.com/en-gb/story/co-op

https://azure.microsoft.com/mediahandler/files/resourcefiles/leveraging-sap-on-azure-for-business-transformation/Leveraging_SAP_on_Azure_for_Business_Transformation.pdf

https://www.appseconnect.com/ipaas-for-sap-ecc/

https://blogs.sap.com/2017/12/08/the-integration-conundrum-a-point-of-view-paper/

The Next Decade in Quantum Computing

The experts are convinced that in time they can build a high-performance quantum computer. Given the technical hurdles that quantum computing faces—manipulations at nanoscale, for instance, or operating either in a vacuum environment or at cryogenic temperatures—the progress in recent years is hard to overstate. In the long term, such machines will very likely shape new computing and business paradigms by solving computational problems that are currently out of reach. They could change the game in such fields as cryptography and chemistry (and thus material science, agriculture, and pharmaceuticals) not to mention artificial intelligence (AI) and machine learning (ML). We can expect additional applications in logistics, manufacturing, finance, and energy. Quantum computing has the potential to revolutionize information processing the way quantum science revolutionized physics a century ago.

Quantum Computing


Every company needs to understand how quantum computing discoveries will affect business.

The future of quantum computing

The full impact of quantum computing is probably more than a decade away. But there is a much closer upheaval gathering force, one that has significance now for people in business and that promises big changes in the next five to ten years. Research underway at multiple major technology companies and startups, among them IBM, Google, Rigetti Computing, Alibaba, Microsoft, Intel, and Honeywell, has led to a series of technological breakthroughs in building quantum computer systems. These efforts, complemented by government-funded R&D, make it all but certain that the near to medium term will see the development of medium-sized, if still error-prone, quantum computers that can be used in business and have the power and capability to produce the first experimental discoveries. Already quite a few companies are moving to secure intellectual property (IP) rights and position themselves to be first to market with their particular parts of the quantum computing puzzle. Every company needs to understand how coming discoveries will affect business. Leaders will start to stake out their positions in this emerging technology in the next few years.

Toward a Quantum Revolution for Computing


This report explores essential questions for executives and people with a thirst to be up-to-speed on quantum computing. We will look at where the technology itself currently stands, who is who in the emerging ecosystem, and the potentially interesting applications. We will analyze the leading indicators of investments, patents, and publications; which countries and entities are most active; and the status and prospects for the principal quantum hardware technologies. We will also provide a simple framework for understanding algorithms and assessing their applicability and potential. Finally, our short tour will paint a picture of what can be expected in the next five to ten years, and what companies should be doing—or getting ready for—in response.

How Quantum Computers Are Different, and Why It Matters

The first classical computers were actually analog machines, but these proved to be too error-prone to compete with their digital cousins. Later generations used discrete digital bits, taking the values of zero and one, and some basic gates to perform logical operations. As Moore’s law describes, digital computers got faster, smaller, and more powerful at an accelerating pace. Today a typical computer chip holds about 20x109 bits (or transistors) while the latest smartphone chip holds about 6x109 bits. Digital computers are known to be universal in the sense that they can in principle solve any computational problem (although they possibly require an impractically long time). Digital computers are also truly reliable at the bit level, with fewer than one error in 1024 operations; the far more common sources of error are software and mechanical malfunction.

Qubits can enable quantum computing to achieve an exponentially higher information density than classical computers.

Quantum computing random walks and adiabatic computation


Quantum computers, building on the pioneering ideas of physicists Richard Feynman and David Deutsch in the 1980s, leverage the unique properties of matter at nanoscale. They differ from classical computers in two fundamental ways. First, quantum computing is not built on bits that are either zero or one, but on qubits that can be overlays of zeros and ones (meaning part zero and part one at the same time). Second, qubits do not exist in isolation but instead become entangled and act as a group. These two properties enable qubits to achieve an exponentially higher information density than classical computers.



There is a catch, however: qubits are highly susceptible to disturbances by their environment, which makes both qubits and qubit operations (the so-called quantum gates) extremely prone to error. Correcting these errors is possible but it can require a huge overhead of auxiliary calculations, causing quantum computers to be very difficult to scale. In addition, when providing an output, quantum states lose all their richness and can only produce a restricted set of probabilistic answers. Narrowing these probabilities to the “right” answer has its own challenges, and building algorithms in a way that renders these answers useful is an entire engineering field in itself.

Future Decoded Quantum Computing Keynote


That said, scientists are now confident that quantum computers will not suffer the fate of analog computers—that is, being killed off by the challenges of error correction. But the requisite overhead, possibly on the order of 1,000 error-correcting qubits for each calculating qubit, does mean that the next five to ten years of development will probably take place without error correction (unless a major breakthrough on high-quality qubits surfaces). This era, when theory continues to advance and is joined by experiments based on these so-called NISQ (Noisy Intermediate-Scale Quantum) devices, is the focus of this report. (For more on the particular properties of quantum computers, see the sidebar, “The Critical Properties of Quantum Computers.” For a longer-term view of the market potential for, and development of, quantum computers, see “The Coming Quantum Leap in Computing,” BCG article, May 2018. For additional context—and some fun—take the BCG Quantum Computing Test.)

The Emerging Quantum Computing Ecosystem

Quantum computing technology is well-enough developed, and practical uses are in sufficiently close sight, for an ecosystem of hardware and software architects and developers, contributors, investors, potential users, and collateral players to take shape. Here’s a look at the principal participants.

TECH COMPANIES

Universities and research institutions, often funded by governments, have been active in quantum computing for decades. More recently, as has occurred with other technologies (big data for example), an increasingly well-defined technology stack is emerging, throughout which a variety of private tech players have positioned themselves.

Efficient Synthesis of Universal Probabilistic Quantum Circuits


An increasingly well-defined technology stack is emerging.

At the base of the stack is quantum hardware, where the arrays of qubits that perform the calculations are built. The next layer is sophisticated control systems, whose core role is to regulate the status of the entire apparatus and to enable the calculations. Control systems are responsible in particular for gate operations, classical and quantum computing integration, and error correction. These two layers continue to be the most technologically challenging.Next comes a software layer to implement algorithms (and in the future, error codes) and to execute applications. This layer includes a quantum-classical interface that compiles source code into executable programs. At the top of the stack are a wider variety of services dedicated to enabling companies to use quantum computing. In particular they help assess and translate real-life problems into a problem format that quantum computers can address.

The actual players fall into four broad categories.



End-to-End Providers. These tend to be big tech companies and well-funded startups. Among the former, IBM has been the pioneer in quantum computing and continues at the forefront of the field. The company has now been joined by several other leading-edge organizations that play across the entire stack. Google and more recently Alibaba have drawn a lot of attention. Microsoft is active but has yet to unveil achievements toward actual hardware. Honeywell has just emerged as a new player, adding to the heft of the group. Rigetti is the most advanced among the startups. (See “Chad Rigetti on the Race for Quantum Advantage: An Interview with the Founder and CEO of Rigetti Computing,” BCG interview, November 2018.)

Each company offers its own cloud-based open-source software platform and varying levels of access to hardware, simulators, and partnerships. In 2016 IBM launched Q Experience, arguably still the most extensive platform to date, followed in 2018 by Rigetti’s Forest, Google’s Cirq, and Alibaba’s Aliyun, which has launched a quantum cloud computing service in cooperation with the Chinese Academy of Sciences. Microsoft provides access to a quantum simulator on Azure using its Quantum Development Kit. Finally, D-Wave Systems, the first company ever to sell quantum computers (albeit for a special purpose), launched Leap, its own real-time cloud access to its quantum annealer hardware, in October 2018.

Particle Physics On a Chip the Search for Majorana Fermions -  Leo Kouwenhoven


Hardware and Systems Players. Other entities are focused on developing hardware only, since this is the core bottleneck today.  Again, these include both technology giants, such as Intel, and startups, such as IonQ, Quantum Circuits, and QuTech. Quantum Circuits, a spinoff from Yale University, intends to build a robust quantum computer based on a unique, modular architecture, while QuTech—a joint effort between Delft University of Technology and TNO, the applied scientific research organization, in the Netherlands—offers a variety of partnering options for companies. An example of hardware and systems players extending into software and services, QuTech launched Quantum Inspire, the first European quantum computing platform, with supercomputing access to a quantum simulator. Quantum hardware access is planned to be available in the first half of 2019.



Software and Services Players. Another group of companies is working on enabling applications and translating real-world problems into the quantum world. They include Zapata Computing, QC Ware, QxBranch, and Cambridge Quantum Computing, among others, which provide software and services to users. Such companies see themselves as an important interface between emerging users of quantum computing and the hardware stack. All are partners of one or more of the end-to-end or hardware players within their mini-ecosystems. They have, however, widely varying commitments and approaches to advancing original quantum algorithms.

Specialists. These are mainly startups, often spun off from research institutions, that provide focused solutions to other quantum computing players or to enterprise users. For example, Q-CTRL works on solutions to provide better system control and gate operations, and Quantum Benchmark assesses and predicts errors of hardware and specific algorithms. Both serve hardware companies and users.

The ecosystem is dynamic and the lines between tech layers easily blurred or crossed.

Particle Physics On a Chip the Search for Majorana Fermions

The ecosystem is dynamic and the lines between layers easily blurred or crossed, in particular by maturing hardware players extending into the higher-level application, or even service layers. The end-to-end integrated companies continue to reside at the center of the technology ecosystem for now; vertical integration provides a performance advantage at the current maturity level of the industry. The biggest investments thus far have flowed into the stack’s lower layers, but we have not yet seen a convergence on a single winning architecture. Several architectures may coexist over a longer period and even work hand-in-hand in a hybrid fashion to leverage the advantages of each technology.

APPLICATIONS AND USERS

For many years, the biggest potential end users for quantum computing capability were national governments. One of the earliest algorithms to demonstrate potential quantum advantage was developed in 1994 by mathematician Peter Shor, now at the Massachusetts Institute of Technology. Shor’s algorithm has famously demonstrated how a quantum computer could crack current cryptography. Such a breach could endanger communications security, possibly undermining the internet and national defense systems, among other things. Significant government funds flowed fast into quantum computing research thereafter. Widespread consensus eventually formed that algorithms such as Shor’s would remain beyond the realm of quantum computers for some years to come and even if current cryptographic methods are threatened, other solutions exist and are being assessed by standard-setting institutions. This has allowed the private sector to develop and pursue other applications of quantum computing. (The covert activity of governments continues in the field, but is outside the scope of this report.)

Quantum Algorithm - The Math of Intelligence


Quite a few industries outside the tech sector have taken notice of the developments in, and the potential of, quantum computing, and companies are joining forces with tech players to explore potential uses. The most common categories of use are for simulation, optimization, machine learning, and AI. Not surprisingly, there are plenty of potential applications.





Despite many announcements, though, we have yet to see an application where quantum advantage—that is, performance by a quantum computer that is superior in terms of time, cost, or quality—has been achieved.

However, such a demonstration is deemed imminent, and Rigetti recently offered a $1 million prize to the first group that proves quantum advantage.

Investments, Publications, and Intellectual Property

The activity around quantum computing has sparked a high degree of interest.2 People have plenty of questions. How much money is behind quantum computing? Who is providing it? Where does the technology stand compared with AI or blockchain? What regions and entities are leading in publications and IP?



With more than 60 separate investments totaling more than $700 million since 2012, quantum computing has come to the attention of venture investors, even if is still dwarfed by more mature and market-ready technologies such as blockchain (1,500 deals, $12 billion, not including cryptocurrencies) and AI (9,800 deals, $110 billion).

The bulk of the private quantum computing deals over the last several years took place in the US, Canada, the UK, and Australia. Among startups, D-Wave ($205 million, started before 2012), Rigetti ($119 million), PsiQ ($65 million), Silicon Quantum Computing ($60 million), Cambridge Quantum Computing ($50 million), 1Qbit ($35 million), IonQ ($22 million), and Quantum Circuits ($18 million) have led the way.



A regional race is also developing, involving large publicly funded programs that are devoted to quantum technologies more broadly, including quantum communication and sensing as well as computing. China leads the pack with a $10 billion quantum program spanning the next five years, of which $3 billion is reserved for quantum computing. Europe is in the game ($1.1 billion of funding from the European Commission and European member states), as are individual countries in the region, most prominently the UK ($381 million in the UK National Quantum Technologies Programme). The US House of Representatives passed the National Quantum Initiative Act ($1.275 billion, complementing ongoing Department of Energy, Army Research Office, and National Science Foundation initiatives). Many other countries, notably Australia, Canada, and Israel are also very active.



The money has been accompanied by a flurry of patents and publishing. (See Exhibit 4.) North America and East Asia are clearly in the lead; these are also the regions with the most active commercial technology activity. Europe is a distant third, an alarming sign, especially in light of a number of leading European quantum experts joining US-based companies in recent years. Australia, a hotspot for quantum technologies for many years, is striking given its much smaller population. The country is determined to play in the quantum race; in fact, one of its leading quantum computing researchers, Michelle Simmons, was named Australian of the Year 2018.



Two things are noteworthy about the volume of scientific publishing regarding quantum computing since 2013. (See Exhibit 5.) The first is the rise of China, which has surpassed the US to become the leader in quantity of scientific articles published.3 The second is the high degree of international collaboration (in which the US remains the primary hub). The cooperation shows that quantum computing is not dominated by national security interests yet, owing in large part to consensus around the view that cryptographic applications are still further in the future and that effective remedies for such applications are in the making. The collaboration activity also reflects the need in the scientific community for active exchange of information and ideas to overcome quantum computing’s technological and engineering challenges.

Simplifying the Quantum Algorithm Zoo

The US National Institute for Standards and Technology (NIST) maintains a webpage entitled Quantum Algorithm Zoo that contains descriptions of more than 60 types of quantum algorithms. It’s an admirable effort to catalog the current state of the art, but it will make nonexperts’ heads spin, as well as those of some experts.

Quantum algorithms are the tools that tell quantum computers what to do. Two of their attributes are especially important in the near term:

Speed-Up. How much faster can a quantum computer running the algorithm solve a particular class of problem than the best-known classical computing counterpart?
Robustness. How resilient is the algorithm to the random “noise,” or other errors, in quantum computing?




There are two classes of algorithm today. (See Exhibit 8.) We call the first purebreds—they are built for speed in noiseless or error-corrected environments. The ones shown in the exhibit have theoretically proven exponential speed-up over conventional computers for specific problems, but require a long sequence of flawless execution, which in turn necessitate very low noise operations and error correction. This class includes Peter Shor’s factorization algorithm for cracking cryptography and Trotter-type algorithms used for molecular simulation. Unfortunately, their susceptibility to noise puts them out of the realm of practical application for the next ten years and perhaps longer.

Why the Deutsch-Josza Algorithm?

Deutsch–Jozsa algorithm. ... Although of little practical use, it is one of the first examples of a quantum algorithm that is exponentially faster than any possible deterministic classical algorithm. It is also a deterministic algorithm, meaning that it always produces an answer, and that answer is always correct.

Deutsch-Jozsa Algorithm

The Deutsch-Jozsa algorithm was the first to show a separation between the quantum and classical difficulty of a problem. This algorithm demonstrates the significance of allowing quantum amplitudes to take both positive and negative values, as opposed to classical probabilities that are always non-negative.


The Deutsch-Jozsa problem is defined as follows. Consider a function f(x)
 that takes as input n
-bit strings x
 and returns 0
 or 1
. Suppose we are promised that f(x)
 is either a constant function that takes the same value c∈{0,1}
 on all inputs x
, or a balanced function that takes each value 0
 and 1
 on exactly half of the inputs. The goal is to decide whether f
 is constant or balanced by making as few function evaluations as possible. Classically, it requires 2n−1+1
 function evaluations in the worst case. Using the Deutsch-Jozsa algorithm, the question can be answered with just one function evaluation. In the quantum world the function f
 is specified by an oracle circuit Uf
 (see the previous section on Grover’s algorithm, such that Uf|x⟩=(−1)f(x)|x⟩
).

Quantum Circuits and Algorithms


To understand how the Deutsch-Jozsa algorithm works, let us first consider a typical interference experiment: a particle that behaves like a wave, such as a photon, can travel from the source to an array of detectors by following two or more paths simultaneously. The probability of observing the particle will be concentrated at those detectors where most of the incoming waves arrive with the same phase.

Imagine that we can set up an interference experiment as above, with 2n
 detectors and 2n
 possible paths from the source to each of the detectors. We shall label the paths and the detectors with n
-bit strings x
 and y
 respectively. Suppose further that the phase accumulated along a path x
 to a detector y
 equals C(−1)f(x)+x⋅y
, where
x⋅y=∑ni=1xiyi

is the binary inner product and C
 is a normalizing coefficient. The probability to observe the particle at a detector y
 can be computed by summing up amplitudes of all paths x
 arriving at y
 and taking the absolute value squared:
Pr(y)=|C∑x(−1)f(x)+x⋅y|2
Normalization condition ∑yPr(y)=1
 then gives C=2−n
. Let us compute the probability Pr(y=0n)
 of observing the particle at the detector y=0n
 (all zeros string). We have Pr(y=0n)=|2−n∑x(−1)f(x)|2
If f(x)=c
 is a constant function, we get Pr(y=0n)=|(−1)c|2=1
. However, if f(x)
 is a balanced function, we get Pr(y=0n)=0
, since all the terms in the sum over x
 cancel each other.
We can therefore determine whether f
 is constant or balanced with certainty by running the experiment just once.
Of course, this experiment is not practical since it would require an impossibly large optical table! However, we can simulate this experiment on a quantum computer with just n
qubits and access to the oracle circuit Uf
. Indeed, consider the following algorithm:
Step 1. Initialize n
 qubits in the all-zeros state |0,…,0⟩
.
Step 2. Apply the Hadamard gate H
 to each qubit.
Step 3. Apply the oracle circuit Uf
.
Step 4. Repeat Step 2.
Step 5. Measure each qubit. Let y=(y1,…,yn)
 be the list of measurement outcomes.
We find that f
 is a constant function if y
 is the all-zeros string. Why does this work? Recall that the Hadamard gate H
 maps |0⟩
 to the uniform superposition of |0⟩
 and |1⟩
. Thus the state reached after Step 2 is 2−n/2∑x|x⟩
, where the sum runs over all n
-bit strings. The oracle circuit maps this state to 2−n/2∑x(−1)f(x)|x⟩
. Finally, let us apply the layer of Hadamards at Step 4. It maps a basis state |x⟩
 to a superposition 2−n/2∑y(−1)x⋅y|y⟩
. Thus the state reached after Step 4 is |ψ⟩=∑yψ(y)|y⟩
, where ψ(y)=2−n∑x(−1)f(x)+x⋅y
.

Using QISkit: The SDK for Quantum Computing

This is exactly what we need for the interference experiment described above. The final measurement at Step 5 plays the role of detecting the particle. As was shown above, the probability to measure y=0n
 at Step 5 is one if f
 is a constant function and zero if f
 is a balanced function. Thus we have solved the Deutsch-Jozsa problem with certainty by making just one function evaluation.
Example circuits
Suppose n=3
 and f(x)=x0⊕x1x2
. This function is balanced since flipping the bit x0
 flips the value of f(x)
 regardless of x1,x2
. To run the Deutsch-Jozsa algorithm we need an explicit description of the oracle circuit Uf
 as a sequence of quantum gates. To this end we need a Z0
 gate such that Z0|x⟩=(−1)x0|x⟩
 and a controlled-Z gate CZ1,2
 such that CZ1,2|x⟩=(−1)x1x2|x⟩
.  Using basic circuit identities (see the Basic Circuit Identities and Larger Circuits section), one can realize the controlled-Z gate as a CNOT sandwiched between two Hadamard gates.

Quantum information and computation: Why, what, and how


A Potential Quantum Winter, and the Opportunity Therein

Like many theoretical technologies that promise ultimate practical application someday, quantum computing has already been through cycles of excitement and disappointment. The run of progress over past years is tangible, however, and has led to an increasingly high level of interest and investment activity. But the ultimate pace and roadmap are still uncertain because significant hurdles remain. While the NISQ period undoubtedly has a few surprises and breakthroughs in store, the pathway toward a fault-tolerant quantum computer may well turn out to be the key to unearthing the full potential of quantum computing applications.

Some experts thus warn of a potential “quantum winter,” in which some exaggerated excitement cools and the buzz moves to other things. Even if such a chill settles in, for those with a strong vison of their future in both the medium and longer terms, it may pay to remember the banker Baron Rothschild’s admonition during the panic after the Battle of Waterloo: “The time to buy is when there’s blood in the streets.” During periods of disillusionment, companies build the basis of true competitive advantage. Whoever stakes out the most important business beachheads in the emerging quantum computing technologies will very likely do so over the next few years. The question is not whether or when, but how, companies should get involved.

More Information

http://www.infocobuild.com/education/audio-video-courses/physics/QuantumInformationComputing-IIT-Bombay/lecture-17.html

https://www.microsoft.com/en-us/research/group/microsoft-quantum-redmond-quarc/

https://www.microsoft.com/en-us/research/group/microsoft-quantum-santa-barbara-station-q/

http://kouwenhovenlab.tudelft.nl/

https://qdev.nbi.ku.dk/

https://equs.org/users/prof-david-reilly

https://www.microsoft.com/en-us/research/research-area/quantum/

https://www.microsoft.com/en-us/research/blog/future-is-quantum-with-dr-krysta-svore/

https://www.microsoft.com/en-us/quantum/development-kit

https://www.tudelft.nl/en/2017/tnw/majorana-highway-on-a-chip/

https://phys.org/news/2017-07-majorana-highway-chip.html

https://www.microsoft.com/en-us/research/blog/future-is-quantum-with-dr-krysta-svore/

https://quantumexperience.ng.bluemix.net/proxy/tutorial/full-user-guide/004-Quantum_Algorithms/080-Deutsch-Jozsa_Algorithm.html

http://demonstrations.wolfram.com/DeutschsAlgorithmOnAQuantumComputer/

https://singularityhub.com/2019/02/26/quantum-computing-now-and-in-the-not-too-distant-future/

https://www.scientificamerican.com/article/is-the-u-s-lagging-in-the-quest-for-quantum-computing/?redirect=1

https://www.quantamagazine.org/tag/the-future-of-quantum-computing/

https://www.quantamagazine.org/quantum-computers-struggle-against-classical-algorithms-20180201/

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6205278/

https://www.bcg.com/en-us/publications/2018/next-decade-quantum-computing-how-play.aspx

https://en.wikipedia.org/wiki/Deutsch–Jozsa_algorithm

Hybrid Cloud: reference architectures for Power Systems

Best practices for deploying your apps in the cloud

Determine the best deployment for your apps on a cloud infrastructure

As a developer, you probably hear a lot about new technologies that promise to increase the speed at which you can develop software, as well as ones that can increase the resiliency of your applications once you have deployed them. Your challenge is to wade through these emerging technologies and determine which ones actually hold promise for the projects that you are currently working on.

IBM Cloud Paks in 2 Minutes


No doubt, you are aware that cloud computing offers great promise for developers. However, you might not know about the areas where this technology can provide value to you and your projects. You also might not know good practices to employ when implementing a project in the cloud. This article explores the types of cloud computing systems available, and provides guidelines that can help you with real-world application deployments on top of a cloud infrastructure.

2019 CIO Think Tank: Pathways to Multicloud Transformation

Choose between IaaS, PaaS, and SaaS

When people begin discussing cloud computing, they are generally speaking about one of three possible deployment choices for application code: infrastructure as a service (IaaS), platform as a service (PaaS), or software as a service (SaaS). Which one is right for your project depends on your specific needs for the code base that you are working on. Let’s examine each one of these cloud choices.

Infrastructure as a service (IaaS)

IaaS is a platform where an infrastructure is provided for you. With a click of a button, you can spin up virtual machines hosted by a provider with an operating system of your choice. The vendor providing the machine is responsible for the connectivity and initial provisioning of the system, and you are responsible for everything else. The vendor provides a machine and an operating system, but you need to install all of the software packages, application runtimes/servers, and databases that your application requires. Generally, IaaS requires that you have a team of system administrators to manage the system and apply firewall rules, patches, and security errata on a frequent basis.

Pro: You have complete control over every aspect of the system.
Con: You need system administration knowledge or a team of system administrators to maintain the systems, since you are responsible for their uptime and security.

Modernize existing z/OS applications using IBM Cloud Private and IBM Cloud

Platform as a service (PaaS)

PaaS is a fairly new technology stack that runs on top of IaaS and was created with the developer in mind. With the PaaS platform, everything is provided except the application code, users, and data. Typically, when using a PaaS, the vendor maintains the application server, databases, and all of the necessary operating system components, giving you time to focus on the application code. Since the vendor manages that platform for you, it is often hard to open up ports that are not specifically called for the application server, runtime, or database in use. PaaS also provides features that are specifically meant for applications, including the ability to scale the application tier up based upon the user demand of the application. In most platforms, this happens with little-to-no interaction from the developer.

Pro: PaaS provides a complete environment that is actively managed, letting you focus on your application code.
Con: Developers are often restricted to certain major/minor versions of packages available on the system so that the vendor can manage the platform effectively.

Pathways to Multicloud Transformation

Software as a service (SaaS)

With the SaaS platform, everything is provided for you except the users and the application data. The vendor provides the application code and the developer has limited access to modify the software in use. This is typically not a choice for deploying custom applications, as the vendor provides the entire software stack. Hosted web email clients and hosted sales automation software are two good examples of how SaaS is used.

Pro: The entire stack is provided by the vendor except the application users and the associated data.
Con: You have limited control over the hosted application and it’s often hard to integrate external workflows into the system.


IBMBusiness Resiliency Services - IBM Cloud Resiliency Orchestration




Which should you choose?
As an application developer, you should choose PaaS, because that the infrastructure is managed for you, so you can focus on your application code.

Scale your application
As mentioned previously, PaaS provides scaling out of the box for most languages and runtimes. However, as a developer you need to be aware of the types of scaling offered and when it makes sense to scale horizontally or vertically.

Portable Apps across IBM Kubernetes Service and IBM Cloud Private



Vertical scaling
Vertical scaling refers to a type of scaling that has been the default choice for decades. This type of scaling refers to the notion that to handle load, you simply use larger systems. This is one of the reasons why there are servers in place today with a terabyte of RAM and a massive number of CPUs and cores to serve a single Java® application. Typically when using vertical scaling, a single large system is used to handle most or all of the application requests from the users.

Horizontal scaling
With horizontal scaling, the application load and requests are spread over a group of smaller servers that are typically behind a load balancer. As a request from a user is made, the load balancer sends the request to a server and then manages the session state across the cluster of servers. There are usually two types of horizontal scaling to use to ensure the best possible experience for the users of your application: manual and automatic scaling.

Manage hybrid IT with IBM Services for Multicloud Management


Manual scaling
With manual scaling, you specify that you want the application to scale up to handle increased traffic when you know you have an upcoming event that will increase application demand. For example, if you know that you are going to be running a marketing campaign to attract more users to your application, you might want to proactively add additional servers to your cluster. Most PaaS providers allow you to accomplish this task with a simple command.

Automatic scaling
With automatic scaling, you specify conditions where your application will automatically scale without any human interaction. This condition can be based on such things as the number of concurrent HTTP requests your application is receiving, or the amount of the CPU that your application is using. This enables the developer to automatically add new servers to the load balancer when the demand for the application is high. Automatic scaling provides a truly hands-off approach to scaling while ensuring that demand from the users is met in a timely fashion. Automatic scaling is crucial when you have unplanned use of your application due to certain circumstances. For example, you might get your mobile application featured on an application store for a short period of time when your back-end services reside in the cloud.







Which application scaling should you choose?

As a developer, you should choose a platform that allows for both manual and automatic horizontal scaling of your application.

Consider application state
Most cloud providers that provide a PaaS want you to start with green field development, which means that projects that are not affected by the constraints of prior work. Porting existing or legacy applications to the platform can be a challenge, mainly because the file systems in place are ephemeral in nature and do not allow for saving application state or resources on the file system.

This restriction is why you might hear that you need to think about future applications as being stateless. To receive the benefits of an infrastructure that resides in the cloud, you need to employ stateless application design in your projects. To achieve that, take into account the following practices for new applications:

Allow the application server or container to maintain the session state of the user across the cluster instead of relying on the file system.
Do not store files or user assets on the physical file system of the server that your code is deployed to. Instead, consider using a cloud-based storage service and delivering assets through the provided REST API for the storage service.
Use a database for storing assets related to a user if you do not have access to use a cloud storage API.

Creating Production-Ready, Secure and Scalable Applications in IBM Cloud Private




Which application state should you choose?
For green field applications, you should design applications that are stateless, which means they do not store user assets or resources on the file system. For legacy or existing applications, choose a PaaS provider that supports both stateful and stateless applications.


Choose a database for cloud-enabled applications

Almost all applications being created today rely on a database of some type on the back end to store and retrieve information to be presented to the user. When developing applications for the cloud, you must also take into consideration what databases you will be using and where those databases should be located. Should the database be hosted on the same servers as the application, or is it better to house the database on a separate server or container?
In many cases, an application relies on information that’s stored in a database that resides behind a corporate firewall, while the application front end is deployed on the public cloud. When this is the case, you have a couple of options for effectively accessing the information that you’ll need to present to the user on the front end.

• Option 1: Choose a provider that allows you to open up a remote VPC connection back to your database.
• Option 2: Communicate to the database through a set of authenticated REST services deployed on the infrastructure that have access to the data.

Both of these options have inherent security risks that you need to consider when connecting to a database behind a corporate firewall from an outside cloud application. When this is the case, your best option is to select a cloud PaaS vendor that allows you to deploy your applications on a non multi-tenant environment.

Data and AI for hybrid cloud and multi cloud world


If your application code does not need to connect to an existing corporate database, the number of options that you have are almost endless. I suggest that you deploy your database in the same geography/datacenter/region as your application code but on different containers or servers than your front-end application code. Use this option to scale the database independently of the web tier. Also, be sure to choose a database that scales quickly and easily regardless of whether it’s a SQL or NOSQL database.

Deploying Kubernetes in the Enterprise IBM



Consider multiple geographies

One of the great benefits of cloud computing is that you can deploy your application infrastructure throughout the world with little or no up-front cost. For example, deploying an application that has servers in both North America and EMEA has traditionally incurred a huge up-front cost to purchase and provision hardware and data centers. With an infrastructure that resides in the cloud, you can effortlessly deploy your application across as many geographies as your vendor supports. For simple applications that only have a limited number of users, this is not required. However, having access to deploy code in multiple geographies is critical to winning customer satisfaction by locating the application code as close to your target audience as possible.

Throw in the ability to manually or automatically scale your application across different geographies, and you’ll have a really interesting value proposition on your hands by incurring a lower cost than deploying a traditional IT infrastructure.

Accelerate AI Deployments with IBM Cloud Pak for Data System


Which cloud provider should you choose for multiple geographies?
Choose a cloud provider that enables you to both deploy and scale your application infrastructure across multiple geographies throughout the world to ensure that your audience has a fast and responsive experience while using your application.


Create and use REST-based web services

As you can see, deploying your application code in the cloud provides many benefits — and one crucial benefit for high-demand applications is the ability to scale out the web and database tiers independently. That being said, it is also good practice to separate your business logic into web services that your front-end code can consume. Use this practice to scale out the web services tier independently from both the database and the front-end code. Separating your application logic from the presentation tier opens new doors for technologies that you might not have considered in the past, such as creating a single-page application using a language like Node.

Accelerate Digital Transformation with IBM Cloud Private


Implement continuous delivery and integration

DevOps seems to be the latest buzzword that is gaining a lot of attraction across enterprises. To get ahead, you should probably start looking at and implementing both continuous integration and delivery on your next software project. When deploying applications to a cloud-based infrastructure, make sure you have workflows in place on your existing build system so that code can be deployed across the different environments. Fortunately, most of the more popular build systems provide plugins for some of the top cloud providers today, making it easy to configure your deployment rules based upon the correct permissions of who has access to deploy code to each environment. If you are not currently using a build system for your development team, start using one now!

Hybrid Cloud Explained


Which cloud provider should you choose for continuous integration and delivery?

Choose a cloud provider that meets all of the requirements above with the added feature of integrated continuous integration and continuous delivery (CI/CD) tools on the platform. The provider you choose should allow you to deploy your own build system or have the ability to easily integrate with existing systems that reside outside of the cloud platform.

Avoid vendor lock-in

If you take one thing away from this article, I hope this is it: While many cloud providers provide great-looking proprietary APIs that reduce the amount of code or work that you have to do, you should avoid them at all costs. This is nothing more than a simple ploy to get you locked into their ecosystem while making it extremely hard to move your application to another provider or to your own data center running in-house. To avoid these custom APIs, stick with tried-and-true technology stacks across your application, including the database tier, storage tier, and any micro service endpoints that you might want to create. While the up-front investment can be a bit higher than using a proprietary solution out of the box, your technical debt is greatly reduced, which can save you money and time in the long run.

Public, Private & Hybrid Clouds


Develop locally or in the cloud

As developers, we often code applications on our local system and then, when we reach a work milestone, we move our code to the team’s development environment. Most developers wish they could develop on a daily basis with an infrastructure that resembles production as closely as possible. That goal can often be challenging due to the system administration overhead incurred to provide each developer with a cluster of machines.

Now that PaaS is available, all developers should begin to develop and deploy their code in the cloud. Most integrated development environments (IDE) provide plugins to streamline the process and make it feel as close to developing locally as possible.

Which IDE should you choose?

Choose an IDE that provides a plugin for the cloud provider of your choice. Consider choosing a provider that provides the ability to hot deploy application code as well as the ability to enable remote debugging of your source code. Once you have selected a provider that offers these two things, you can continue to set break points inside of your IDE and step through code just as if you were deploying locally. This enables you to more quickly catch bugs that only appear when moving to a clustered environment.

IBM Cloud Direct Link provides fast, secure and reliable performance for hybrid workloads


What to look for in the coming years from cloud providers

This article focused on the current state of applications being deployed to the cloud. One thing to look for and consider this year and next is the mass-industry movement to container-based deployments — you have probably already heard about Docker and rocket containers. When selecting a cloud provider, make sure the roadmap for application migration to container-based deployments is called out clearly with a timeline that clearly defines your migration path. Also, be on the lookout for vendors that are sticking with industry-standard solutions around containers and orchestration, versus creating proprietary solutions.

Conclusion

Cloud computing has many benefits that you should take advantage of in your daily software development and deployment to make your software more stable, scalable, and secure. When moving applications to the cloud, consider the following guidance:

• For application development, choose PaaS. The infrastructure is managed by a vendor, which gives you more time to focus on your application code.
• For application development, choose a platform enabled for both manual and automatic horizontal scaling of your application.
• For green-field applications, design apps that are stateless.
• For legacy or existing applications, choose a PaaS provider that supports both stateful and stateless applications.
• Choose a database that is scalable and located on a separate server or container from your application code. Then you can scale the database independently.
• Choose a cloud provider that enables you to both deploy and scale your application infrastructure across multiple geographies throughout the world.
• Develop using REST-based web services.
• Choose a cloud provider that meets all of the previous requirements with the added feature of integrated continuous integrations and continuous delivery tools in the platform.
• Avoid being locked in by a vendor.

The evolving hybrid integration reference architecture

How to ensure your integration landscape keeps pace with digital transformation



Over time, integration inevitably increases in complexity. This complexity is a result of the greater diversity of resources that we need to integrate, in ever-increasing permutations of infrastructures and platforms. Furthermore, the people who are involved in integrating systems are no longer centralized in a single technical team, but are spread throughout and beyond the enterprise.

In parallel, and in part as a result of this increase in complexity, a competing drive aims to simplify and rationalize integration. Web APIs have matured to become a common platform and language-agnostic way for applications to communicate. Infrastructure is ever more virtualized and containerized to free run times from hardware and operating system specifics and enable elastic workload orchestration. Teams are learning to more effectively join their development and operations staff together and automate from build to deployment for rapid release cycles.

IBM Cloud Private


The challenges are truly hybrid in many different senses of the word. Locations, teams, methods, platforms, and more are continuously diverging. The architecture of the integration across this hybrid environment is evolving at a rapid pace.

We explore how hybrid integration has evolved. First, we examine how the scope of integration has changed with the move to cloud. Next, we define the high-level characteristics of a hybrid integration capability. Then, we explain how IT is less often a central function within the organization. We continue by looking at the fundamental building blocks of a hybrid integration architecture and how integration can be productized though the API economy. Finally, we highlight how you can recognize and satisfy the needs of the digital team and improve consistency across the hybrid environment.


The extended surface area of hybrid integration
Today, the ownership boundary of an enterprise spreads well beyond its walls, encompassing IT assets across a truly hybrid environment. In this architecture, existing applications are moved to the infrastructure as a service (IaaS) of cloud providers. New applications are often built on the cloud as a platform as a service (PaaS). In line with this trend, a surge is also taking place in the use of pre-built cloud-based software as a service (SaaS).

In addition, interaction with external parties, whether customers or business partners, is increasingly automated. The degree of automation with those external parties is often a business differentiator.
Extended surface area of hybrid integration

Run Analytics Faster with IBM Integrated Analytics System


As such, any integration capability must fundamentally address connectivity across cloud boundaries. This capability must also simplify the security and management issues that it brings and embrace the standards that are evolving within hybrid architectures.

Hybrid integration is often simplistically defined as the ability to integrate on-premises systems with cloud systems. The reality for most organizations has much broader dimensions. A true hybrid integration architecture considers integration between all the owned environments, spanning on-premises and cloud environments, and whether that cloud is local, dedicated, or public. It also spans from a self-built environment to platforms to SaaS. It also must factor how the enterprise connects with its partners and customers. Hybrid integration has a vast scope. The key challenge is how to interpret that complexity and find common architectural patterns within it to help simplify the problem.

Assembling your cloud orchestra: A field guide to multi-cloud management




Hybrid integration core capabilities
At a high level, hybrid integration is a broad integration framework. It seamlessly bridges all owned environments, whether direct data sources, applications, or APIs, and can connect to them wherever they might be: on-premises, IaaS, PaaS, or SaaS.
Capabilities and scope of hybrid integration

Hybrid integration must contain broad connectivity capabilities for modern cloud-based applications and to equally critical, but older systems of record (SOR). It must have tools to simplify and accelerate productivity, such as flexible and fast mapping and transformation. From a non-functional perspective, it must also provide enterprise-grade and Internet-grade scalability, security, and resilience.

IBM Public Cloud for Power

However, although it is important to define hybrid integration as a whole, we must consider how integration needs are changing and recognize the two different audiences that are involved.


Adoption of IT by the line of business
For some years now, particularly accelerated by the mobile revolution, centralized IT has diverged into at least two different camps: often termed enterprise IT and digital IT.

Enterprise IT retains its vital role of ensuring that business-critical systems maintain their service levels and provide the type of data integrity and secure governance that are expected of core systems on which the business depends. However, this heavily regulated and controlled environment does not meet the needs of the lines of business (LOBs) who must keep the public face of the enterprise fresh with new propositions and innovations in a rapidly changing market.

LOBs have an increasing focus on requirements, such as the following examples:

Agility. LOBs need to explore and adapt, rapidly iterating over prototypes, and where necessary, failing fast and moving on to the next idea. This approach implies the use of different techniques to build more componentized applications, such as a microservices architecture. It also means increasing focus on a DevOps culture, where the distance between implementation changes and production delivery is minimized.

Scalability.If a prototype is launched, LOBs must be able to elastically scale it, moving a prototype from a handful of sponsored users to the open market, without significant additional development effort. LOBs must be able to scale the infrastructure up and down by using a simple and predictable cost model, implying use of IaaS or PaaS, and by adhering to an inherently scalable application architecture.

Latency. LOBs might also have different needs regarding the real-time responsiveness of the applications they provide. The latency that is required to create an engaging mobile experience might be significantly lower than the latency that is provided by existing systems of record. The latency might require using pre-aggregated or cached data and designing around the challenging issues of data duplication and eventual consistency. It might also require adequate traffic management to prioritize workloads.

Availability.The reputation of a primarily online business can be irreversibly damaged by even the smallest amount of downtime at the wrong moment. Proven availability is a differentiator. LOBs need applications that are designed for continuous high availability. These applications often require different approaches to resilience than you might choose for internal enterprise applications.

IBM Cloud Private

Given these requirements, you can see how shadow IT departments have arisen within LOBs and are now taking on significant new projects by using techniques that are different from the techniques that are used by central IT. These LOB IT departments are quickly moving out of the shadows and becoming recognized as the digital IT team who is responsible for creating the next generation of applications. This team often has a different culture and focus, recruiting business minded people with technical skills rather than raw technical specialists. As such, the team’s focus is on gaining revenue and market share by using rapid innovation. However, the applications that these individuals are creating represent the public face of the company. Therefore, they also require a solid technical backbone to satisfy core non-functional requirements.

Both enterprise IT and digital IT teams have integration needs. They need to integrate both within their own domains and with one another. At a high level, those hybrid integration capabilities sound similar, such as connectivity, transformation, API exposure, and composition. However, they differ dramatically in their details.

More Information:

https://developer.ibm.com/depmodels/hybrid/

https://developer.ibm.com/depmodels/cloud/

https://developer.ibm.com/tutorials/category/cloud/

https://developer.ibm.com/components/cloud-foundry/

https://developer.ibm.com/components/cloud-private/

https://www.ibm.com/it-infrastructure/z/capabilities/hybrid-cloud

https://slideplayer.com/slide/12091649/

https://www.infoq.com/news/2017/11/ibm-cloud-private/

https://www.ibm.com/us-en/marketplace/disaster-recovery-orchestration

https://www.ibm.com/cloud/learn/cloud-computing

Earlier this month, Oracle announced the availability of the latest Exadata machine, the X8-2. As well as the option to swap in big 14 TB disks in older versions of Exadata that were shipped with far smaller disks, allowing these older systems to substantially increase there capacity.

The changes in specifications compared to X7 are not huge. The most eye catching change is the size of the disks: from 10 TB to 14 TB. Considering the fact that licence costs for Exadata are calculated based on number of disks, this means that for the same money, the capacity of Exadata is substantially increased.

Oracle engineered systems executive presentation


All In on Exadata

Focusing on some details:

Compared to X7, the database nodes have a new CPU. It has the same number of cores (2x 24 per  node)  but a somewhat higher clockspeed: 2.4GHz vs. 2.1GHz in X7. The size of the memory has not changed: standard 384 GB per server with a maximum of 1.5 TB.

The inifiniband setup has stayed the same.

As mentioned, things have changed regarding the storage nodes. The disk size went up from 10 TB to 14 TB. The number of cores on the Storage Node went to 2x 16 per node, up from 2x 10 in X7. A slightly higher clockspeed: 2.3 GHz vs 2.2 GHz in X7.  Memory and Flash cards stayed the same at 192 GB and 4x 6.4 TB respectively per storage node. All in all, the number number of IOPS is only slightly increased compared to X7.

A third type of storage node has been introduced: Storage Server XT. This is a storage node without flash cards and with less memory and only a single CPU. This type of node is intended for long term data storage for archiving purposes. It will cost considerably less than the regular storage nodes.

Check out this data sheet for the full specification overview: https://www.oracle.com/a/ocom/docs/engineered-systems/exadata/exadata-x8-2-ds.pdf




Disk Swap in older Exadata models

Michael points out a very interesting new option announced by Oracle along with Exadata X8-2: the option to swap the disks in older Exadata models to the new 14 TB disks. This allows customers to replace the disks in the storage nodes in their Exadata X4, X5, X6 and X7 models (not for the models prior to X4) with the new 14 TB disks. When the capacity of the current Exadata system is not enough for today’s needs, swapping disks is far cheaper than adding storage nodes. Adding new storage nodes is not attractive, for several reasons: the increased license costs is the main one and additionally you will probably not be able to use all space on the newly added disks. This is because ASM slices data across the same size on all disks; the smallest disk determines the space used on all disks, including far larger disks. Swapping disks allows you to have bigger disks in all storage nodes, leveraging disks (and the associated license costs) to the fullest. And you get to keep your old disks after swapping in the ones. With a little operation, these disks can be turned into standard hard disks that can be used in any standard rack.



As a little telling example: assume an existing Exadata X4 system – a quarter rack. It has three storage nodes with a total of 36 x 4 TB diskspace or 144 TB in totaal. After swapping the disks, this goes up to 504 TB in total. over three times as much. Without increased license costs – you only pay for the new disks.

The swap of the disks can be performed on line – no downtime – when certain conditions are met. This requires the latest release of the Exadata software, 19.2.0.0 in order to work with the firmware for the new 14TB disks. More information can be found in Oracle Support site note 1544637.1: https://support.oracle.com/epmos/faces/DocumentDisplay?id=1544637.1



Summary of Eaxadata 8 Capabilities and Use Cases

The Oracle Exadata Database Machine (Exadata[1]) is a computing platform that is specialized and optimized for running Oracle Database. The goal of Exadata[2] is to achieve higher performance and availability at lower cost by moving database algorithms and intelligence into storage and networking, bypassing the traditional processing layers.[3]

Exadata X8-2 Full Rack

Exadata is a combined hardware and software platform that includes scale-out compute servers, scale-out intelligent storage servers, ultra-fast InfiniBand networking, ultra fast NVMe flash, and specialized Exadata Software[4] in a wide range of shapes and price points. Exadata Storage uses high-performance servers to store data and run Exadata Software to run data-intensive database processing directly in the shared storage tier.

Exadata debuted[5] in 2008 as the first in Oracle Corporation's family of Engineered Systems[6] for use in corporate data centers deployed as "private clouds". In October 2015, Exadata became available in the Oracle Cloud as a subscription service, known as the Exadata Cloud Service.[7]

Oracle MAA Best Practices - Applications Considerations


[8] Oracle databases deployed in the Exadata Cloud Service are 100% compatible with databases deployed on Exadata on-premises, which enables customers to transition to the Oracle Cloud with no application changes. Oracle Corporation manages this service, including hardware, network, Linux software and Exadata software, while customers have complete ownership of their databases.

In early 2017, a third Exadata deployment choice became available. Exadata Cloud at Customer[9] is Exadata Cloud Service technology deployed on-premises (behind the corporate firewall) and managed by Oracle Cloud experts. Like the Exadata Cloud Service, Exadata Cloud at Customer is owned and managed by Oracle, and licensed through a pay-as-you-go subscription. The Oracle Cloud at Customer[10] program is intended to bring all the benefits of the Oracle public cloud while still satisfying security and regulatory constraints.

Exadata Use Cases

Exadata is designed to optimally run any Oracle Database workload or combination of workloads, such as an OLTP application running simultaneously with Analytics processing. The platform is frequently used to consolidate many databases that were previously running on dedicated database servers. Exadata's scale-out architecture is naturally suited to running in the Oracle Cloud, where computing requirements can dynamically grow and sometimes shrink.

Historically, specialized database computing platforms were designed for a particular workload, such as Data Warehousing, and poor or unusable for other workloads, such as OLTP. Exadata has optimizations for all database workloads, implemented such that mixed workloads share system resources fairly. Resource management features also allow for prioritized allocation of system resources, such as always favoring workloads servicing interactive users over reporting and batch, even if they are accessing the same data.

Long running requests, characterized by Data Warehouses, reports, batch jobs and Analytics, are reputed to run many times faster compared to a conventional, non-Exadata database server.[11][12] Customer references often cite performance gains of 10x or greater. Analytics workloads can also use the Oracle Database In-Memory[13] option on Exadata for additional acceleration, and In-Memory Databases on Exadata have been extended to take advantage of Flash memory capacity, many times larger than the capacity of DRAM. Exadata’s Hybrid Columnar Compression[4] feature is intended to reduce the storage consumption of Data Warehouses and archival data as well as increase performance by reducing the amount of IO.

Transactional (OLTP) workloads on Exadata benefit from the incorporation of Flash memory into Exadata’s storage hierarchy, and the automatic "tiering" of data into memory, Flash or disk storage. Special Flash algorithms optimize Flash for response time sensitive database operations such as log writes. For the most demanding OLTP, all-Flash storage eliminates the latency of disk media completely.

What's New in Oracle Exadata 19.1 and Oracle Database 19c

Exadata Design Concepts

The hardware components that make up a typical database computing platform are a compute server connected over a network to a storage array. The database software runs on the compute server and sends or receives database information to and from the storage array over the network. The hardware components use standard software protocols to "talk" to each other. This separation via standard interfaces is what allows a computing platform to run a wide variety of software and hardware from different vendors. All of the application logic and the processing of the data is performed on the compute server, to which all the data must be sent. With this approach, a computing platform can be used for a wide range of software applications, though it will not be optimized for any particular application.

The goal of Exadata was to create a complete stack of software and hardware focused on the Oracle Database, that allowed processing to be moved to its optimal location. If Exadata is only processing Oracle Database requests it can take advantage of that in all the software layers. The hardware design can include elements that are most advantageous to Oracle Database applications, such as very fast InfiniBand networking and Flash memory. Given the importance of data storage to databases, Oracle was particularly focused on optimizing that aspect of the Exadata platform.

Why to Use an Oracle Database?


Oracle wanted a storage layer for Exadata that could easily scale out and parallelize Oracle Database requests. It also recognized the opportunity for storage to cooperate in the processing of database requests beyond just storing and shipping data. For example, rather than send an entire database table across the network to the compute server to find a small number of records, such data filtering could be done in storage and only the resulting records sent across the network. The addition of Flash memory to Exadata Storage Servers also opened up a range of possibilities for optimizing performance in the storage layer. Over time, as the performance and capacity of Flash storage increased at a rapid rate, the network became a performance bottleneck for traditional database platforms and Exadata's offloading of database processing into Exadata Storage Servers avoided that problem.

The foundation of Exadata is the Exadata Storage Server[14][1], invented by Oracle to replace the traditional storage array. Also important is Oracle's ownership of all the main software and hardware components of Exadata, enabling changes to be deeply integrated and released in coordinated fashion. A further benefit for customers is the ability to support the entire Exadata platform from one vendor.

Oracle Exadata X8 Overview

Software Enhancements

A more detailed listing of software enhancements is below, grouped by their value to Analytics or OLTP workloads, or their impact on Availability. Similar enhancements cannot be duplicated on other platforms because they require software and API modifications and integration across database software, operating systems, networking and storage.
Refer to the Exadata documentation[17] and Data Sheet[1] for descriptions of these features.

For ANALYTICS
•Automatically parallelize and offload data scans to storage
•Filter rows in storage based on 'where' clause
•Filter rows in storage based on columns selected
•JSON and XML offload
•Filter rows in storage based on join with other table
•Hybrid Columnar Compression
•Storage Index data skipping
•IO Resource Management by user, query, service, DB
•Automatic transformation to columnar format in Flash Cache
•Smart Flash Cache for table scans
•Offload index fast full scans
•Offload scans on encrypted data, with FIPS compliance
•Storage offload for LOBs and CLOBs
•Storage offload for min/max operations
•Data Mining offload
•All ports active InfiniBand messaging
•Reverse offload to DB servers if storage CPUs are busy
•Database In-Memory automatic memory population/depopulation
•In-Memory support for external tables
•In-Memory optimized arithmetic
•Automatics statistic and indexing


Innovating With The Oracle Platform for Data Science and Big Data


For AVAILABILITY
•Instant detection of node or cell failure
•In-Memory Fault Tolerance
•Sub-second failover of IO on stuck disk or flash
•Offload backups to storage servers
•Exadata data validation (H.A.R.D.)
•Instant data file creation
•Prioritize rebalance of critical files
•Automatic hard disk scrub and repair
•Power cycle failed drives to eliminate false drive failures
•Avoid reading predictive failed disks
•Cell software transparent restart
•Flash and disk life cycle management alert
•Confinement of temporarily poor performing drives
•Prevent shutdown if mirror server is down
•Automatic Software Updates on an entire "fleet" of Exadata systems with one operation
•Hot pluggable Flash cards
•Keep standby database consistent when NO FORCE logging is used
•Fast, secure eraser of disk and Flash
•Advanced Intrusion Detection Environment (AIDE) detects and alerts when unknown
                changes to system software are made
•Automatic monitoring of CPU, network and memory using Machine Learning

Oracle Database Availability & Scalability Across Versions & Editions


For OLTP
•Database Aware PCI Flash
•Exadata Smart Flash Logging
•Write-back Flash Cache
•IO Prioritization by DB, user, or workload to ensure QoS
•Direct-to-Wire Protocol
•Network Resource Management
•EXAchk full-stack validation
•Full-stack security scanning
•NVMe flash interface for lowest latency IO
•Active AWR includes storage stats for end to end monitoring
•Database scoped security
•Cell-to-cell rebalance preserving flash cache
•In-Memory commit cache
•Memory optimized OLTP and IoT lookups
•Automatics statistic and indexing


Oracle RAC 19c - the Basis for the Autonomous Database


A Quick Introduction to Oracle Exadata X8

Database Software

Exadata compute servers run the Oracle Linux 7.6 operating system and Oracle Database 11g Release 2 Enterprise Edition through Oracle Database 19c Enterprise Edition. Exadata system resources can be optionally virtualized using the Xen-based Oracle VM. All Oracle Database options, such as Real Application Clusters, Multitenant, Database In-Memory, Advanced Compression, Advanced Security, Partitioning, Active Data Guard and others are optionally available with Exadata. Applications that are certified to a supported version of the Oracle Database are automatically compatible with Exadata. No additional modifications or certifications are required[18].

The same database software that runs on Exadata on-premises will run in the Exadata Cloud Service and Exadata Cloud at Customer. In addition, on-premises software licenses are eligible for the BYOL[19] (Bring Your Own License) transfer into the Oracle Cloud or Cloud at Customer.

Exadata X8 - Was ist neu?

Networking

Exadata provides high-speed networks for internal and external connectivity. A 40 gigabits per second (40 Gbit/s) InfiniBand network is used for internal connectivity between compute and storage servers and 25, 10 and 1 Gbit/s Ethernet ports are included for data center connectivity. The InfiniBand network is also used as the cluster interconnect between compute servers.

Exadata uses a custom-designed, database-oriented protocol over the InfiniBand network to achieve higher performance. It makes extensive use of remote direct memory access (RDMA) to improve efficiency by avoiding data copies when moving data between servers. Exadata also has a direct-to-wire protocol[20] that allows the database to "talk" directly to the InfiniBand hardware, bypassing the operating system.

Exadata also takes advantage of InfiniBand Lanes[21] in its Network Resource Management[15] feature to prioritize important traffic across the network. In this feature the Oracle Database software tags network messages that require low latency, such as transaction commits, lock messages and IO operations issued by interactive users, enabling them to bypass messages issued by less critical high-throughput workloads such as reporting and batch. The result is analogous to how an emergency vehicle with its siren on is able to move more quickly through heavy traffic - high-priority network messages are moved to the front of the server, network switch, and storage queues, bypassing lower-priority messages and resulting in shorter and more predictable response times.

Best practices to maximize the ROI on your Oracle Exadata investment.

Database Server Components of Oracle Exadata Database Machine X8-8

Oracle Exadata Database Machine X8-8 database servers include the following components:

8x 24-core Intel(R) Xeon(R) Platinum 8268 Processors (2.9GHz)
3TB (48 x 64 GB) RAM, expandable to 6 TB (96 x 64 GB) with memory expansion kit
2 x 6.4TB flash accelerator PCIe cards (Hot-Pluggable)
8 x InfiniBand 4X QDR (40 Gbps) ports (PCIe 3.0) - all ports active
8x 1/10 GbE Base-T Ethernet ports (8 embedded ports based on the Intel 722 1/10GbE Controller)
8x 10GbE/25GbE Ethernet SFP28 Ports (4 Dual-port 10/25 GbE PCIe 3.0 network card based on the Broadcom BCM57414 10Gb/25Gb Ethernet Controller technology)
1 Ethernet port for Integrated Lights Out Manager (ILOM) for remote management
Redundant hot swappable power supplies and fans

Oracle Big Data Architecture

Evolution of Exadata

Oracle Corporation releases a new generation of Exadata every twelve to eighteen months[39][40][41][42][43][44][45][46][47]. At each release, Oracle refreshes most hardware components to the latest Intel Xeon processors, memory, disk, flash and networking. The hardware refreshes in themselves result in performance increases with every release. Exadata software is also refreshed with each generation and periodically in between, enhancing some combination of performance, availability, security, management and workload consolidation. In October 2015, features to support the Oracle Cloud were introduced[48].

The emphasis of each Exadata generation is described below.

Exadata V1[39], released in 2008, focused on accelerating Data Warehousing by delivering the full throughput of storage to the database. Per Oracle, Exadata achieved this by moving database filtering operations into storage, instead of sending all data to the compute servers and filtering it there. Oracle refers to this capability as Exadata Smart Scan[49][50]. Exadata V1 also supported a consolidation feature for allocating IO bandwidth between databases or workloads, called IORM (IO Resource Manager)[51].

Exadata V1 was available in Full Rack or Half Rack sizes, and the choice of High Performance or High Capacity storage servers.

Exadata V2[40][52][53], released in 2009, added a Quarter Rack configuration and support for OLTP workloads via Flash storage and database-aware Flash Caching.[54]

Exadata V2 also introduced Hybrid Columnar Compression[55] to reduce the amount of storage consumed by large Data Warehousing tables.

Storage Indexes[56] in Exadata V2 increased performance by eliminating the need to read entire regions of storage, based on knowledge of the data contained in the region.

Exadata X2-2[26], the third generation, was released in 2010 and a second model of Exadata, Exadata X2-8[41], was introduced. The X2-8 and subsequent “8 socket” Exadata models feature Intel processors targeted at large memory, scale-up workloads. The use of Flash storage beyond caching began in this release with a Smart Flash Logging[57][58] feature. Support for 10 Gigabit per second (Gb/sec) Ethernet connectivity was also added.

Data security through encryption was encouraged with the incorporation of hardware decryption[59] in Exadata X2-2, largely eliminating the performance overhead compared to software decryption.

A Storage Expansion Rack[60] based on Exadata X2-2 was added in 2011 to accommodate large, fast-growing Data Warehouses and archival databases. All subsequent 2-socket Exadata generations have included a new Storage Expansion Rack.

Exadata X3-2[42][61][27] and X3-8[33] were released in 2012, including a new Eighth Rack X3-2 entry-level configuration. Flash storage capacity quadrupled and OLTP write throughput reportedly increased by 20x via the Write-Back Flash Cache[62] feature.

A number of availability enhancements were added, bypassing slow or failed storage media[63], reducing the duration of storage server brownouts and simplifying replacement of failed disks.

Exadata X4-2[43][28] was released in 2013. Flash capacity doubled and Flash compression was added, effectively doubling capacity again. Network Resource Management[15] was introduced, automatically prioritizing critical messages. InfiniBand bandwidth doubled with support for active/active connections.

Exadata X4-8[34] released in 2014, plus Capacity on Demand[64] licensing, IO latency capping and timeout thresholds.

Exadata X5-2[44][29] and X5-8[35] were released in 2015 with a major set of enhancements. Flash and disk capacity doubled. Elastic configurations[65] were introduced to enable expansion one server at a time. Virtualization was added as an option to Exadata along with Trusted Partitions[66] for flexible licensing within a virtual machine. Database snapshots[67] on Exadata storage enabled efficient development and testing. Oracle Database In-Memory on Exadata included Fault Tolerant[68][69]redundancy. The High Performance Exadata storage servers were replaced with all-Flash (Extreme Performance) storage servers and Exadata became the first major vendor to adopt the NVMe Flash interface. Columnar Flash cache was introduced to automatically reformat analytics data into row format in Flash. IPv6 support was completed. Exadata Cloud Service[48][70] was launched on the Oracle Cloud.

Exadata X6-2[45][30] and X6-8[36] were released in 2016. Flash capacity doubled. Exafusion Direct-to-Wire protocol[71] reduced messaging overhead in a cluster and Smart Fusion Block Transfer[72] eliminated log write delays for OLTP applications in a cluster. Exadata Cloud at Customer[73][9] debuted, enabling Oracle Cloud benefits within corporate data centers.

Exadata X7-2[46] and X7-8 were released in 2017[74]. Flash capacity doubled. Flash cards became Hot pluggable for online replacement. 10 Terabyte (TB) disk drives debuted along with 25 Gb/sec Ethernet connectivity. Oracle Database In-Memory processing was extended into Flash storage, and storage server DRAM was utilized for faster OLTP.

Imagery Data with Oracle Spatial Technologies Karin Patenge


Exadata X8-2 [47]and X8-8 were released in 2019. Exadata Storage Server Extended (XT) was introduced for low-cost storage of infrequently accessed data.14 Terabyte (TB) disk drives debuted along with 60% more compute cores in Exadata storage servers. Machine Learning algorithms were added to automatically monitor CPU, network and memory to detect anomalies such as stuck processes, memory leaks and flaky networks, and to automatically create (Auto index), rebuild or drop indexes. Optimizer statistics are also gathered in real-time as DML executes. For enhanced security, Advanced Intrusion Detection Environment (AIDE) was added to detect and alert when unknown changes to system software are made.

Easily Migrate your IT Environments to Oracle Cloud Infrastructure using RackWare

More Information:

https://www.oracle.com/technetwork/database/exadata/exadata-x8-8-ds-5444364.pdf

https://www.oracle.com/technetwork/database/exadata/exadata-x8-2-ds-5444350.pdf

https://www.oracle.com/engineered-systems/exadata/database-machine-x8/

https://www.oracle.com/technetwork/database/exadata/exadata-x7-2-ds-3908482.pdf

https://www.oracle.com/technetwork/database/exadata/exadata-technical-whitepaper-134575.pdf

https://docs.oracle.com/en/engineered-systems/exadata-database-machine/dbmso/hardware-components-exadata-db-machine.html#GUID-2954CDC1-0630-4A4F-8F1A-B0CDB039199C

https://docs.oracle.com/en/engineered-systems/exadata-database-machine/dbmso/lot.html

https://www.oracle.com/technetwork/database/exadata/overview/index.html

https://www.oracle.com/a/ocom/docs/engineered-systems/exadata/ai-transforms-oracle-exadata-x8.pdf

https://www.oracle.com/a/ocom/docs/engineered-systems/exadata/exadata-x8-reduces-need-for-hadoop-spark.pdf

https://www.oracle.com/corporate/pressrelease/oracle-exadata-builds-in-machine-learning-061219.html

https://technology.amis.nl/2019/04/20/newly-released-oracle-exadata-x8-2-bigger-disks-for-saving-money-expanding-capacity/

IBM and Oracle Join Forces

In early July, IBM and Red Hat officially closed their most significant acquisition of 2019–an important milestone combining the power and flexibility of Red Hat’s open hybrid portfolio and IBM’s technology and deep industry expertise.

IBM Oracle International Competency Center



The feedback from our clients and partners is clear. A recent IBM report found that 80 percent want solutions that support hybrid cloud, including containers and orchestration. Today IBM is announcing plans to bring Red Hat OpenShift and IBM Cloud Paks to the IBM Z and LinuxONE enterprise platforms*. Together these two platforms power about 30 billion transactions a day globally. Our goal is for you to harness the scalability and security of IBM Z and LinuxONE alongside the flexibility to run, build, manage and modernize cloud-native workloads on your choice of architecture.

Cloud Impact Assessment for Oracle



For more than 20 years, IBM Systems and Red Hat have worked to drive open source systems innovation, make Linux enterprise-grade, and help joint customers like MetOffice and Techcombank with mission-critical workloads to build, deploy and manage next-gen apps and protect data through advanced security.

Today, IBM supports Red Hat Enterprise Linux on IBM Power Systems, IBM Z and LinuxONE, as well as Red Hat OpenShift on IBM POWER. IBM will also support Red Hat OpenShift and Red Hat OpenShift Container Storage across IBM’s all-flash and software-defined storage portfolio.

The combination of Red Hat OpenShift with IBM Z and LinuxONE reflects our shared values: to provide a flexible, open, hybrid, multicloud and secured enterprise platform for mission-critical workloads.

Our goal for Red Hat OpenShift for IBM Z and LinuxONE will be to help clients enable greater agility and portability through integrated tooling and a feature-rich ecosystem for cloud-native development to:

• Deliver containerized applications that can scale vertically and horizontally;
• Accelerate deployment and orchestration of containers with Kubernetes;
• Help IT to support rapid business growth;
• Optimize workloads to take advantage of pervasive encryption; 

and

Increase container density that can make systems management easier which should help reduce total cost of ownership.

“Containers are the next generation of software-defined compute that enterprises will leverage to accelerate their digital transformation initiatives,” says Gary Chen, Research Director at IDC. “IDC forecasts that the worldwide container infrastructure software opportunity is growing at a 63.9% 5 year CAGR and is predicted to reach over $1.5B by 2022.”

How the Results of Summit and Sierra are Influencing Exascale


This is also exciting news for our business partners and ecosystems. This offering will provide an enterprise platform for hybrid multicloud that can help ISVs and others develop the next generation of applications. They will have the benefit of flexibility of a microservices-based architecture to deploy containers on infrastructure of their choice. For ISVs not currently running on Red Hat Enterprise Linux on IBM Z and LinuxONE enterprise platforms, this is an opportunity to bring your software to these two platforms, where the most critical workloads are run.

For more information, please visit www.ibm.com/linuxone and www.ibm.com/redhat, or contact your local IBM sales representative.

[1] (IBM Sponsored Primary Research, MD&I Systems and Cloud NDB 2019)

[2] https://www-03.ibm.com/press/uk/en/pressrelease/52824.wss


New Oracle Exadata X8M PMEM and RoCE capabilities and benefits


IBM LinuxONE servers running Oracle Database 12c on Linux

Today, enterprises require a trusted IT infrastructure that is dynamic, scalable
and flexible enough to support both mission-critical work and the development
and deployment of new workloads. This infrastructure must help decision makers
to use their company’s most valuable asset—their data—with insight rather than
hindsight, and it must assist in using IT to gain a competitive edge.

IBM® LinuxONE™ is a family of systems designed for more-secure data serving.
Expect advanced performance, security, resiliency, availability and virtualization
for a high quality of service. Ideal for larger enterprises that are embracing digital
business, the dual-frame LinuxONE Emperor II™ offers massive scalability in a
high-volume transaction processing and large-scale consolidation platform.

Meet the new IBM z15


LinuxONE is exceptionally good for deploying Oracle data-serving workloads
The Emperor II delivers outstanding transaction processing and data serving
performance for excellent economies of scale and more-efficient use of critical
data. With up to 170 LinuxONE cores, up to 32 TB of memory, and simultaneous
multithreading (SMT) support, the Emperor II is ideally suited for consolidating
large-scale distributed environments and for new in-memory and Java™ workloads.

New IBM z15 Up & Running at IBM Systems Center Montpellier.


The IBM LinuxONE Rockhopper II™, the newest member of the LinuxONE
family, is a single-frame system in a 19-inch industry standard rack allowing it
to sit side-by-side with any other platform in a data center. The Rockhopper II
also supports SMT, up to 30 Linux cores and up to 8 TB of memory. This system
is ideal for any growing business that seeks to use LinuxONE technologies’
qualities of service, flexibility and performance.

IBM Data Privacy Passports on IBM z15

As environmental concerns raise the focus on energy consumption, the ASHRAE
A3 rated Emperor II and Rockhopper II systems promote energy efficiency.
Their design helps to dramatically reduce energy consumption and save floor
space by consolidating workloads into a simpler, more manageable and efficient
IT infrastructure.

Linux and IBM LinuxONE

A Linux infrastructure on LinuxONE provides an enterprise-grade Linux
environment. It combines the advantages of the LinuxONE hardware servers
and leading IBM z/VM® virtualization—along with the flexibility and open
standards of the Linux operating system.
IBM LinuxONE virtualization technology
During spikes in demand, the Emperor II and Rockhopper II systems can quickly
redistribute system resources and scale up, scale out, or both in a way that can
make the difference between flawless execution or costly, slow response times
and system crashes.

Inside the new IBM z15


You can further improve the virtualization management capabilities of Linux
and z/VM by using the intelligent visualization, simplified monitoring, and
unified management features of IBM Wave and IBM Dynamic Partition Manager.
These solutions are designed to help simplify everyday administrative and
configuration tasks and to help you transform your Linux environment to a
virtualized private cloud


The enterprise-grade Linux infrastructure on Emperor II and Rockhopper II is
designed to bring unique business value in the areas of operational efficiency,
scalability, autonomic workload management, reliability, business continuance
and security. Linux on LinuxONE solutions can further benefit from the following

IBM technologies to enhance this infrastructure:

• High availability capabilities are provided by the IBM Spectrum Scale™
high-performance data and file management solution based on the IBM
General Parallel File System (GPFS™). The Spectrum Scale solution is a
cluster file system that provides access to storage that can deliver greater
speed, flexibility, cost efficiency and security that are achievable by using
built-in encryption and data protection.

• IBM GDPS® Virtual Appliance provides near-continuous availability and
disaster recovery by extending GDPS capabilities for Linux guests on z/VM
environments. It can help substantially reduce recovery time, recovery
point objectives and the complexity that is associated with manual
disaster recovery.

Lecture 2: Introducing IBM Z Hardware & Operating Systems | z/OS Introduction


Oracle Database 12c and IBM LinuxONE

Oracle Database 12c, has a major focus on cloud and enables customers
to make more efficient use of their IT resources. Oracle Database 12c has
a new multitenant architecture, and includes several enhancements and new
features for:

• Consolidating multiple databases into multitenant containers
• Automatically optimizing data storage
• Providing continuous access with high availability features
• Securing enterprise data with a comprehensive defense-in-depth strategy
• Simplifying in-database analysis of big data

Introducing Oracle Gen 2 Exadata Cloud at Customer


Multitenant architecture
Oracle Multitenant delivers an architecture that simplifies consolidation and
delivers the high density of schema-based consolidation without requiring
changes to existing applications. This Oracle Database 12c option offers the
benefits of managing many databases as one, yet retains the isolation and
resource control of separate databases. In this architecture, a single multitenant
container database can host many “pluggable” databases, up to 4,096 pluggable
databases can run on a single container database. Each database consolidated
or “plugged in” to a multitenant container looks and feels to applications the
same as the other existing Oracle Databases and administrators can control the
prioritization of available resources between consolidated databases.

Database In-Memory
Oracle Database In-Memory uses a new dual-format in-memory architecture
that allows customers to improve the performance of online transaction
processing and enables analytics and data warehousing applications. The
dual-format architecture that allows simultaneous row and column format
in-memory enables existing applications to run transparently with better
performance without additional programming changes. New features such
as In-Memory Virtual Columns and In-Memory expressions can further
improve performance.

High availability
Basic high availability architectures using redundant resources can prove costly
and fall short of availability and service level expectations due to technological
limitations, complex integration, and inability to offer availability through planned
maintenance. Oracle Database 12c goes beyond the limitations of basic high
availability and with hardware features such as provided by IBM storage devices
and servers, offers customers a solution that can be deployed at minimal cost
and that addresses the common causes of unforeseen and planned downtime.

What is RDMA over Converged Ethernet (RoCE)?


Reducing planned downtime
Planned downtime for essential maintenance such as hardware upgrades,
software upgrades and patching are standard for every IT operation. Oracle
Database 12c offers a number of solutions to help customers reduce the
amount of planned downtime required for maintenance activities, including
these features of Oracle Database 12c and other Oracle offerings:

• Hardware Maintenance and Migration Operations to Oracle Database 12c
infrastructure can be performed without taking users offline.
• Online Patching of database software can be applied to server nodes in a
‘rolling’ manner using Oracle Real Application Clusters. Users are simply
migrated from one server to another; the server is quiesced from the cluster,
patched, and then put back online.
• Rolling Database Upgrades using Oracle Data Guard or Oracle Active
Data Guard enables upgrading of a standby database, testing of the
upgraded environment and then switching users to the new environment,
without any downtime.
• Online Redefinition can reduce maintenance downtime by allowing changes
to a table structure while continuing to support an online production system.
• Edition Based Redefinition enables online application upgrades. With
edition-based redefinition, changes to program code can be made in the
privacy of a new edition within the database, separated from the current
production edition.
• Data Guard Far Sync provides zero data loss protection for a production
database by maintaining a synchronized standby database at any distance
from the primary location.
• Global Data Services provides inter-region and intra-region load balancing
across Active Data Guard and Golden Gate replicated databases. This
service effectively provides Real Application Cluster failover and load balancing
capabilities to Active Data Guard and Golden Gate distributed databases.

RSOCKETS - RDMA for Dummies

Simplifying Analysis of Big Data

Oracle Database 12c fully supports a wide range of Business Intelligence tools
that take advantage of optimizations, including advanced indexing operations,
OLAP aggregations, automatic star query transformations, partitioning pruning
and parallelized database operations.

By providing a comprehensive set of integration tools, customers can use their
existing Oracle resources and skills to bring together big data sources into their
data warehouse. With this, customers can add to the existing Oracle Database
12c features, the ability to better analyze data throughout the enterprise.

Oracle’s stated goal is to help lower total cost of ownership (TCO) by delivering
customer requested product features, minimizing customizations and providing
pre-built integration to other Oracle solutions. These Oracle Database benefits
further complement the IT infrastructure TCO savings gained by implementing
Oracle Database on a LinuxONE server.

The enterprise-grade Linux on LinuxONE solution is designed to add
value to Oracle Database solutions, including the new functions that were
introduced in Oracle Database 12c. Oracle Database on LinuxONE includes
the following benefits:

• Provides high levels of security with the industry highest EAL5+ and
virtualization ratings, and high quality of service
• Optimizes performance by deploying powerful database hardware engines
that are available on Emperor II and Rockhopper II systems
• Achieves greater flexibility through the LinuxONE workload management
capability by allowing the Oracle Database environment to dynamically
adjust to user demand
• Reduces TCO by using the specialized LinuxONE cores that run the

OFI Overview 2019 Webinar



Oracle Database and management of the environment

Sizing and capacity planning for Oracle Database 12c on IBM LinuxONE
By working together, IBM and Oracle have developed a capacity-estimation
capability to aid in designing an optimal configuration for each specific
Oracle Database 12c client environment. You can obtain a detailed sizing
estimate that is customized for your environment from the IBM Digital
Techline Center, which is accessible through your IBM or IBM Business Partner
representative. You can download a questionnaire to start the sizing process at
ibm.com/partnerworld/wps/servlet/ContentHandler/techline/FAQ00000750

The IBM and Oracle alliance

Since 1986, Oracle and IBM have been providing clients with compelling
joint solutions, combining Oracle’s technology and application software with
IBM’s complementary hardware, software and services solutions. More than
100,000 joint clients benefit from the strength and stability of the Oracle and
IBM alliance. Through this partnership, Oracle and IBM offer technology,
applications, services and hardware solutions that are designed to mitigate
risk, boost efficiency and lower total cost of ownership.

IBM is a Platinum level partner in the Oracle Partner Network, delivering
the proven combination of industry insight, extensive real-world Oracle
applications experience, deep technical skills and high performance servers
and storage to create a complete business solution with a defined return
on investment. From application selection, purchase and implementation
to upgrade and maintenance, we help organizations reduce the total cost
of ownership and the complexity of managing their current and future
applications environment while building a solid base for business growth.

A Taste of Open Fabrics Interfaces



For more information

For more information about joint solutions from IBM and Oracle,
please contact an IBM sales representative at 1-866-426-9989.

For more information about IBM LinuxONE, see ibm.com/LinuxONE

For more information about Oracle Database 12c, visit
oracle.com/us/corporate/features/database-12c/index.html


IBM Oracle International Competency Center overview
CesarCantua | July 12 2012 | Tags:  international_competency_... icc | 10,988 Views


The IBM Oracle International Competency Center (ICC) works with our technical sales teams and business partners to provide technical pre-sales support for all Oracle Solutions. With locations across North America, including Foster City CA, Pleasanton CA, Redwoods Shores CA and Denver CO, the center supports all Oracle solutions including: Oracle Database, Oracle Fusion Middleware, Oracle E-Business Suite, Oracle Retail, PeopleSoft Enterprise, JD Edwards EnterpriseOne, JD Edwards World, and Siebel.

All ICC personnel work on-site at Oracle locations and include IBM hardware and software brand experts, technology managers, and solutions specialists. Working closely with the Advanced Technical Skills and Solutions Technical Sales teams, the ICC executes benchmarks and platform certifications and develops technical white papers and solution collateral.

The ICC is also responsible for the development and maintenance of the tools used worldwide to size Oracle applications on IBM hardware. Finally, The ICC offers individualized customer briefings tailored to a customer's unique requirements. These consultations demonstrate the close technology relationship between IBM and Oracle and help customers understand the hardware options and sizing implications of their Oracle solution implementation. The ICC-hosted customer briefings are a great tool to use with your prospects and clients. The briefings are tailored specifically around your client's area of interest. Don't miss the opportunity to demonstrate the depth of our business and technical relationship with Oracle!

Accelerating TensorFlow with RDMA for high-performance deep learning



For Oracle related sales technical questions, contact: ibmoracle@us.ibm.com

IBM Oracle International Competency Center (ICC). 
IBM Oracle Technical Quick Reference Guide.  
IBM Oracle International Competency Center (ICC) Mission.  

IBM, Oracle Join Forces For Midmarket Foray

IBM and Oracle are teaming to develop "affordable and easy to deploy and maintain" midmarket solutions based on Oracle's enterprise applications.The deal expands the companies' relationship in the enterprise. More specifically, IBM and Oracle will tailor Oracle's JD Edwards Enterprise One and the Oracle E-Business Suite, bundling them with Big Blue's hardware, software and services, for midsize companies. The first solutions are aimed at midsize industrial manufacturers in the United States and midsize food and beverage companies in Europe. Solutions for midsize clients in the life sciences and high-tech industries are expected out by the end of the year.

"We see this collaboration with IBM as a monumental step in our effort to provide midsize companies with enterprise-level functionality that is affordable and easy to deploy and maintain," said Oracle SVP Tony Kender in a statement.ChannelWeb

Why rivals Microsoft and Oracle are teaming up to take on Amazon

We’re living in a new digital age, defined by always-on connectivity, empowered customers, and groundbreaking IT solutions. In this new world, a business can’t thrive by relying on the same tired processes and disjointed IT systems they’ve used up until now.

5G Cellular D2D RDMA Clusters



At IBM, we’re dedicated to helping Oracle users upgrade their systems and processes to prepare for the age of Digital Transformation. In place of the complex multivendor platforms that are so common in business today, we offer streamlined, end-to-end solutions that include everything you need to optimize your Oracle applications. We offer our customers best-in-class solutions and services, deep experience across many different industries, and an intimate understanding of Oracle technology based on a decades-long partnership—all under the same roof.

Go to market quickly
We employ about 16,000 people who work directly with Oracle systems on a daily basis, across cloud and on-prem environments, making us one of the leading systems integrators in the world. This hands-on experience, combined with our wide breadth of systems and services offerings, allows us to ramp up new Oracle applications much quicker than our clients could working alone.

For example, we were able to help Epic Piping go from startup to enterprise-scale operation in a matter of months. With IBM Cloud Managed Services to support their Oracle JD Edwards Enterprise ERP solutions, Epic Piping gained a hugely scalable, fully integrated business platform that supports both organic growth and acquisitions. This new platform, along with application management services from IBM, played an instrumental role in helping Epic Piping achieve exponential growth. After starting with only four employees, the company topped out at over 900 just 18 months later.

Simplify processes to cut costs
Far too often, complexity forms a barrier that keeps businesses from fully capitalizing on the opportunities of Digital Transformation. When businesses run different sets of processes across different teams, it can create serious inefficiency, which in turn leads to higher costs.

This was the situation at Co-operative Group Limited when they came to IBM for help. The company wanted to move to a shared services model to increase the simplicity and efficiency of its HR function. However, after a period of rapid growth, including multiple acquisitions, Co-op’s HR policies had become so disjointed that it was simply not possible to get everyone using the shared services.
With the power of Oracle HCM Cloud solutions implemented by IBM Global Business Services, the company simplified and standardized its HR processes, increasing productivity and lowering HR costs. Co-op is also deploying IBM Watson solutions to collate and cleanse its old HR data. This will enable cognitive analytics, allowing the company to further optimize its HR services in the future.

Deep industry expertise
At IBM, our elite services professionals have a unique combination of Oracle experience and industry expertise, helping us drive even better results for users of Oracle’s industry-specific applications.

When Shop Direct, a multi-brand online retailer from the UK, wanted to introduce a new retail software platform based on Oracle applications, working with IBM to implement and manage those new applications was a natural choice. The company was already a long-time user of Oracle solutions, and ever since it first introduced Oracle E-Business Suite, it has been working with IBM to optimize those solutions. As a result, they already knew we understood their business, and that we’re intimately familiar with the needs of retailers in general.

The Z15 Enterprise Platform


Moving fast is absolutely critical in retail. By deploying the new Oracle applications quickly, and creating a centralized platform for accessing and managing product data, we were able to help Shop Direct ensure leaner, faster operations. As a result, they can now respond quickly to changing customer demand.

Learn More

Visit the IBM-Oracle Alliance website to learn more about how we can help you maximize your Oracle investments.
https://www.ibm.com/blogs/insights-on-business/oracle-consulting/ibm-oracle-global-alliance-optimizing-applications-digital-transformation/

More Information:

https://www.ibm.com/it-infrastructure/z/news

https://www.ibm.com/blogs/systems/announcing-our-direction-for-red-hat-openshift-for-ibm-z-and-linuxone/

https://www.ibm.com/blogs/systems/topics/servers/mainframes/

https://www.informationweek.com/mobile/ibm-oracle-join-forces-for-midmarket-foray/d/d-id/1066798?piddl_msgorder=

https://www.ibm.com/blogs/insights-on-business/oracle-consulting/ibm-oracle-global-alliance-optimizing-applications-digital-transformation/

https://www.ibm.com/services/oracle

https://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS2947

https://www.ibm.com/developerworks/community/blogs/fd4076f7-7d3d-4080-a198-e62d7bb263e8/entry/international_competency_center?lang=en

https://www.ibm.com/blogs/insights-on-business/oracle-consulting/ibm-oracle-global-alliance-optimizing-applications-digital-transformation/

What Google's Quantum Supremacy Claim Means for Quantum Computing

Leaked details about Google's quantum supremacy experiment stirred up a media frenzy about the next quantum computing milestone

Return to editingThe Limits of Quantum Computers





Google’s claim to have demonstrated quantum supremacy—one of the earliest and most hotly anticipated milestones on the long road toward practical quantum computing—was supposed to make its official debut in a prestigious science journal. Instead, an early leak of the research paper has sparked a frenzy of media coverage and some misinformed speculation about when quantum computers will be ready to crack the world’s computer security algorithms.

Google’s new Bristlecone processor brings it one step closer to quantum supremacy


The moment when quantum computing can seriously threaten to compromise the security of digital communications remains many years, if not decades, in the future. But the leaked draft of Google’s paper likely represents the first experimental proof of the long-held theoretical premise that quantum computers can outperform even the most powerful modern supercomputers on certain tasks, experts say. Such a demonstration of quantum supremacy is a long-awaited signpost showing researchers that they’re on the right path to the promised land of practical quantum computers.

“For those of us who work in quantum computing, the achievement of quantum supremacy is a huge and very welcome milestone,” says Scott Aaronson, a computer scientist and director of the Quantum Information Center at the University of Texas at Austin, who was not involved in Google’s research. “And it’s not a surprise—it’s something we all expected was coming in a matter of a couple of years at most.”

What Is Quantum Computing? 

Quantum computing harnesses the rules of quantum physics that hold sway over some of the smallest particles in the universe in order to build devices very different from today’s “classical” computer chips used in smartphones and laptops. Instead of classical computing’s binary bits of information that can only exist in one of two basic states, a quantum computer relies on quantum bits (qubits) that can exist in many different possible states. It’s a bit like having a classical computing coin that can only go “heads” or “tails” versus a quantum computing marble that can roll around and take on many different positions relative to its “heads” or “tails” hemispheres.

Because each qubit can hold many different states of information, multiple qubits connected through quantum entanglement hold the promise of speedily performing complex computing operations that might take thousands or millions of years on modern supercomputers. To build such quantum computers, some research labs have been using lasers and electric fields to trap and manipulate atoms as individual qubits.

Quantum Computing and Quantum Supremacy


Other groups such as the Google AI Quantum Lab led by John Martinis at the University of California, Santa Barbara, have been experimenting with qubits made of loops of superconducting metal. It’s this approach that enabled Google and its research collaborators to demonstrate quantum supremacy based on a 54-qubit array laid out in a flat, rectangular arrangement—although one qubit turned out defective and reduced the number of working qubits to 53. (Google did not respond to a request for comment.)

“For the past year or two, we had a very good idea that it was going to be the Google group, because they were the ones who were really explicitly targeting this goal in all their work,” Aaronson says. “They are also on the forefront of building the hardware.”

D Wave Webinar: A Machine of a Different Kind, Quantum Computing, 2019


Google’s Quantum Supremacy Experiment
Google’s experiment tested whether the company’s quantum computing device, named Sycamore, could correctly produce samples from a random quantum circuit—the equivalent of verifying the results from the quantum version of a random number generator. In this case, the quantum circuit consisted of a certain random sequence of single- and two-qubit logical operations, with up to 20 such operations (known as “gates”) randomly strung together.

The Sycamore quantum computing device sampled the random quantum circuit one million times in just three minutes and 20 seconds. When the team simulated the same quantum circuit on classical computers, it found that even the Summit supercomputer that is currently ranked as the most powerful in the world would require approximately 10,000 years to perform the same task.

“There are many in the classical computer community, who don't understand quantum theory, who have claimed that quantum computers are not more powerful than classical computers,” says Jonathan Dowling, a professor in theoretical physics and member of the Quantum Science and Technologies Group at Louisiana State University in Baton Rouge. “This experiment pokes a stick into their eyes.”



via GIPHY

“This is not the top of Mount Everest, but it’s certainly crossing a pretty big peak along the way.”
—Daniel Lidar, University of Southern California

In a twist that even Google probably didn’t see coming, a draft of the paper describing the company’s quantum supremacy experiment leaked early when someone—possibly a research collaborator at the NASA Ames Research Center—uploaded the paper to the NASA Technical Reports Server. It might have sat there unnoticed before being hastily removed, if not for Google’s own search engine algorithm, which plucked the paper from its obscure server and emailed it to Dowling and anyone else who had signed up for Google Scholar alerts related to quantum computing.

The random number generator experiment may seem like an arbitrary benchmark for quantum supremacy without much practical application. But Aaronson has recently proposed that such a random quantum circuit could become the basis of a certified randomness protocol that could prove very useful for certain cryptocurrencies and cryptographic protocols. Beyond this very specific application, he suggests that future quantum computing experiments could aim to perform a useful quantum simulation of complex systems such as those found in condensed matter physics.

Introduction to Quantum Computing


What’s Next for Quantum Computing?
Google’s apparent achievement doesn’t rule out the possibility of another research group developing a better classical computing algorithm that eventually solves the random number generator challenge faster than Google’s current quantum computing device. But even if that happens, quantum computing capabilities are expected to greatly outpace classical computing’s much more limited growth as time goes on.

“This horse race between classical computing and quantum computing is going to continue,” says Daniel Lidar, director of the Center for Quantum Information Science and Technology at the University of Southern California in Los Angeles. “Eventually though, because quantum computers that have sufficiently high fidelity components just scale better as far as we know—exponentially better for some problems—eventually it’s going to become impossible for classical computers to keep up.”

Google’s team has even coined a term to describe how quickly quantum computing could gain on classical computing: “Neven’s Law.” Unlike Moore’s Law that has predicted classical computing power will approximately double every two years—exponential growth—Neven’s Law describes how quantum computing seems to gain power far more rapidly through double exponential growth.

“If you’ve ever plotted a double exponential [on a graph], it looks like the line is zero and then you hit the corner of a box and you go straight up,” says Andrew Sornborger, a theoretical physicist who studies quantum computers at Los Alamos National Laboratory in New Mexico. “And so before and after, it’s not so much like an evolution, it’s more like an event—before you hit the corner and after you hit the corner.”

Quantum computing’s exponential growth advantage has the potential to transform certain areas of scientific research and real-world applications in the long run. For example, Sornborger anticipates being able to use future quantum computers to perform far more complex simulations that go well beyond anything that’s possible with today’s best supercomputers.

The Integration Algorithm A quantum computer could integrate a function in less computational time then a classical computer... The integral of a one dimensional.

Wanted: Quantum Error Correction
Another long-term expectation is that a practical, general-purpose quantum computer could someday crack the standard digital codes used to safeguard computer security and the Internet. That possibility triggered premature alarm bells from conspiracy theorists and at least one U.S. presidential candidate when news first broke about Google’s quantum supremacy experiment via the Financial Times. (The growing swirl of online speculation eventually prompted Junye Huang, a Ph.D. candidate at the National University of Singapore, to share a copy of the leaked Google paper on his Google Drive account.)

In fact, the U.S. government is already taking steps to prepare for the future possibility of practical quantum computing breaking modern cryptography standards. The U.S. National Institute of Standards and Technology has been overseeing a process that challenges cryptography researchers to develop and test quantum-resistant algorithms that can continue to keep global communications secure.

The moment when quantum computing can seriously threaten to compromise the security of digital communications remains many years, if not decades, in the future.

The apparent quantum supremacy achievement marks just the first of many steps necessary to develop practical quantum computers. The fragility of qubits makes it challenging to maintain specific quantum states over longer periods of time when performing computational operations. That means it’s far from easy to cobble together large arrays involving the thousands or even millions of qubits that will likely be necessary for practical, general-purpose quantum computing.

Quantum computing



Such huge qubit arrays will require error correction techniques that can detect and fix errors in the many individual qubits working together. A practical quantum computer will need to have full error correction and prove itself fault tolerant—immune to the errors in logical operations and qubit measurements—in order to truly unleash the power of quantum computing, Lidar says.

Many experts think the next big quantum computing milestone will be a successful demonstration of error correction in a quantum computing device that also achieves quantum supremacy. Google’s team is well-positioned to shoot for that goal given that its quantum computing architecture showcased in the latest experiment is built to accommodate "surface code” error correction. But it will almost certainly have plenty of company on the road ahead as many researchers look beyond quantum supremacy to the next milestones.

“You take one step at a time and you get to the top of Mount Everest,” Lidar says. “This is not the top of Mount Everest, but it’s certainly crossing a pretty big peak along the way.”

This could be the dawn of a new era in computing. Google has claimed that its quantum computer performed a calculation that would be practically impossible for even the best supercomputer – in other words, it has attained quantum supremacy.

If true, it is big news. Quantum computers have the potential to change the way we design new materials, work out logistics, build artificial intelligence and break encryption. That is why firms like Google, Intel and IBM – along with plenty of start-ups – have been racing to reach this crucial milestone.

The development at Google is, however, shrouded in intrigue. A paper containing details of the work was posted to a NASA server last week, before being quickly removed. Several media outlets reported on the rumours, but Google hasn’t commented on them.

Read more: Revealed: Google’s plan for quantum computer supremacy
A copy of the paper seen by New Scientist contains details of a quantum processor called Sycamore that contains 54 superconducting quantum bits, or qubits. It claims that Sycamore has achieved quantum supremacy. The paper identifies only one author: John Martinis at the University of California, Santa Barbara, who is known to have partnered with Google to build the hardware for a quantum computer.

“This dramatic speedup relative to all known classical algorithms provides an experimental realization of quantum supremacy on a computational task and heralds the advent of a much-anticipated computing paradigm,” the paper says.

Google appears to have partnered with NASA to help test its quantum computer. In 2018, the two organisations made an agreement to do this, so the news isn’t entirely unexpected.

Making an impossible universe with IBM's quantum processor

The paper describes how Google’s quantum processor tackled a random sampling problem – that is, checking that a set of numbers has a truly random distribution. This is very difficult for a traditional computer when there are a lot of numbers involved.

But Sycamore does things differently. Although one of its qubits didn’t work, the remaining 53 were quantum entangled with one another and used to generate a set of binary digits and check their distribution was truly random. The paper calculates the task would have taken Summit, the world’s best supercomputer, 10,000 years – but Sycamore did it in 3 minutes and 20 seconds.

This benchmarking task isn’t particularly useful beyond producing truly random numbers – it was a proof of concept. But in the future, the quantum chip may be useful in the fields of machine learning, materials science and chemistry, says the paper. For example, when trying to model a chemical reaction or visualise the ways a new molecule may connect to others, quantum computers can handle the vast amount of variables to create an accurate simulation.

“Google’s recent update on the achievement of quantum supremacy is a notable mile marker as we continue to advance the potential of quantum computing,” said Jim Clarke at Intel Labs in a statement.

Yet we are still at “mile one of this marathon”, Clarke said. This demonstration is a proof of concept, but it isn’t free of errors within the processor. Better and bigger processors will continue to be built and used to do more useful calculations.

Read more: Google’s quantum computing plans threatened by IBM curveball
At the same time, classical computing isn’t giving up the fight. Over the past few years, as quantum computing took steps towards supremacy, classical computing moved the goal posts as researchers showed it was able to simulate ever more complex systems. It is likely that this back-and-forth will continue.

“We expect that lower simulation costs than reported here will eventually be achieved, but we also expect they will be consistently outpaced by hardware improvements on larger quantum processors,” says the Google paper.

A month ago, news broke that Google had reportedly achieved “quantum supremacy”: it had gotten a quantum computer to run a calculation that would take a classical computer an unfeasibly long time. While the calculation itself—essentially, a very specific technique for outputting random numbers—is about as useful as the Wright brothers’ 12-second first flight, it would be a milestone of similar significance, marking the dawn of an entirely new era of computing.

But in a blog post published today, IBM disputes Google’s claim. The task that Google says might take the world’s fastest classical supercomputer 10,000 years can actually, says IBM, be done in just days.




As John Preskill, the CalTech physicist who coined the term “quantum supremacy,” wrote in an article for Quanta magazine, Google specifically chose a very narrow task that a quantum computer would be good at and a classical computer is bad at. “This quantum computation has very little structure, which makes it harder for the classical computer to keep up, but also means that the answer is not very informative,” he wrote.

Google’s research paper hasn’t been published, but a draft was leaked online last month. In it, researchers say they got a machine with 53 quantum bits, or qubits, to do the calculation in 200 seconds. They also estimated that it would take the world’s most powerful supercomputer, the Summit machine at Oak Ridge National Laboratory, 10,000 years to repeat it with equal “fidelity,” or the same level of uncertainty as the inherently uncertain quantum system.

The problem is that such simulations aren’t just a matter of porting the code from a quantum computer to a classical one. They grow exponentially harder the more qubits you’re trying to simulate. For that reason, there are a lot of different techniques for optimizing the code to arrive at a good enough equivalent.

And that’s where Google and IBM differ. The IBM researchers propose a method that they say would take just two and a half days on a classical machine “with far greater fidelity,” and that “with additional refinements” this could come down even further.

Quantum Computing and Quantum Supremacy at Google


The key difference? Hard drives. Simulating a quantum computer in a classical one requires storing vast amounts of data in memory during the process to represent the condition of the quantum computer at any given moment. The less memory you have available, the more you have to slice up the task into stages, and the longer it takes. Google’s method, IBM says, relied heavily on storing that data in RAM, while IBM’s “uses both RAM and hard drive space.” It also proposes using a slew of other classical optimization techniques, in both hardware and software, to speed up the computation. To be fair, IBM hasn't tested it in practice, so it's hard to know if it would work as proposed. (Google declined to comment.)

So what’s at stake? Either a whole lot or not much, depending on how you look at it. As Preskill points out, the problem Google reportedly solved is of almost no practical consequence, and even as quantum computers get bigger, it will be a long time before they can solve any but the narrowest classes of problems. Ones that can crack modern codes will likely take decades to develop, at a minimum.

IAS Distinguished Lecture: Prof Leo Kouwenhoven

Moreover, even if IBM is right that Google hasn’t achieved it this time, the quantum supremacy threshold is surely not far off. The fact that simulations get exponentially harder as you add qubits means it may only take a slightly larger quantum machine to get to the point of being truly unbeatable at something.

Still, as Preskill notes, even limited quantum supremacy is “a pivotal step in the quest for practical quantum computers.” Whoever ultimately achieves it will, like the Wright brothers, get to claim a place in history.

Every major tech company is looking at quantum computers as the next big breakthrough in computing. Teams at Google,  Microsoft, Intel, IBM and various startups and academic labs are racing to become the first to achieve quantum supremacy — that is, the point where a quantum computer can run certain algorithms faster than a classical computer ever could.

Quantum Computing Germany Meetup v1.0

Today, Google said that it believes that Bristlecone, its latest quantum processor, can put it on a path to reach quantum supremacy in the future. The purpose of Bristlecone, Google says, it to provide its researchers with a testbed “for research into system error rates and scalability of our qubit technology, as well as applications in quantum simulation, optimization, and machine learning.”

One of the major issues that all quantum computers have to contend with is error rates. Quantum computers typically run at extremely low temperatures (we’re talking millikelvins here) and are shielded from the environment because today’s quantum bits are still highly unstable and any noise can lead to errors.

Because of this, the qubits in modern quantum processors (the quantum computing versions of traditional bits) aren’t really single qubits but often a combination of numerous bits to help account for potential errors. Another limited factor right now is that most of these systems can only preserve their state for under 100 microseconds.

The systems that Google previously demonstrated showed an error rate of one percent for readout, 0.1 percent for single-qubit and 0.6 percent for two-qubit gates.

Quantum computing and the entanglement frontier



Every Bristlecone chip features 72 qubits. The general assumption in the industry is that it will take 49 qubits to achieve quantum supremacy, but Google also cautions that a quantum computer isn’t just about qubits. “Operating a device such as Bristlecone at low system error requires harmony between a full stack of technology ranging from software and control electronics to the processor itself,” the team writes today. “Getting this right requires careful systems engineering over several iterations.”

Google’s announcement today will put some new pressure on other teams that are also working on building functional quantum computers. What’s interesting about the current state of the industry is that everybody is taking different approaches.

Microsoft is currently a bit behind in that its team hasn’t actually produced a qubit yet, but once it does, its approach — which is very different from Google’s — could quickly lead to a 49 qubit machine. Microsoft is also working on a programming language for quantum computing. IBM currently has a 50-qubit machine in its labs and lets developers play with a cloud-based simulation of a quantum computer.

Technical quarrels between quantum computing experts rarely escape the field’s rarified community. Late Monday, though, IBM’s quantum team picked a highly public fight with Google.

In a technical paper and blog post, IBM took aim at potentially history-making scientific results accidentally leaked from a collaboration between Google and NASA last month. That draft paper claimed Google had reached a milestone dubbed “quantum supremacy”—a kind of drag race in which a quantum computer proves able to do something a conventional computer can’t.

Monday, Big Blue’s quantum PhDs said Google’s claim of quantum supremacy was flawed. IBM said Google had essentially rigged the race by not tapping the full power of modern supercomputers. “This threshold has not been met,” IBM’s blog post says. Google declined to comment.

It will take time for the quantum research community to dig through IBM’s claim and any responses from Google. For now, Jonathan Dowling, a professor at Louisiana State University, says IBM appears to have some merit. “Google picked a problem they thought to be really hard on a classical machine, but IBM now has demonstrated that the problem is not as hard as Google thought it was,” he says.

Whoever is proved right in the end, claims of quantum supremacy are largely academic for now. The problem crunched to show supremacy doesn’t need to have immediate practical applications. It's a milestone suggestive of the field’s long-term dream: That quantum computers will unlock new power and profits by enabling progress in tricky areas such as battery chemistry or health care. IBM has promoted its own quantum research program differently, highlighting partnerships with quantum-curious companies playing with its prototype hardware, such as JP Morgan, which this summer claimed to have figured out how to run financial risk calculations on IBM quantum hardware.

Quantum Computing 2019 Update

The IBM-Google quantretemps illustrates the paradoxical state of quantum computing. There has been a burst of progress in recent years, leading companies such as IBM, Google, Intel, and Microsoft to build large research teams. Google has claimed for years to be close to demonstrating quantum supremacy, a useful talking point as it competed with rivals to hire top experts and line up putative customers. Yet while quantum computers appear closer than ever, they remain far from practical use, and just how far isn’t easily determined.

The draft Google paper that appeared online last month described posing a statistical math problem to both the company’s prototype quantum processor, Sycamore, and the world’s fastest supercomputer, Summit, at Oak Ridge National Lab. The paper used the results to estimate that a top supercomputer would need approximately 10,000 years to match what Sycamore did in 200 seconds.


IBM, which developed Summit, says the supercomputer could have done that work in 2 ½ days, not millennia—and potentially even faster, given more time to finesse its implementation. That would still be slower than the time posted by Google’s Sycamore quantum chip, but the concept of quantum supremacy as originally conceived by Caltech professor John Preskill required the quantum challenger to do something that a classical computer could not do at all.

This is not the first time that Google’s rivals have questioned its quantum supremacy plans. In 2017, after the company said it was closing in on the milestone, IBM researchers published results that appeared to move the goalposts. Early in 2018, Google unveiled a new quantum chip called Bristlecone said to be ready to demonstrate supremacy. Soon, researchers from Chinese ecommerce company Alibaba, which has its own quantum computing program, released analysis claiming that the device could not do what Google said.

Google is expected to publish a peer-reviewed version of its leaked supremacy paper, based on the newer Sycamore chip, bringing its claim onto the scientific record. IBM’s paper released Monday is not yet peer reviewed either, but the company says it will be.

Did Google Just Achieve 'Quantum Supremacy'?


Jay Gambetta, one of IBM’s top quantum researchers and a coauthor on the paper, says he expects it to influence whether Google’s claims ultimately gain acceptance among technologists. Despite the provocative way IBM chose to air its technical concerns, he claims the company’s motivation is primarily to head off unhelpful expectations around the term “quantum supremacy,” not to antagonize Google. “Quantum computing is important and is going to change how computing is done,” Gambetta says. “Let’s focus on the road map without creating hype.”

CeBIT-Australia-2016-michael-bremner-big-data-and-analytics-commercialisation-quantum-computing



Other physicists working on quantum computing agree that supremacy is not a top priority—but say IBM’s tussle with Google isn’t either.

“I don't much like these claims of quantum supremacy. What might be quantum supreme today could just be classical inferior tomorrow,” says Dowling of Louisiana State. “I am much more interested in what the machine can do for me on any particular problem.”

Chris Monroe, a University of Maryland professor and cofounder of quantum computing startup IonQ agrees. His company is more interested in demonstrating practical uses for early quantum hardware than academic disputes between two tech giants, he says. “We’re not going to lose much sleep over this debate,” he says.

More Information:

https://towardsdatascience.com/google-has-cracked-quantum-supremacy-cd70c79a774b

https://spectrum.ieee.org/tech-talk/computing/hardware/how-googles-quantum-supremacy-plays-into-quantum-computings-long-game

https://ai.googleblog.com/2018/03/a-preview-of-bristlecone-googles-new.html

https://www.technologyreview.com/s/614604/quantum-supremacy-from-google-not-so-fast-says-ibm/

https://www.newscientist.com/article/2217347-google-claims-it-has-finally-reached-quantum-supremacy/

https://www.newscientist.com/article/mg23130894-000-revealed-googles-plan-for-quantum-computer-supremacy/

https://www.newscientist.com/article/2151032-googles-quantum-computing-plans-threatened-by-ibm-curveball/

https://www.cs.virginia.edu/~robins/The_Limits_of_Quantum_Computers.pdf

What is Azure Synapse Analytics 

Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources—at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs

Vision Keynote with Satya Nadella | Microsoft Ignite 2019


Azure Synapse has four components:

• SQL Analytics: Complete T-SQL based analytics – Generally Available
• SQL pool (pay per DWU provisioned)
• SQL on-demand (pay per TB processed) – (Preview)
• Spark: Deeply integrated Apache Spark (Preview)
• Data Integration: Hybrid data integration (Preview)
• Studio: Unified user experience. (Preview)

Note
To access the preview features of Azure Synapse, request access here. Microsoft will triage all requests and respond as soon as possible.

SQL Analytics and SQL pool in Azure Synapse

SQL Analytics refers to the enterprise data warehousing features that are generally available with Azure Synapse.

Azure Synapse Analytics - Next-gen Azure SQL Data Warehouse


SQL pool represents a collection of analytic resources that are being provisioned when using SQL Analytics. The size of SQL pool is determined by Data Warehousing Units (DWU).

Import big data with simple PolyBase T-SQL queries, and then use the power of MPP to run high-performance analytics. As you integrate and analyze, SQL Analytics will become the single version of truth your business can count on for faster and more robust insights.

Modern Data Warehouse overview | Azure SQL Data Warehouse


In a cloud data solution, data is ingested into big data stores from a variety of sources. Once in a big data store, Hadoop, Spark, and machine learning algorithms prepare and train the data. When the data is ready for complex analysis, SQL Analytics uses PolyBase to query the big data stores. PolyBase uses standard T-SQL queries to bring the data into SQL Analytics tables.

Azure data platform overview


SQL Analytics stores data in relational tables with columnar storage. This format significantly reduces the data storage costs, and improves query performance. Once data is stored, you can run analytics at massive scale. Compared to traditional database systems, analysis queries finish in seconds instead of minutes, or hours instead of days.

The analysis results can go to worldwide reporting databases or applications. Business analysts can then gain insights to make well-informed business decisions.

Azure Synapse Analytics (formerly SQL DW) architecture

Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources—at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs.

Azure Synapse has four components:

• SQL pool (pay per DWU provisioned) – Generally Available
• SQL on-demand (pay per TB processed) – (Preview)

2) Spark : Deeply integrated Apache Spark (Preview)
3) Data Integration : Hybrid data integration (Preview)
4) Studio : unified user experience. (Preview)


On November fourth, Microsoft announced Azure Synapse Analytics, the next evolution of Azure SQL Data Warehouse. Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources—at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs.

With Azure Synapse, data professionals can query both relational and non-relational data using the familiar SQL language. This can be done using either serverless on-demand queries for data exploration and ad hoc analysis or provisioned resources for your most demanding data warehousing needs. A single service for any workload.

In fact, it’s the first and only analytics system to have run all the TPC-H queries at petabyte-scale. For current SQL Data Warehouse customers, you can continue running your existing data warehouse workloads in production today with Azure Synapse and will automatically benefit from the new preview capabilities when they become generally available. You can sign up to preview new features like Serverless on-demand query, Azure Synapse studio, and Apache Spark™ integration.

Building a modern data warehouse

Taking SQL beyond data warehousing

A cloud native, distributed SQL processing engine is at the foundation of Azure Synapse and is what enables the service to support the most demanding enterprise data warehousing workloads. This week at Ignite we introduced a number of exciting features to make data warehousing with Azure Synapse easier and allow organizations to use SQL for a broader set of analytics use cases.

Unlock powerful insights faster from all data
Azure Synapse deeply integrates with Power BI and Azure Machine Learning to drive insights for all users, from data scientists coding with statistics to the business user with Power BI. And to make all types of analytics possible, we’re announcing native and built-in prediction support, as well as runtime level improvements to how Azure Synapse handles streaming data, parquet files, and Polybase. Let’s dive into more detail:

With the native PREDICT statement, you can score machine learning models within your data warehouse—avoiding the need for large and complex data movement. The PREDICT function (available in preview) relies on open model framework and takes user data as input to generate predictions. Users can convert existing models trained in Azure Machine Learning, Apache Spark™, or other frameworks into an internal format representation without having to start from scratch, accelerating time to insight.

Azure SQL Database & Azure SQL Data Warehouse


We’ve enabled direct streaming ingestion support and ability to execute analytical queries over streaming data. Capabilities such as: joins across multiple streaming inputs, aggregations within one or more streaming inputs, transform semi-structured data and multiple temporal windows are all supported directly in your data warehousing environment (available in preview). For streaming ingestion, customers can integrate with Event Hubs (including Event Hubs for Kafka) and IoT Hubs.

We’re also removing the barrier that inhibits securely and easily sharing data inside or outside your organization with Azure Data Share integration for sharing both data lake and data warehouse data.

Modern Data Warehouse Overview


By using new ParquetDirect technology, we are making interactive queries over the data lake a reality (in preview). It’s designed to access Parquet files with native support directly built into the engine. Through improved data scan rates, intelligent data caching and columnstore batch processing, we’ve improved Polybase execution by over 13x.

Introducing the modern data warehouse solution pattern with Azure SQL Data Warehouse

Workload isolation
To support customers as they democratize their data warehouses, we are announcing new features for intelligent workload management. The new Workload Isolation functionality allows you to manage the execution of heterogeneous workloads while providing flexibility and control over data warehouse resources. This leads to improved execution predictability and enhances the ability to satisfy predefined SLAs.


COPY statement
Analyzing petabyte-scale data requires ingesting petabyte-scale data. To streamline the data ingestion process, we are introducing a simple and flexible COPY statement. With only one command, Azure Synapse now enables data to be seamlessly ingested into a data warehouse in a fast and secure manner.

This new COPY statement enables using a single T-SQL statement to load data, parse standard CSV files, and more.

COPY statement sample code:

COPY INTO dbo.[FactOnlineSales] FROM ’https://contoso.blob.core.windows.net/Sales/’

Safe keeping for data with unmatched security
Azure has the most advanced security and privacy features in the market. These features are built into the fabric of Azure Synapse, such as automated threat detection and always-on data encryption. And for fine-grained access control businesses can ensure data stays safe and private using column-level security, native row-level security, and dynamic data masking (now generally available) to automatically protect sensitive data in real time.

To further enhance security and privacy, we are introducing Azure Private Link. It provides a secure and scalable way to consume deployed resources from your own Azure Virtual Network (VNet). A secure connection is established using a consent-based call flow. Once established, all data that flows between Azure Synapse and service consumers is isolated from the internet and stays on the Microsoft network. There is no longer a need for gateways, network addresses translation (NAT) devices, or public IP addresses to communicate with the service.


SQL Analytics MPP architecture components

SQL Analytics leverages a scale out architecture to distribute computational processing of data across multiple nodes. The unit of scale is an abstraction of compute power that is known as a data warehouse unit. Compute is separate from storage which enables you to scale compute independently of the data in your system.

AI for Intelligent Cloud and Intelligent Edge: Discover, Deploy, and Manage with Azure ML Services


SQL Analytics uses a node-based architecture. Applications connect and issue T-SQL commands to a Control node, which is the single point of entry for SQL Analytics. The Control node runs the MPP engine which optimizes queries for parallel processing, and then passes operations to Compute nodes to do their work in parallel.

The Compute nodes store all user data in Azure Storage and run the parallel queries. The Data Movement Service (DMS) is a system-level internal service that moves data across the nodes as necessary to run queries in parallel and return accurate results.

With decoupled storage and compute, when using SQL Analytics one can:

• Independently size compute power irrespective of your storage needs.
• Grow or shrink compute power, within a SQL pool (data warehouse), without moving data.
• Pause compute capacity while leaving data intact, so you only pay for storage.
• Resume compute capacity during operational hours.

Data Warehousing And Big Data Analytics in Azure Basics Tutorial

Azure storage

SQL Analytics leverages Azure storage to keep your user data safe. Since your data is stored and managed by Azure storage, there is a separate charge for your storage consumption. The data itself is sharded into distributions to optimize the performance of the system. You can choose which sharding pattern to use to distribute the data when you define the table. These sharding patterns are supported:

• Hash
• Round Robin
• Replicate
• Control node

The Control node is the brain of the architecture. It is the front end that interacts with all applications and connections. The MPP engine runs on the Control node to optimize and coordinate parallel queries. When you submit a T-SQL query to SQL Analytics, the Control node transforms it into queries that run against each distribution in parallel.
Compute nodes

The Compute nodes provide the computational power. Distributions map to Compute nodes for processing. As you pay for more compute resources, SQL Analytics re-maps the distributions to the available Compute nodes. The number of compute nodes ranges from 1 to 60, and is determined by the service level for SQL Analytics.
Each Compute node has a node ID that is visible in system views. You can see the Compute node ID by looking for the node_id column in system views whose names begin with sys.pdw_nodes. For a list of these system views, see MPP system views.
Data Movement Service

Data Movement Service (DMS) is the data transport technology that coordinates data movement between the Compute nodes. Some queries require data movement to ensure the parallel queries return accurate results. When data movement is required, DMS ensures the right data gets to the right location.

Machine Learning and AI


Distributions
A distribution is the basic unit of storage and processing for parallel queries that run on distributed data. When SQL Analytics runs a query, the work is divided into 60 smaller queries that run in parallel.
Each of the 60 smaller queries runs on one of the data distributions. Each Compute node manages one or more of the 60 distributions. A SQL pool with maximum compute resources has one distribution per Compute node. A SQL pool with minimum compute resources has all the distributions on one compute node.


Hash-distributed tables

A hash distributed table can deliver the highest query performance for joins and aggregations on large tables.
To shard data into a hash-distributed table, SQL Analytics uses a hash function to deterministically assign each row to one distribution. In the table definition, one of the columns is designated as the distribution column. The hash function uses the values in the distribution column to assign each row to a distribution.

The following diagram illustrates how a full (non-distributed table) gets stored as a hash-distributed table.



Distributed table
Each row belongs to one distribution.
A deterministic hash algorithm assigns each row to one distribution.
The number of table rows per distribution varies as shown by the different sizes of tables.
There are performance considerations for the selection of a distribution column, such as distinctness, data skew, and the types of queries that run on the system.



Round-robin distributed tables
A round-robin table is the simplest table to create and delivers fast performance when used as a staging table for loads.
A round-robin distributed table distributes data evenly across the table but without any further optimization. A distribution is first chosen at random and then buffers of rows are assigned to distributions sequentially. It is quick to load data into a round-robin table, but query performance can often be better with hash distributed tables. Joins on round-robin tables require reshuffling data and this takes additional time.

Replicated Tables
A replicated table provides the fastest query performance for small tables.
A table that is replicated caches a full copy of the table on each compute node. Consequently, replicating a table removes the need to transfer data among compute nodes before a join or aggregation. Replicated tables are best utilized with small tables. Extra storage is required and there is additional overhead that is incurred when writing data which make large tables impractical.
The diagram below shows a replicated table which is cached on the first distribution on each compute node.

AI for an intelligent cloud and intelligent edge: Discover, deploy, and manage with Azure ML services

Compare price-performance of Azure Synapse Analytics and Google BigQuery

Azure Synapse Analytics (formerly Azure SQL Data Warehouse) outperforms Google BigQuery in all TPC-H and TPC-DS* benchmark queries. Azure Synapse Analytics consistently demonstrated better price-performance compared with BigQuery, and costs up to 94 percent less when measured against Azure Synapse Analytics clusters running TPC-H* benchmark queries.


*Performance and price-performance claims based on data from a study commissioned by Microsoft and conducted by GigaOm in January 2019 for the TPC-H benchmark report and March 2019 for the TPC-DS benchmark report. Analytics in Azure is up to 14 times faster and costs 94 percent less, according to the TPC-H benchmark, and is up to 12 times faster and costs 73 percent less, according to the TPC-DS benchmark. Benchmark data is taken from recognized industry standards, TPC Benchmark™ H (TPC-H) and TPC Benchmark™ DS (TPC-DS), and is based on query execution performance testing of 66 queries for TPC-H and 309 queries for TPC-DS, conducted by GigaOm in January 2019 and March 2019, respectively; testing commissioned by Microsoft. Price-performance is calculated by GigaOm as the TPC-H/TPC-DS metric of cost of ownership divided by composite query. Prices are based on publicly available US pricing as of January 2019 for TPC-H queries and March 2019 for TPC-DS queries. Actual performance and prices may vary. Learn more about the GigaOm benchmark study

QSSUG: Azure Cognitive Services – The Rise of the Machines


Forrester interviewed four customers and surveyed 364 others on their use of Azure analytics with Power BI. Of those surveyed customers, 85 percent agreed or strongly agreed that well-integrated analytics databases and storage, a data management stack, and business intelligence tools were beneficial to their organization. Customers also reported a 21.9 percent average reduction in the overall cost of Microsoft analytics and BI offerings when compared to alternative analytics solutions.

Based on the companies interviewed and surveyed, Forrester projects that a Microsoft analytics and business intelligence (BI) solution could provide:
Benefits of $22.1 million over three years versus costs of $6 million, resulting in a net present value of $16.1 million and a projected return on investment of 271 percent.
Reduced total cost of ownership by 25.7 percent.
Better overall analytics system performance with improved data security, enhanced decision making, and democratized data access.

Modern Data Warehousing with BigQuery (Cloud Next '19)


Analytics in Azure is up to 14x faster and costs 94% less than other cloud providers. Why go anywhere else?

Julia White Corporate Vice President, Microsoft Azure
It’s true. With the volume and complexity of data rapidly increasing, performance and security are critical requirements for analytics. But not all analytics services are built equal. And not all cloud storage is built for analytics.

Only Azure provides the most comprehensive set of analytics services from data ingestion to storage to data warehousing to machine learning and BI. Each of these services have been finely tuned to provide industry leading performance, security and ease of use, at unmatched value. In short, Azure has you covered.

Unparalleled price-performance

When it comes to analytics, price-performance is key. In July 2018, GigaOm published a study that showed that Azure SQL Data Warehouse was 67 percent faster and 23 percent cheaper than Amazon Web Service RedShift.

That was then. Today, we’re even better!

In the most recent study by GigaOm, they found that Azure SQL Data Warehouse is now outperforming the competition up to a whopping 14x times. No one else has produced independent, industry-accepted benchmarks like these. Not AWS Redshift or Google BigQuery. And the best part? Azure is up to 94 percent cheaper.

This industry leading price-performance extends to the rest of our analytics stack. This includes Azure Data Lake Storage, our cloud data storage service, and Azure Databricks, our big data processing service. Customers like Newell Brands – worldwide marketer of consumer and commercial products such as Rubbermaid, Mr. Coffee and Oster – recently moved their workload to Azure and realized significant improvements.

“Azure Data Lake Storage will streamline our analytics process and deliver better end to end performance with lower cost.” 
– Danny Siegel, Vice President of Information Delivery Systems, Newell Brands
Secure cloud analytics

All the price-performance in the world means nothing without security. Make the comparison and you will see Azure is the most trusted cloud in the market. Azure has the most comprehensive set of compliance offerings, including more certifications than any other cloud vendor combined with advanced identity governance and access management with Active Directory integration.

For analytics, we have developed additional capabilities to meet customers’ most stringent security requirements. Azure Data Lake Storage provides multi-layered security including POSIX compliant file and folder permissions and at-rest encryption. Similarly, Azure SQL Data Warehouse utilizes machine learning to provide the most comprehensive set of security capabilities across data protection, access control, authentication, network security, and automatic threat detection.

Insights for all

What’s the best compliment to Azure Analytics’ unmatched price-performance and security? The answer is Microsoft Power BI.

Power BI’s ease of use enables everyone in your organization to benefit from our analytics stack. Employees can get their insights in seconds from all enterprise data stored in Azure. And without limitations on concurrency, Power BI can be used across teams to create the most beautiful visualizations that deliver powerful insights.

Leveraging Microsoft’s Common Data Model, Power BI users can easily access and analyze enterprise data using a common data schema without needing complex data transformation. Customers looking for petabyte-scale analytics can leverage Power BI Aggregations with Azure SQL Data Warehouse for rapid query. Better yet, Power BI users can easily apply sophisticated AI models built with Azure. Powerful insights easily accessible to all.

Customers like Heathrow Airport, one of the busiest airports in the world, are empowering their employees with powerful insights:

“With Power BI, we can very quickly connect to a wide range of data sources with very little effort and use this data to run Heathrow more smoothly than ever before. Every day, we experience a huge amount of variability in our business. With Azure, we’re getting to the point where we can anticipate passenger flow and stay ahead of disruption that causes stress for passengers and employee.”

– Stuart Birrell, Chief Information Officer, Heathrow Airport

Future-proof

We continue to focus on making Azure the best place for your data and analytics. Our priority is to meet your needs for today and tomorrow.

So, we are excited to make the following announcements:

General availability of Azure Data Lake Storage: The first cloud storage that combines the best of hierarchical files system and blob storage.
General availability of Azure Data Explorer: A fast, fully managed service that simplifies ad hoc and interactive analysis over telemetry, time-series, and log data. This service, powering other Azure services like Log Analytics, App Insights, Time Series Insights, is useful to query streaming data to identify trends, detect anomalies, and diagnose problems.
Preview of new Mapping Data Flow capability in Azure Data Factory: Visual Flow provides a visual, zero-code experience to help data engineers to easily build data transformations. This complements the Azure Data Factory’s code-first experience to enable data engineers of all skill levels to collaborate and build powerful hybrid data transformation pipelines.
Azure provides the most comprehensive platform for analytics. With these updates, Azure solidifies its leadership in analytics.

More Information:

https://azure.microsoft.com/en-us/blog/azure-sql-data-warehouse-is-now-azure-synapse-analytics/

https://azure.microsoft.com/en-us/services/synapse-analytics/compare

https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-overview-what-is

https://docs.microsoft.com/en-us/azure/sql-data-warehouse/what-is-a-data-warehouse-unit-dwu-cdwu

https://azure.microsoft.com/en-us/resources/forrester-tei-microsoft-azure-analytics-with-power-bi

https://azure.microsoft.com/mediahandler/files/resourcefiles/data-warehouse-in-the-cloud-benchmark/FINAL%20data-warehouse-cloud-benchmark.pdf

https://www.gartner.com/doc/reprints?id=1-3U1LC65&ct=170222&st=sb

https://clouddamcdnprodep.azureedge.net/gdc/gdcEbYaLj/original

https://clouddamcdnprodep.azureedge.net/gdc/gdcpLECbc/original

https://azure.microsoft.com/en-us/blog/analytics-in-azure-is-up-to-14x-faster-and-costs-94-less-than-other-cloud-providers-why-go-anywhere-else/

IBM's QISKit open source project

Qiskit is an open-source framework for working with quantum computers at the level of circuits, pulses, and algorithms.

"The Arrival of Quantum Computing" by Will Zeng


A central goal of Qiskit is to build a software stack that makes it easy for anyone to use quantum computers. However, Qiskit also aims to facilitate research on the most important open issues facing quantum computation today.



You can use Qiskit to easily design experiments and run them on simulators and real quantum computers.



Qiskit Quantum Computing tech talk



Qiskit consists of four foundational elements:

Qiskit Terra: Composing quantum programs at the level of circuits and pulses with the code foundation.

Qiskit Aer: Accelerating development via simulators, emulators, and debuggers

Qiskit Ignis: Addressing noise and errors

Qiskit Aqua: Building algorithms and applications


Introduction to Quantum Computer

The Qiskit Elements

Terra

Terra, the ‘earth’ element, is the foundation on which the rest of Qiskit lies. Terra provides a bedrock for composing quantum programs at the level of circuits and pulses, to optimize them for the constraints of a particular device, and to manage the execution of batches of experiments on remote-access devices. Terra defines the interfaces for a desirable end-user experience, as well as the efficient handling of layers of optimization, pulse scheduling and backend communication.

Using QISkit: The SDK for Quantum Computing


Qiskit Terra is organized in six main modules:

Circuit A quantum circuit is a model for quantum computing in which a computation is done by performing a sequence of quantum operations (usually gates) on a register of qubits. A quantum circuit usually starts with the qubits in the |0,…,0> state and these gates evolve the qubits to states that cannot be efficiently represented on a classical computer. To extract information on the state a quantum circuit must have a measurement which maps the outcomes (possible random due to the fundamental nature of quantum systems) to classical registers which can be efficiently represented.

Pulse A pulse schedule is set of pulses which are sent to a quantum experiment that are applied to a channel (experimental input line). This is a lower level than circuits and requires each gate in the circuit to be represented as a set of pulses. At this level the experiments can be designed to reduce errors (dynamical decoupling, error mitigation, and optimal pulse shapes).

Transpiler A major part of research on quantum computing is working out how to run a quantum circuits on real devices. In these devices, experimental errors and decoherence introduce errors during computation. Thus, to obtain a robust implementation it is essential to reduce the number of gates and the overall running time of the quantum circuit. The transpiler introduces the concept of a pass manager to allow users to explore optimization and find better quantum circuits for their given algorithm. We call it a transpiler as the end result is still a circuit.

The Arrival of Quantum Computing – Quantum Networks


Providers Once the user has made the circuits to run on the backend they need to have a convenient way of working with it. In Terra we do this using four parts:

A Provider is an entity that provides access to a group of different backends (for example, backends available through the IBM Q Experience). It interacts with those backends to, for example, find out which ones are available, or retrieve an instance of a particular backend.

Backend represent either a simulator or a real quantum computer and are responsible for running quantum circuits and returning results. They have a run method which takes in a qobj as input and returns a BaseJob object. This object allows asynchronous running of jobs for retrieving results from a backend when the job is completed.

Job instances can be thought of as the “ticket” for a submitted job. They find out the execution’s state at a given point in time (for example, if the job is queued, running, or has failed) and also allow control over the job.

Result. Once the job has finished Terra allows the results to be obtained from the remote backends using result = job.result(). This result object holds the quantum data and the most common way of interacting with it is by using result.get_counts(circuit). This method allows the user to get the raw counts from the quantum circuit and use them for more analysis with quantum information tools provided by Terra.

Quantum Information To perform more advanced algorithms and analysis of the circuits run on the quantum computer, it is important to have tools to implement simple quantum information tasks. These include methods to both estimate metrics and generate quantum states, operations, and channels.

IBM Q Quantum Computing


Visualization In Terra we have many tools to visualize a quantum circuit. This allows a quick inspection of the quantum circuit to make sure it is what the user wanted to implement. There is a text, python and latex version. Once the circuit has run it is important to be able to view the output. There is a simple function (plot_histogram) to plot the results from a quantum circuit including an interactive version. There is also a function plot_state and plot_bloch_vector that allow the plotting of a quantum state. These functions are usually only used when using the statevector_simulator backend but can also be used on real data after running state tomography experiments (Ignis).

Aer
Aer, the ‘air’ element, permeates all Qiskit elements. To really speed up development of quantum computers we need better simulators, emulators and debuggers. Aer helps us understand the limits of classical processors by demonstrating to what extent they can mimic quantum computation. Furthermore, we can use Aer to verify that current and near-future quantum computers function correctly. This can be done by stretching the limits of simulation, and by simulating the effects of realistic noise on the computation.

Aer provides a high performance simulator framework for quantum circuits using the Qiskit software stack. It contains optimized C++ simulator backends for executing circuits compiled in Terra. Aer also provides tools for constructing highly configurable noise models for performing realistic noisy simulations of the errors that occur during execution on real devices.

Quantum Computing: Technology, Market and Ecosystem Overview

Qiskit Aer includes three high performance simulator backends:

Qasm Simulator
Allows ideal and noisy multi-shot execution of qiskit circuits and returns counts or memory. There are multiple methods that can be used that simulate different cirucits more efficiently. These inlude:

statevector - Uses a dense statevector simulation.

stabilizer - Uses a Clifford stabilizer state simulator that is only valid for Clifford circuits and noise models.

extended_stabilizer - Uses an approximate simulator that decomposes circuits into stabilizer state terms, the number of which grows with the number of non-Clifford gates.

matrix_product_state - Uses a Matrix Product State (MPS) simulator.

Statevector Simulator
Allows ideal single-shot execution of qiskit circuits and returns the final statevector of the simulator after application.

Unitary Simulator
Allows ideal single-shot execution of qiskit circuits and returns the final unitary matrix of the circuit itself. Note that the circuit cannot contain measure or reset operations for this backend.

Ignis
Ignis, the ‘fire’ element, is dedicated to fighting noise and errors and to forging a new path. This includes better characterization of errors, improving gates, and computing in the presence of noise. Ignis is meant for those who want to design quantum error correction codes, or who wish to study ways to characterize errors through methods such as tomography, or even to find a better way for using gates by exploring dynamical decoupling and optimal control.

Ignis provides code for users to easily generate circuits for specific experiments given a minimal set of user input parameters. Ignis code contains three fundamental building blocks:

Circuits
The circuits module provides the code to generate the list of circuits for a particular Ignis experiment based on a minimal set of user parameters. These are then run on Terra or Aer.

Fitters
The results of an Ignis experiment are passed to the Fitters module where they are analyzed and fit according to the physics model describing the experiment. Fitters can plot the data plus fit and output a list of parameters.

Filters
For certain Ignis experiments, the fitters can output a Filter object. Filters can be used to mitigate errors in other experiments using the calibration results of an Ignis experiment.

Qiskit Ignis is organized into three types of experiments that can be performed:

Characterization
Characterization experiments are designed to measure parameters in the system such as noise parameters (T1, T2-star, T2), Hamiltonian parameters such as the ZZ interaction rate and control errors in the gates.

Verification
Verification experiments are designed to verify gate and small circuit performance. Verification includes state and process tomography, quantum volume and randomized benchmarking (RB). These experiments provide the information to determine performance metrics such as the gate fidelity.

Mitigation
Mitigation experiments run calibration circuits that are analyzed to generate mitigation routines that can be applied to arbitrary sets of results run on the same backend. Ignis code will generate a list of circuits that run calibration measurements. The results of these measurements will be processed by a Fitter and will output a Filter than can be used to apply mitigation to other results.

Aqua
Aqua, the ‘water’ element, is the element of life. To make quantum computing live up to its expectations, we need to find real-world applications. Aqua is where algorithms for quantum computers are built. These algorithms can be used to build applications for quantum computing. Aqua is accessible to domain experts in chemistry, optimization, finance and AI, who want to explore the benefits of using quantum computers as accelerators for specific computational tasks.

Problems that may benefit from the power of quantum computing have been identified in numerous domains, such as Chemistry, Artificial Intelligence (AI), Optimization and Finance. Quantum computing, however, requires very specialized skills. To address the needs of the vast population of practitioners who want to use and contribute to quantum computing at various levels of the software stack, we have created Qiskit Aqua.

Programming Existing Quantum Computers

Development Strategy

Roadmap
We are going to look out 12 months to establish a set of goals we want to work towards. When planning, we typically look at potential work from the perspective of the elements.

Qiskit Terra
In 2018 we worked on formalizing the backends and user flow in Qiskit Terra. The basic idea is as follows: the user designs a quantum circuit and then, through a set of transpiler passes, rewrites the circuit to run on different backends with different optimizations. We also introduced the concept of a provider, whose role is to supply backends for the user to run quantum circuits on. The provider API we have defined at version one supplies a set of schemas to verify that the provider and its backends are Terra-compatible.

In 2019, we have many extensions planned. These include:

Add passes to the transpiler. The goal here is to be more efficient in circuit depth as well as adding passes that find approximate circuits and resource estimations.

Introduce a circuit foundry and circuit API. This has the goal of making sure that a user can easily build complex circuits from operations. Some of these include adding controls and power to operations, and inserting unitary matrices directly.

Provide an API for OpenPulse. Now that OpenPulse is defined, and the IBM Q provider can accept it, we plan to build out the pulse features. These will include a scheduler and tools for building experiments out of pulses. Also included will be tools for mapping between experiments with gates (QASM) to experiments with pulses.

Qiskit Aer
The first release of Qiskit Aer was made avaialble at the end of 2018. It included C++ implementations of QASM, statevector, and unitary simulators. These are the core to Qiskit Aer, and replace the simulators that existed in Terra. The QASM simulator includes a customizable general (Kraus) noise model, and all simulators include CPU parallelization through the OpenMP library.

In 2019, Aer will be extended in many ways:

Optimize simulators. We are going to start profiling the simulators and work on making them faster. This will include automatic settings for backend configuration and OpenMP parallelization configuration based on the input Qobj and available hardware.

Classical simulation algorithms for quantum computational supremacy experiments


Develop additional simulator backends. We will include several approximate simulator backends that are more efficient for specific subclasses of circuits, such as the T-gate simulator, which works on Clifford and T gates (with low T-depth), and a stabilizer simulator, which works just on Clifford gates.

Add noise approximation tools. We plan to add noise approximation tools to mapping general (Kraus) noise models to approximate noise model that may be implemented on an approximate backends (for example only mixed Clifford and reset errors in the noise model).

Qiskit Ignis
This year, we are going to release the first version of Qiskit Ignis. The goal of Ignis is to be a set of tools for characterization of errors, improving gates, and enhancing computation in the presence of noise. Examples of these tools include optimal control, dynamical decoupling, and error mitigation.

In 2019, Ignis will include tools for:

quantum state/process tomography

randomized benchmarking over different groups

optimal control (e.g., pulse shaping)

dynamical decoupling

circuit randomization

error mitigation (to improve results for quantum chemistry experiments)

Qiskit Aqua
Aqua is an open-source library of quantum algorithms and applications, introduced in June 2018. As a library of quantum algorithms, Aqua comes with a rich set of quantum algorithms of general applicability—such as VQE, QAOA, Grover’s Search, Amplitude Estimation and Phase Estimation—and domain-specific algorithms-such as the Support Vector Machine (SVM) Quantum Kernel and Variational algorithms, suitable for supervised learning. In addition, Aqua includes algorithm-supporting components, such as optimizers, variational forms, oracles, Quantum Fourier Transforms, feature maps, multiclass classification extension algorithms, uncertainty problems, and random distributions. As a framework for quantum applications, Aqua provides support for Chemistry (released separately as the Qiskit Chemistry component), as well as Artificial Intelligence (AI), Optimization and Finance. Aqua is extensible across multiple domains, and has been designed and structured as a framework that allows researchers to contribute their own implementations of new algorithms and algorithm-supporting components.

Over the course of 2019, we are planning to enrich Aqua as follows:

We will include several new quantum algorithms, such as Deutsch-Jozsa, Simon’s, Bernstein-Vazirani, and Harrow, Hassidim, and Lloyd (HHL).

We will improve the performance of quantum algorithms on top of both simulators and real hardware.

We will provide better support for execution on real quantum hardware.

We will increase the set of problems supported by the AI, Optimization and Finance applications of Aqua.

Summary

These are examples of just some of the work we will be focusing on in the next 12 months. We will continuously adapt the plan based on feedback. Please follow along and let us know what you think!

IBM Quantum Computing

Versioning

The Qiskit project is made up of several elements each performing different functionality. Each is independently useful and can be used on their own, but for convenience we provide this repository and meta-package to provide a single entrypoint to install all the elements at once. This is to simplify the install process and provide a unified interface to end users. However, because each Qiskit element has it’s own releases and versions some care is needed when dealing with versions between the different repositories. This document outlines the guidelines for dealing with versions and releases of both Qiskit elements and the meta-package.

Quantum programming


For the rest of this guide the standard Semantic Versioning nomenclature will be used of: Major.Minor.Patch to refer to the different components of a version number. For example, if the version number was 0.7.1, then the major version is 0, the minor version 7, and the patch version 1.

Meta-package Version
The Qiskit meta-package version is an independent value that is determined by the releases of each of the elements being tracked. Each time we push a release to a tracked component (or add an element) the meta-package requirements, and version will need to be updated and a new release published. The timing should be coordinated with the release of elements to ensure that the meta-package releases track with element releases.

Adding New Elements
When a new Qiskit element is being added to the meta-package requirements, we need to increase the Minor version of the meta-package.

For example, if the meta-package is tracking 2 elements qiskit-aer and qiskit-terra and it’s version is 0.7.4. Then we release a new element qiskit-ignis that we intend to also have included in the meta-package. When we add the new element to the meta-package we increase the version to 0.8.0.

Patch Version Increases
When any Qiskit element that is being already tracked by the meta-package releases a patch version to fix bugs in a release we need also bump the requirement in the setup.py and then increase the patch version of the meta-package.

For example, if the meta-package is tracking 3 elements qiskit-terra==0.8.1, qiskit-aer==0.2.1, and qiskit-ignis==0.1.4 with the current version 0.9.6. When qiskit-terra release a new patch version to fix a bug 0.8.2 the meta-package will also need to increase it’s patch version and release, becoming 0.9.7.

Additionally, there are occasionally packaging or other bugs in the meta-package itself that need to be fixed by pushing new releases. When those are encountered we should increase the patch version to differentiate it from the broken release. Do not delete the broken or any old releases from pypi in any situation, instead just increase the patch version and upload a new release.

Minor Version Increases
Besides adding a new element to the meta-package the minor version of the meta-package should also be increased anytime a minor version is increased in a tracked element.

For example, if the meta-package is tracking 2 elements qiskit-terra==0.7.0 and qiskit-aer==0.1.1 and the current version is 0.7.5. When the qiskit-aer element releases 0.2.0 then we need to increase the meta-package version to be 0.8.0 to correspond to the new release.

Major Version Increases
The major version is different from the other version number components. Unlike the other version number components, which are updated in lock step with each tracked element, the major version is only increased when all tracked versions are bumped (at least before 1.0.0). Right now all the elements still have a major version number component of 0 and until each tracked element in the meta-repository is marked as stable by bumping the major version to be >=1 then the meta-package version should not increase the major version.

The behavior of the major version number component tracking after when all the elements are at >=1.0.0 has not been decided yet.

Qiskit Element Requirement Tracking
While not strictly related to the meta-package and Qiskit versioning how we track the element versions in the meta-package’s requirements list is important. Each element listed in the setup.py should be pinned to a single version. This means that each version of Qiskit should only install a single version for each tracked element. For example, the requirements list at any given point should look something like:

requirements = [
    "qiskit_terra==0.7.0",
    "qiskit-aer==0.1.1",
]
This is to aid in debugging, but also make tracking the versions across multiple elements more transparent.

It is also worth pointing out that the order we install the elements is critically important too. pip does not have a real dependency solver which means the installation order matters. So if there are overlapping requirements versions between elements or dependencies between elements we need to ensure that the order in the requirements list installs everything as expected. If the order needs to be change for some install time incompatibility it should be noted clearly.

Small quantum computers and big classical data


Module Status
Qiskit is developing so fast that is it is hard to keep all different parts of the API supported for various versions. We do our best and we use the rule that for one minor version update, for example 0.6 to 0.7, we will keep the API working with a deprecated warning. Please don’t ignore these warnings. Sometimes there are cases in which this can’t be done and for these in the release history we will outline these in great details.

This being said as we work towards Qiskit 1.0 there are some modules that have become stable and the table below is our attempt to label them

Providers

There are three providers that come with the default installation of Qiskit

Basic Aer Provider
This provider simulates ideal quantum circuits and has three backends. As Aer becomes more stable and can work on any operating system this provider will be removed.

Aer Provider
This is a more advance simulator that is written in C++. It runs faster than Basic Aer and also allows you to add noise to your circuits. This allow you to explore what happens to your circuits for realistic models of the experiments and design experiments that might be more resilient to the noise in today’s quantum computers.

IBM Q Provider
This provider gives you access to real experiments. You will need an IBM Q Experience account to use it. It also has an online HPC simulator that can be used. It is a hosted version of the Aer Provider.

Community Extensions
Qiskit has been designed with modularity in mind. It is extensible in many different ways; on the page, we highlight the ways in which the Qiskit community has engaged with Qiskit and developed extensions and packages on top of it.

Providers
The Qiskit base provider is an entity that provides access to a group of different backends (for example, backends available through IBM Q). It interacts with those backends to do many things: find out which ones are available, retrieve an instance of a particular backend, get backend properties and configurations, and handling running and working with jobs.

Additional providers
Decision diagram-based quantum simulator

- Organization: Johannes Kepler University, Linz, Austria (Alwin Zulehner and Robert Wille)
- Description: A local provider which allows Qiskit to use decision diagram-based quantum simulation
- Qiskit Version: 0.7
- More info: Webpage at JKU, Medium Blog and Github Repo
Quantum Inspire

- Organization: QuTech-Delft
- Description: A provider for the Quantum Inspire backend
- Qiskit Version: 0.7
- More info: Medium Blog and Github.
Transpiler
Circuit optimization is at the heart of making quantum computing feasible on actual hardware. A central component of Qiskit is the transpiler, which is a framework for manipulating quantum circuits according to certain transformations (known as transpiler passes). The transpiler enables users to create customized sets of passes, orchestrated by a pass manager, to transform the circuit according to the rules specified by the passes. In addition, the transpiler architecture is designed for modularity and extensibility, enabling Qiskit users to write their own passes, use them in the pass manager, and combine them with existing passes. In this way, the transpiler architecture opens up the door for research into aggressive optimization of quantum circuits.

Additional passes
t|ket〉 optimization & routing pass

- Organization: Cambridge Quantum Computing
- Description: Transpiler pass for circuit optimization and mapping to backend using CQC’s t|ket〉compiler.
- Qiskit Version: 0.7
- More info: Tutorial Notebook and Github.
Tools
Extending Qiskit with new tools and functionality is an important part of building a community. These tools can be new visualizations, slack integration, Juypter extensions and much more.

Project Highlight: Quantum Computing Meets Machine Learning

If learning is the first step towards intelligence, it’s no wonder we’re sending machines to school.
Machine learning, specifically, is the self-learning process by which machines use patterns to learn rather than (in the ideal case) asking humans for assistance. Seen as a subset of artificial intelligence, machine learning has been gaining traction in the development community as many frameworks are now available.

Quantum Programming A New Approach to Solve Complex Problems Francisco Gálvez Ramirez IBM Staff fjgramirez@es.ibm.com


And soon, you may have a machine learning framework available in your favorite quantum computing framework!



The winning project of 2019 Qiskit Camp Europe, QizGloria, is a hybrid quantum-classical machine learning interface with full Qiskit and PyTorch capabilities. PyTorch is a machine learning library that, like Qiskit, is free and open-source. By integrating Qiskit and PyTorch frameworks during the 24-hour hackathon, the QizGloria group demonstrated that you can use the best of the quantum and classical world for machine learning. The project is still ongoing modifications but may soon be integrated into Qiskit Aqua.
Below, we interview the four members of the QizGloria group about their project, their experiences, and their future outlook on the field. Interviews are edited for clarity.
Why did you think to combine Qiskit, a quantum-computing framework, with PyTorch, a machine-learning framework?

Controlling a Quantum Computer with Code


Karel Dumon: Classical machine learning is currently benefiting hugely from the open-source community, and this is something we want to leverage in quantum too. Our project focuses on the potential application of quantum computing for machine learning, but also on the use of machine learning to help progress quantum computing itself. Through our project, we hope to make it easier for machine learning developers to explore the quantum world.
Patrick Huembeli: To that effect, it makes Qiskit very accessible for people with a classical machine learning background — they can treat the quantum nodes just as another layer of their machine learning algorithm.

Amira Abbas: In that sense, this project bridges the gap between two communities, machine learning and quantum computing, whose research could seriously complement each other instead of diverging.
How do you think your integration will benefit the Qiskit community?

Dumon: There are a lot of open-source tools available for both quantum computing and machine learning, but those integrations do not provide the optimal synergy between the two worlds. What we tried to build is a tighter integration between Qiskit and PyTorch (an open-source machine learning framework from Facebook) that makes optimal use of the existing capabilities.

Isaac Turtletaub: In quantum computing, we commonly have circuits that need to be optimized with a classical computer. PyTorch is one of the largest machine learning libraries out there, and opens up the possibilities of using deep learning for optimizing quantum circuits.

Do you plan to continue working on this project?

Dumon: During the hackathon, we built the bridge between the two worlds, and showcased some possibilities — but we definitely believe that this is just the beginning of what is possible! While our Qiskit Camp submission was a proof-of-concept, we are currently working with the Qiskit team to include our work in the Qiskit Aqua codebase.

Turtletaub: I plan on continuing to work on this project by contributing to a generalized interface between PyTorch and Qiskit, allowing this to work on any variational quantum circuit. I hope collaborating with the IBM coaches will let all Qiskitters take advantage of our project.

Abbas: We also plan on writing a chapter on hybrid quantum-classical machine learning using PyTorch for the open-source Qiskit textbook and created an pull request for this on GitHub.
What is one of the more difficult challenges still ahead?

Huembeli: Getting the parameter binding of Qiskit right. This will be very important if we want to continue this project. This has to be thought through very well.
In what other ways could this project be expanded?
Turtletaub: This project could be expanded by not only opening up Qiskit to PyTorch, but to another machine learning library, such as TensorFlow.

Huembeli: And if we integrate it well into Qiskit, people will be able to add any nice classical machine learning feature to Qiskit. There is really no limit of applications.

Abbas: Since everything is open source, members of the community can contribute to the code (via pull requests) and add functionalities; make things more efficient, and even create more tutorials demonstrating new ideas or research.

Dumon: We hope that others start playing around with our code and help shape the idea further. This is at the core of the open-source spirit.

And on another topic — Qiskit Camp Europe — what was your favorite part?

Huembeli: The hackathon. It was amazing to see what you can get done in 24 hours.

Turtletaub: My favorite aspect was being able to meet people interested in quantum computing from all across the world and being able to collaborate with some of the top researchers and engineers at IBM.

Abbas: Hands down, my favourite aspect of the hackathon was the people. Coming from South Africa, I was really worried I wouldn’t fit in or be good enough because I’m just a master’s student from the University of KwaZulu-Natal with no undergraduate experience in physics. But as soon as I arrived, I realised that the intention of others at the camp wasn’t to undermine others’ capabilities or differences, but to highlight them and use them to build beautiful applications with Qiskit. There were people from all types of backgrounds with differing levels of experience, and all so helpful, open and keen to learn. I was blown away by the creativity of the projects and I am convinced that the world of quantum computing has a very bright future if these are some of the individuals contributing to it.

Quantum Programming A New Approach to Solve Complex Problems Francisco Gálvez Ramirez IBM Staff fjgramirez@es.ibm.com

More Information:

https://www.ibm.com/quantum-computing/learn/what-is-quantum-computing

https://www.ibm.com/quantum-computing/learn/what-is-ibm-q/

https://www.ibm.com/quantum-computing/technology/systems/

https://developer.ibm.com/code/videos/qiskit-quantum-computing-tech-talk/

https://developer.ibm.com/dwblog/2017/quantum-computing-api-sdk-david-lubensky/

https://www-03.ibm.com/press/us/en/pressrelease/51740.wss

https://developer.ibm.com/tutorials/os-quantum-computing-shell-game/

QISKit OPpenSource

https://developer.ibm.com/components/

https://medium.com/qiskit/project-highlight-hybrid-quantum-classical-machine-learning-e5319982e3b1

https://community.qiskit.org/textbook/preface

https://delapuente.github.io/qiskit-textbook/preface

https://qiskit.org/documentation/install.html

https://qiskit.org/

https://quantum-computing.ibm.com/support