Cloud Computing For Big Data¶
Definition of Cloud Computing¶
Cloud computing is a model for delivering on-demand access to shared computing resources (such as servers, storage, applications, and services) over the internet. Big data refers to large and complex datasets that cannot be processed or analyzed using traditional data processing methods. Combining cloud computing with big data technologies can help businesses to manage, store, process and analyze their large datasets.
Cloud storage is often discussed separately because storage technologies have their own design trade-offs, such as latency, durability, consistency, and data layout. However, in modern cloud platforms the boundary is no longer clean: storage services are frequently coupled with metadata catalogs, event notifications, lifecycle policies, security policies, and managed analytics services. For that reason, it is useful to treat cloud storage as a distinct topic while also recognizing that it is deeply integrated into the broader cloud computing platform.
Characteristics¶
On-demand self-service: With on-demand self-service, users can access cloud computing resources and services quickly and easily, without needing to go through IT personnel. For example, a business may use a cloud computing service like Amazon Web Services (AWS) to spin up a cluster of virtual machines to process a large dataset. With just a few clicks, the business can provision the resources it needs and start processing data right away.
Broad network access: Broad network access allows users to access cloud computing resources and services over the internet, enabling them to work from anywhere using any kind of devices. For example, a data scientist may need to access a large dataset stored in the cloud from their home office or while traveling. With cloud computing, they can access the data from anywhere with an internet connection.
Resource pooling: Resource pooling enables multiple users to share computing resources and services, making it easier to scale up or down as demand changes. For example, a business may use a cloud computing service like Microsoft Azure to store and process large amounts of data. With resource pooling, the business can easily scale up or down its computing resources as needed, without needing to buy or maintain its own hardware.
Rapid elasticity: Rapid elasticity enables users to quickly scale up or down their computing resources and services as needed. For example, a business may need to process a large amount of data for a short period of time, such as during a sales promotion. With cloud computing, the business can quickly provision the additional resources it needs to handle the workload, and then de-provision them when the promotion is over.
Measured service: Measured service provides users with visibility into their usage of computing resources and services, enabling them to monitor and control costs more effectively. For example, a business may use a cloud computing service like Google Cloud Platform to store and process large amounts of data. With measured service, the business can track its usage of computing resources and services, and then adjust its usage to optimize costs.
Advantages of cloud computing¶
Potentially cost-effective: One of the key advantages of cloud computing is that it can reduce the upfront costs associated with purchasing and maintaining hardware and software infrastructure. Instead, organizations can pay for computing resources and services on demand. This is often more economical for workloads with variable demand or for teams that want to start quickly without large capital expenses. However, cloud costs must be managed carefully because always-on resources, large-scale storage, and data transfer charges can still become expensive. For example, a business may use a cloud-based file storage service like Dropbox or Google Drive instead of maintaining its own file servers, which can reduce hardware, energy, and maintenance costs.
Scalable: Cloud computing is highly scalable, which means that businesses can easily adjust their computing resources and services to meet changing demands. With cloud computing, businesses can quickly provision additional computing resources when they need them and then de-provision them when they no longer need them. This can be particularly useful for businesses that experience spikes in demand, such as during peak sales periods or when launching new products. For example, a business may use a cloud-based e-commerce platform like Shopify or Magento to quickly scale up its online store during a busy holiday season.
Flexible: Cloud computing is highly flexible, which means that businesses can choose the computing resources and services that best meet their needs. With cloud computing, businesses can choose from a wide range of computing resources and services, including storage, processing power, networking, and more. Additionally, businesses can easily switch between different cloud providers or services as their needs change. For example, a business may use a cloud-based email service like Gmail or Outlook instead of maintaining its own email server, which can provide greater flexibility and scalability.
Reliable: Cloud computing platforms are designed for high availability and often provide built-in redundancy, backup, and disaster recovery options. In practice, reliability depends on architecture choices such as using multiple availability zones, regions, or failover strategies. Cloud providers can reduce operational burden, but outages still happen, so mission-critical systems should be designed for resilience rather than assuming the provider will eliminate downtime.
Secure when managed correctly: Major cloud providers offer strong security controls such as encryption, identity and access management, network isolation, logging, and compliance tooling. However, cloud security follows a shared-responsibility model: the provider secures the cloud platform, and the customer is still responsible for securing data, identities, applications, and configuration. Misconfiguration remains one of the most common causes of cloud security incidents.
Drawbacks of cloud computing¶
Cost: While cloud computing can be cost-effective, it can also be expensive in many cases.
Vendor lock-in and egress costs: Cloud platforms make it easy to adopt managed services, but moving large datasets or tightly coupled workloads to another provider can be difficult and expensive.
Data privacy: When you store your data in the cloud, you are entrusting it to a third-party provider, which can raise concerns about data privacy and security.
Downtime: Cloud computing services can experience downtime, which can disrupt business operations and cause data loss. Cloud providers typically have service level agreements (SLAs) that guarantee a certain level of uptime, but downtime can still occur.
Security concerns: If your data is stored in the cloud, it is vulnerable to cyber attacks and data breaches. Cloud providers have implemented security measures to protect your data, but there is always a risk of unauthorized access or data loss.
Cloud computing service models¶
Infrastructure as a Service (IaaS): IaaS is a cloud computing model that provides businesses with virtualized computing resources, such as virtual machines, storage, and networking. With IaaS, businesses can easily provision and manage their own computing infrastructure, without the need to purchase and maintain physical hardware. Examples of IaaS providers include Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP).
Platform as a Service (PaaS): PaaS is a cloud computing model that provides businesses with a platform for developing, deploying, and managing their own applications. With PaaS, businesses can focus on building and delivering applications, without the need to manage the underlying infrastructure. PaaS providers typically provide pre-built tools and services for application development, such as development frameworks, databases, and testing tools. Examples of PaaS providers include Google App Engine, Microsoft Azure App Service, and Red Hat OpenShift.
Software as a Service (SaaS): SaaS is a cloud computing model that provides businesses with access to pre-built software applications that are hosted in the cloud. With SaaS, businesses can easily access and use software applications without the need to install or manage the software on their own devices or servers. SaaS applications can include a wide range of software, such as customer relationship management (CRM) tools, project management software, and email services. Examples of SaaS providers include Salesforce, Dropbox, and Microsoft 365.
Functions as a Service (FaaS): FaaS is a cloud computing model that provides businesses with serverless computing resources, which are used to run individual functions or pieces of code. With FaaS, businesses can focus on building and deploying specific functions or services, without the need to manage the underlying infrastructure. FaaS providers typically charge businesses based on the number of times their functions are executed or the amount of time they are executed for. Examples of FaaS providers include AWS Lambda, Google Cloud Run functions, and Microsoft Azure Functions.
Cloud Computing for Big Data Today¶
In modern big data platforms, cloud computing is not only about renting virtual machines. Organizations increasingly rely on managed services for storage, data processing, orchestration, streaming, and analytics. Instead of building every cluster manually, teams often combine object storage, managed Spark services, data warehouses, message brokers, and workflow tools.
This shift changes how big data systems are designed:
Object storage systems such as Amazon S3, Azure Data Lake Storage, and Google Cloud Storage are often the default foundation for data lakes.
Cloud storage is no longer just a passive place to keep files. It commonly participates in the pipeline through event triggers, lifecycle management, access policies, metadata integration, and table formats such as Iceberg or Delta Lake that make stored data directly usable by analytics engines.
Compute is increasingly elastic and short-lived, with clusters or serverless jobs started only when needed.
Managed services reduce operational overhead, but they can increase vendor dependence and require closer attention to pricing, quotas, and data governance.
For big data work, cloud architecture decisions are now closely tied to cost control, data locality, security policy, and interoperability across storage, streaming, and analytics services.
Separating Cloud Storage from Cloud Computing¶
For teaching purposes, it still makes sense to discuss cloud storage and cloud computing separately:
Cloud storage focuses on how data is organized, persisted, versioned, governed, and retrieved.
Cloud computing focuses on how processing resources and managed services are provisioned and coordinated.
The important modern caveat is that these layers are tightly coupled. A file written into cloud object storage may trigger a serverless function, update a catalog, become part of a lakehouse table, and then be queried by a warehouse or Spark job. In other words, storage remains a separate architectural layer, but in cloud systems it is often the control point for downstream compute.