← Back to Hub ← 返回主页

Week 5: DDP Implications & Assessment 1 第 5 周:DDP 网络影响与评估 1

Core Concepts 核心概念

1. Networking Implications of DDP (3.4)

Distributed Data Processing relies heavily on a robust network backbone. Key challenges include maintaining consistent directory services (identifying where files and databases reside), managing synchronization traffic, handling concurrency control, and ensuring network link redundancy to support high-availability.

2. Big Data Infrastructure (3.5)

Big Data environments require highly scalable distributed architectures to handle the "3 Vs" (Volume, Velocity, Variety). Infrastructure components include distributed file systems (like HDFS or cloud object stores), high-throughput ingestion pipelines (like Kafka), and distributed query processing frameworks (like Spark or MapReduce) that partition execution workloads across clusters of servers.

1. DDP 的网络影响 (3.4)

分布式数据处理高度依赖强大的网络主干。关键挑战包括维持一致的目录服务(识别文件和数据库所在位置)、管理同步流量、处理并发控制以及确保网络链路冗余以支持高可用性。

2. 大数据基础设施 (3.5)

大数据环境需要高度可扩展的分布式架构来处理“3 个 V”(量 Volume、速度 Velocity、多样性 Variety)。基础设施组件包括分布式文件系统(如 HDFS 或云对象存储)、高吞吐量摄取管道(如 Kafka)以及分布式查询处理框架(如 Spark 或 MapReduce),这些框架在服务器集群中划分执行工作负载。

Concept Visualization: Big Data Infrastructure Pipeline 概念可视化:大数据基础设施管道

Ingestion (Kafka) Storage (HDFS) Process (Spark) BI Dashboard

Tips 提示

  • Bottleneck Identification: In distributed data tasks, a network bottleneck occurs when network transmission speed (link capacity) limits system throughput, keeping server CPUs idle while waiting for data.
  • Batch vs. Stream Ingestion: Batch processing (Hadoop MapReduce) is suited for massive historical datasets analyzed overnight, while Stream processing (Apache Spark/Flink) handles real-time feeds (IoT, transactions) in milliseconds.
  • 瓶颈识别:在分布式数据任务中,当网络传输速度(链路容量)限制系统吞吐量时,就会发生网络瓶颈,使服务器 CPU 在等待数据时处于闲置状态。
  • 批处理 vs. 流式摄取:批处理(Hadoop MapReduce)适用于夜间分析的海量历史数据集,而流处理(Apache Spark/Flink)可在几毫秒内处理实时数据(物联网、交易)。

Practical Exercises & Step-by-Step Guide 实践练习与分步指南

Exercise 1: Network Bottleneck Analysis & EtherChannel Solution 练习 1:网络瓶颈分析与以太通道 (EtherChannel) 解决方案

Steps to complete:

  1. Open Cisco Packet Tracer. Place two 2960 Switches (SwitchA and SwitchB) side by side.
  2. Connect them using two Copper Cross-Over cables between ports FastEthernet0/1 and FastEthernet0/2. By default, Spanning Tree Protocol (STP) will block one port to prevent loops, limiting bandwidth to 100 Mbps.
  3. Bundle the interfaces using EtherChannel (LACP) to double the interface capacity to 200 Mbps and provide active load balancing.
  4. Configure EtherChannel on SwitchA and SwitchB. Both ports will turn green and act as a single logical trunk link.

完成步骤:

  1. 打开 Cisco Packet Tracer。并排放置两台 2960 交换机(SwitchA 和 SwitchB)。
  2. 使用两根 铜交叉线 连接它们,接口使用 FastEthernet0/1 和 FastEthernet0/2。默认情况下,生成树协议 (STP) 将阻塞其中一个接口以防止环路,可用带宽限制在 100 Mbps。
  3. 通过配置以太网通道 (EtherChannel - LACP) 将这两个接口捆绑在一起,使可用带宽增加一倍至 200 Mbps,并提供主动负载均衡。
  4. 在 SwitchA 和 SwitchB 上输入配置命令。绑定成功后,两个端口都将变成绿色,并作为一个逻辑中继链路工作。

Continuous Assessment 1 (10% Weight) 持续性阶段评估 1 (权重 10%)

Covers: Chapters 1, 2, and 3 (10 Questions) 范围:第 1、2 和 3 章(共 10 题)

1. Which data flow mode is unidirectional?哪种数据流模式是单向的?

2. LAN stands for:LAN 代表:

3. Which topology uses a central hub?哪种拓扑结构使用中央集线设备/中心节点?

4. Data communication requires a:数据通信必须要有一个:

5. Which of the following is a physical connection line configuration type?以下哪项属于物理线路连接配置类型?

6. Information is defined as:信息被定义为:

7. Protocols are:协议是:

8. Which international organization officially publishes standards like OSI?哪个国际组织正式发布了诸如 OSI 等标准?

9. Distributed data processing involves:分布式数据处理涉及到:

10. Timeliness in data quality refers to data:数据质量中的“时效性”是指数据:

Vocabulary Matching Game 词汇配对游戏

Match the Big Data/DDP term on the left with its definition on the right. (Click a term, then click its matching definition). 将左侧的大数据/DDP 术语与右侧的定义进行配对。(点击术语,然后点击匹配的定义)。

Kafka
Spark
Network Bottleneck
HDFS
High-throughput distributed storage system for massive files 用于海量文件的高吞吐量分布式存储系统
Data ingestion queue for real-time streaming pipelines 用于实时数据流管道的高效数据摄取队列
Constraint where network link capacity limits CPU compute speed 网络链路吞吐容量限制 CPU 计算速度的瓶颈约束
High-speed distributed memory-based query processing framework 基于内存的高速分布式查询与运算处理框架