大數據Hive離線計算開發實戰

ISBN：9787115448088

資深的資訊工程師常有一種特別的煩惱，如何在眾多實作技術中，挑選出最恰當的工具來執行軟體專案。從中牽扯的包括組織資源、同儕能力、工具成熟度、操作便易性，這項任務恰好沒有一體適用的標準答案，如同撰寫程式時「變數命名」般，十足仰賴負責人的主觀決擇。

大數據專案開發也會遇到一樣的問題，在今日開源軟體、雲端平台遍地開花的時代，要收集、分析與運用企業本身的巨量資料，可以從最精典的Map/Reduce框架到高效版本的Spark平台。不論企業最後挑選的大數據解決方案為何，勢必需要熟悉該平台的工程師運籌帷幄。

由於大數據開發有別於傳統的關聯式資料庫，因此對於工程團隊來說，大數據技術變成一個得跨越過去的門檻。而Apache Hive正是主張能一勞永逸解決這個技術「痛點」的開源工具。它無縫地運行在Hadoop平台之上，充分利用HDFS的分散式檔案系統以及Yarn平行處理框架，並在自身的對外介面套上一層相容SQL(HQL)的外衣，讓各項軟體工具、運用程式甚至是開發人員，都能用熟練的SQL語法，遨遊於巨量資料的國度中。

對我而言，由於已經熟練Map/Reduce與Spark等技術，可以隨心所欲地撰寫出大數據程式碼。其實沒必要疊床架屋的再添一層虛擬資料庫，用SQL語法來執行巨量運算。但考量到企業整體運用情境，想擴增大數據的應用面以及簡化同儕的數據使用門檻，就值得投入心力建置一個獨立運作的Hive環境。

市面上Hive的專書不多，而且絕大多數是四、五年前所出版。在決定再次架設Hive服務的當下，理所當然地挑選2020年出版的「大數據Hive離線計算開發實戰」。內文從最基礎的概念說起，再由簡入深地提及到分區、自定義函式等進階的功能，作者將Hive定位為能處理大數據的資料倉儲解決方案，全書也呼應這個想法具體地呈現如何實踐。閱後不免心有所思，在這麼強大便利的開源方案的競爭下，未來商用資料倉儲廠商肯定不好擴展市場。

作者用最後三章舉出大數據的常見運用情境，分別是推薦系統、網站流量分析以及商業數據分析。題目夠大但內容卻僅是入門級的基本範例。特別是推薦系統一章中，是用Mahout來執行協同過瀘演算法來做範例。先不論將推薦系統，用單一個協同過瀘法來涵蓋，是否會過於簡化問題以致誤導讀者。書中介紹的Mahout版本(運行在Hadoop 2.x上)，已經是太舊的版本，令人懷疑這本書的出版年份是否有誤植？

總得來說本書可以讓讀者快速領略Hive在大數據平台中所能扮演的角色，免除耗費心力瀏覽散落各處的官網文件，最後還落得一頭霧水。如果動心起念想要著手大數據運算，Hive和這本「大數據Hive離線計算開發實戰」算是不錯的起點。

Senior software engineers often face a particular kind of dilemma: how to choose the most appropriate tools from a vast array of implementation technologies for a software project. The decision involves considerations such as organizational resources, team capabilities, tool maturity, and ease of use. Unfortunately, there is no universally applicable answer to this challenge. Much like “naming variables” in programming, the final decision relies heavily on the subjective judgment of the person in charge.

Big data projects encounter the same issue. In today’s era of flourishing open-source software and cloud platforms, enterprises seeking to collect, analyze, and leverage massive volumes of data can choose from everything from the classic MapReduce framework to high-performance platforms such as Spark. Regardless of which big data solution an organization ultimately adopts, it will inevitably require engineers who are proficient in that platform to guide and manage the project.

Because big data development differs significantly from traditional relational databases, mastering big data technologies has become a hurdle engineering teams must overcome. Apache Hive is an open-source tool that claims to solve this technical “pain point” once and for all. It operates seamlessly on top of the Hadoop ecosystem, fully leveraging HDFS as a distributed file system and YARN as a parallel processing framework. On top of that, Hive provides an SQL-compatible interface (HQL), allowing software tools, applications, and even developers themselves to navigate the world of big data using the familiar syntax of SQL.

From my perspective, since I am already proficient in technologies such as MapReduce and Spark, I can freely develop big data applications directly. Strictly speaking, there is no real need to add another layer of abstraction in the form of a virtual database and execute large-scale computations through SQL. However, when considering enterprise-wide adoption, expanding the accessibility of big data applications and lowering the barrier for colleagues to work with data make it worthwhile to invest in building an independently operated Hive environment.

There are not many books on Hive available on the market, and most of them were published four or five years ago. Therefore, when I decided to set up a Hive service once again, it was only natural to choose the 2020 publication Big Data Hive Offline Computing Development in Practice. The book begins with the most fundamental concepts and gradually progresses toward advanced features such as partitioning and user-defined functions. The author positions Hive as a data warehouse solution capable of handling big data, and the entire book consistently demonstrates how to implement that vision in practice. After finishing the book, one cannot help but reflect that under the competition of such powerful and convenient open-source solutions, commercial data warehouse vendors will likely find it increasingly difficult to expand their markets in the future.

In the final three chapters, the author presents several common big data application scenarios, including recommendation systems, website traffic analysis, and business data analytics. Although the topics themselves are broad, the actual content remains at an introductory level with only basic examples. In particular, the chapter on recommendation systems uses Mahout to demonstrate collaborative filtering algorithms. Setting aside the question of whether using only a single collaborative filtering approach oversimplifies recommendation systems and potentially misleads readers, the version of Mahout introduced in the book (running on Hadoop 2.x) is already severely outdated. This raises doubts as to whether the publication year of the book may have been printed incorrectly.

Overall, this book allows readers to quickly grasp the role Hive can play within a big data platform, sparing them the effort of piecing together scattered official documentation only to end up more confused. For anyone interested in getting started with big data computing, both Hive and Big Data Hive Offline Computing Development in Practice serve as solid starting points.

演算法圖鑑：26種演算法 + 7種資料結構，人工智慧、數據分析、邏輯思考的原理和應用全圖解

依舊記得在自學程式的過程中，看過某本電腦書上寫著「程式=資料結構+演算法」，簡單又有說服力，便對「演算法」這詞彙留下深刻印象。那時真心覺得要成為一個程式高手，肯定得在演算法下苦功。更因此到某大學的資訊工程研究所在職專班，修習「計算機演算法」學分，原文課本唸得我嫑嫑的。

隨著工作經驗的與日俱增，漸漸地不再迷信演算法對工作實務的重要性。不過這是因為我從事的是應用系統(AP)的設計/開發，即便後期主要在做大數據/機器學習的應用，也是直接受惠於高階程式語言和各種開源函式庫/架框。這領域的工程師已經很少花費心力在處理底層運算中枝微末節的效能議題，像Big O這個評量算法複雜性的衡量單位，真的在職場上沒遇到有同事拿來相互討論。

這不表示程式設計師可以不用瞭解精典演算法背後的概念。列表(List)、陣列(Array)、堆疊(Stack)、佇列(Queue)等資料結構，是各種程式語言中必備的核心類別/元件。而排序(Sort)算法則是實際開發時最常被使用到的基礎算法，瞭解不同排序算法的優缺點，會大大地改善程式執行的效能。因此「資料結構」和「排序算法」這兩章可視為程式工程師必備的業務知識(Domain Knowledge)。

陣列搜尋、圖形搜尋、分群以及最後一章的「其它演算法」則是各擅勝場，在特定應用領域中都是最廣為人知的核心演算法。最值得一提的是第5章介紹的「安全性演算法」，它從最基本的數位加密觀念開始介紹，一步步地擴展到「共用金鑰密碼系統」和「公開金鑰密碼系統」，而後者正是目前網站加密技術的核心。「數位簽章」和「數位憑證」說明了HTTPS的運作原理，更是網頁程式設計師不可遺漏的業務知識。

嚴格說，書名副標提到的「人工智慧、數據分析」有點言過其實。所提到的資料結構和演算法，對於數據分析、機器學習/人工智慧領域，都是不足掛齒。本書適合給有志成為程式設計師的人當做學習教材，抑或是程式設計同業用來複習或做為消遣的小讀物。在主旨為講解計算機演算法的書海中，這本「演算法圖鑑」是最容易讓讀者吸收的傑作。