ISBN：9787115524669
軟體工程師在接案公司和一般公司裡的工作模式截然不同。在公司的資訊部門任職的工程師，除了應接不暇的開發工單外，勢必也得分心維護既有系統，畢竟這些神奇的小玩意兒，有可能就是自己費盡心力打造而成的。而隨著年資增加，參與的專案系統也逐漸多到連雙手手指都數不完，此時維護系統就變成了記憶力大考驗。也難怪《論語》說：「溫故而知新，可以為師矣。」在資訊業界，在不斷學習新事物的同時，還要牢記已經學過的舊技術，真的不簡單。
使用 Hadoop 大數據平台已經十年了，期間從 2.x 升級到 3.x，也把 Apache Hadoop 換成 Apache Ambari，後續平台也逐漸步入穩定狀態，很少進行大幅度的調整或異動。但不巧的是，Apache Ambari 母公司遭併購後，將軟體改為商業授權才能安裝；再加上大數據應用遍地開花，既有平台的胃納量也已略顯吃緊，可預見未來有必要再安裝另一套 Apache Hadoop。因此便著手翻閱《大數據 Hadoop 3.X 分佈式處理實戰》，當作精進與複習技術之用。
以 Hadoop 為核心的生態系，相關軟體族繁不及備載，寫成三本書大概也寫不完。幸好這本書的作者並不貪心，只挑選生態系中最成熟、實用且具有代表性的軟體收錄其中。有 NoSQL 的 HBase、資料倉儲的 Hive，以及 MapReduce 框架的後繼者 Spark，都是一時之選，恰好多數也是目前有用到的解決方案。書中除了基礎概念解說、安裝教學外，也附上有如 Hello World 的簡易程式，提供讀者動手練習。本來以為只要按部就班地翻閱，應該花不了多少時間就能看完整本書，但代誌不是憨人想的那麼簡單。
即便書中也特別強調採用的是 Hadoop 3.x 版本，讓讀者在操作上不用苦惱書中版本與主流版本之間的差異，但在實際安裝的過程中，我還是遇到了 Hadoop 生態系中典型的版本迷宮，在裡面繞來繞去好幾天。主因是新版 Hive 將執行引擎改成依賴 Tez，而 Tez 的版本更新看似已經停滯，下載到的版本只能在 JDK 8 上執行。一旦決定主機上安裝的是 JDK 8，便會影響到 Hadoop、HBase 和 Hive 可安裝的版本。整個排列組合下來，真正能成功同時跑起來的版號組合，大概也只有一種。不幸的是，書裡介紹的版本屬於舊版，因此無法成功運作，讀者只能靠自己努力，用試誤的方法找出黎明的曙光。
最後，書中也附了一些 MapReduce 的應用程式範例，提供讀者練習並體會大數據程式的運作模式。包括最常見且實用的網站 Web Log 分析，以及打造推薦系統的協同過濾法，還有機器學習的垃圾訊息分類器。除了 Web Log 的程式算是中規中矩外，其餘兩個範例都只能僅供參考，千萬別直接拿來應用或仿效。不論是協同過濾，還是分類器所使用的貝氏定理，書中的程式碼邏輯都不完全正確，只具其形而未得其神。更別說 Spark 早已內建機器學習相關功能，再回頭介紹如何用 MapReduce 硬刻演算法，多少有些誤人子弟。如果將它視為學習大數據平台的教材之一，再搭配其他同主題書籍交互參照，《大數據 Hadoop 3.X 分佈式處理實戰》仍是可以考慮入手的一本書。但如果只打算靠一本書學習 Hadoop 技術，這本書未必是最佳選擇。
Software engineers work very differently in consulting firms and in-house IT departments. Engineers working within a company's IT team not only have to deal with a constant stream of development requests, but also need to maintain existing systems. After all, many of these "little wonders" were painstakingly built by their own hands. As the years go by, the number of projects and systems one has participated in gradually grows beyond what can be counted on two hands. At that point, system maintenance becomes a true test of memory. No wonder Confucius wrote in *The Analects*: “Reviewing what has been learned and gaining new insights makes one qualified to teach others.” In the IT industry, continuously learning new technologies while retaining knowledge of older ones is no easy task.
I have been using the Hadoop big data platform for nearly ten years. During that time, I upgraded from Hadoop 2.x to 3.x and migrated from Apache Hadoop's traditional management approach to Apache Ambari. As the platform matured, major changes became increasingly rare. Unfortunately, after Apache Ambari's parent company was acquired, the software became commercially licensed and could no longer be freely installed. At the same time, big data applications continued to proliferate, and the capacity of the existing platform was beginning to show its limits. It seemed inevitable that another Hadoop cluster would eventually be required. With that in mind, I picked up *Big Data Hadoop 3.X Distributed Processing in Practice* as both a refresher and an opportunity to further sharpen my skills.
The Hadoop ecosystem is vast, with so many related projects that even three books would hardly be enough to cover them all. Fortunately, the author does not attempt to cover everything. Instead, the book focuses on some of the most mature, practical, and representative components in the ecosystem. These include HBase for NoSQL storage, Hive for data warehousing, and Spark, the successor to the MapReduce framework. Coincidentally, these are also among the solutions I currently use in production. In addition to explaining fundamental concepts and installation procedures, the book includes simple “Hello World”-style examples that allow readers to gain hands-on experience. Initially, I assumed that reading through the book step by step would be fairly straightforward and wouldn't take much time. As it turned out, things were not nearly that simple.
Although the book specifically states that it uses Hadoop 3.x and attempts to minimize the gap between its examples and mainstream versions, I still found myself trapped in one of the Hadoop ecosystem's classic challenges: version compatibility. I spent several days navigating what felt like an endless maze. The primary issue is that newer versions of Hive rely on Tez as their execution engine, while Tez development appears to have largely stagnated. The available versions of Tez can only run on JDK 8. Once JDK 8 is chosen, the versions of Hadoop, HBase, and Hive that can be installed become constrained as well. After considering all possible combinations, there appears to be only one set of versions that can successfully run together. Unfortunately, the versions introduced in the book are outdated and no longer work correctly. Readers are therefore left to rely on trial and error to find a compatible configuration on their own.
The book also includes several MapReduce application examples designed to help readers understand how big data programs operate. These cover common and practical topics such as web log analysis, collaborative filtering for recommendation systems, and a machine-learning-based spam classifier.
While the web log example is reasonably implemented, the other two should be treated strictly as references and not copied directly into real-world projects. Whether discussing collaborative filtering or the Bayesian theorem used in the classifier, the code examples fail to fully capture the underlying principles. They resemble the concepts superficially but miss much of their essence. Furthermore, Spark has long provided built-in machine learning capabilities. Revisiting algorithm implementation through raw MapReduce feels somewhat outdated and may even mislead newcomers. If viewed as one of several learning resources for understanding big data platforms—and supplemented with other books covering similar topics—*Big Data Hadoop 3.X Distributed Processing in Practice* remains a worthwhile addition to one's bookshelf. However, for readers hoping to learn Hadoop from a single book alone, this may not be the best choice available.
