Apache Hop: metadata-driven data integration platform

1 min read
etldata-integrationorchestrationopen-source
View as Markdown
Originally from hop.apache.org
View source

My notes

Summary

Apache Hop (Hop Orchestration Platform) is an open-source, metadata-driven data integration platform for building ETL/orchestration pipelines visually via drag-and-drop. Pipelines are “design once, run anywhere”: the same workflow runs on Hop’s native engine (local/remote) or on Spark, Flink, Google Dataflow, and AWS EMR through Apache Beam. It is an Apache Software Foundation project under the Apache v2 license.

Key Insight

  • Successor to Pentaho Kettle/PDI in spirit: Hop is a from-scratch rewrite aimed at replacing code-heavy ETL with visual, metadata-driven pipelines. Everything (every object type) is metadata describing how data is read, manipulated, written, or orchestrated.
  • Design once, run anywhere is the real differentiator: one visual pipeline can execute on the native Hop engine or be pushed to Spark, Flink, Google Dataflow, or AWS EMR via the Apache Beam abstraction. No rewrite per runtime.
  • Kernel + plugin architecture: a small robust engine, with all functionality added through plugins that bring their own metadata. This makes it extensible without touching core.
  • Full lifecycle management in the GUI: switch between projects, environments, runtime configs, and manage git versions without leaving the tool. Environment-aware config solves the classic dev/test/prod ETL promotion pain.
  • Vendor-neutral, ASF-governed: Apache v2 license, open mailing lists, issue tracker, and contribution path. Low lock-in risk vs commercial ETL suites. Current release: 2.18.1 (June 2026).