What is a successful Data Warehouse (Part 2)?

In Data Warehousing, the perception of success is different between users and engineers. In part 2, let’s talk about the success factors from the point of view of the data engineer.

  • Low starting cost, Grow when needed, Prove then spend
  • Deliver often, Discover issues early (before it is too late to change course)
  • Load the raw data (without tranasformations)
  • Load the transformed data (using the loaded raw data and business rules)
  • Load structured data and unstructured data (big or small)
  • 100% loaded, no suspension process, nothing left behind
  • History tracking, when needed
  • Decouple collection, storage and presentation
  • Efficient storage, scalable
  • Fast load, Parallel load, Append-Only, Immutable
  • Fast queries, Fast tuning, Fast improvements
  • Append-only database changes (keep the existing queries/reports running)
  • Support any presentation style (normalized, dimensional, big flat tables, custom)
  • Support multiple presentation tools (no commitment to one vendor)
  • Support automation and code generation
  • Avoid technical debt, achieve sustainable development
  • Use engineering practices and repeatable patterns
Written on October 20, 2016