Counting 3,834 Big Data & Machine Learning Frameworks, Toolsets, and Examples...
Suggestion? Feedback? Tweet @stkim1

Author
Last Commit
May. 24, 2019
Created
Feb. 6, 2019

Amundsen

PyPI version Build Status Coverage Status License PyPI - Python Version PRs Welcome Slack Status

Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data. It does that today by indexing data resources (tables, dashboards, streams, etc.) and powering a page-rank style search based on usage patterns (e.g. highly queried tables show up earlier than less queried tables). Think of it as Google search for data. The project is named after Norwegian explorer Roald Amundsen, the first person to discover South Pole.

It includes three microservices and a data ingestion library.

  • amundsenfrontendlibrary: Frontend service which is a Flask application with a React frontend.
  • amundsensearchlibrary: Search service, which leverages Elasticsearch for search capabilities, is used to power frontend metadata searching.
  • amundsenmetadatalibrary: Metadata service, which leverages Neo4j or Apache Atlas as the persistent layer, to provide various metadata.
  • amundsendatabuilder: Data ingestion library for building metadata graph and search index. Users could either load the data with a python script with the library or with an Airflow DAG importing the library.

Requirements

  • Python >= 3.4
  • Node = v8.x.x or v10.x.x (v11.x.x has compatibility issues)
  • npm >= 6.x.x

User Interface

Please note that the mock images only served as demonstration purpose.

  • Landing Page: The landing page for Amundsen including 1. search bars; 2. popular used tables;

  • Table Detail Page: Visualization of a Hive / Redshift table

  • Column detail: Visualization of columns of a Hive / Redshift table which includes an optional stats display

  • Data Preview Page: Visualization of table data preview which could integrate with Apache Superset

Get Involved in the Community

Want help or want to help? Use the button in our header to join our slack channel. Please join our mailing list as well.

Getting started

Please visit the Amundsen documentation for help with installing Amundsen and getting a quick start with dummy data or an overview of the architecture.

Architecture Overview

Please visit Architecture for Amundsen architecture overview.

Installation

Please visit Installation guideline on how to install Amundsen.

Configuration

Please visit Configuration doc on how to configure Amundsen various enviroment settings(local vs production).

Developer Guidelines

Please visit Developer guidelines if you want to build Amundsen in your local environment.

Roadmap

Please visit Roadmap if you are interested in Amundsen upcoming roadmap items.

Publications

License

Apache 2.0 License.

Latest Releases
Add NPM installation into Docker file
 May. 20 2019
1.0.5
 May. 13 2019
Publish to Dockerhub
 May. 10 2019
Publish to Dockerhub
 May. 10 2019
Release to Dockerhub
 May. 10 2019