Sahara Project Onboarding Telles Nobrega Sahara Overview Big
Sahara Project Onboarding Telles Nobrega
Sahara Overview Big Data Processing as a Service ● Big Data processing framework provision ○ Ambari ○ Cloudera ○ Vanilla (upstream hadoop) ○ Map. R ○ Spark ○ Storm ● EDP (Elastic Data Processing) ○ Running jobs on those frameworks
Sahara Overview Namings ● Services ○ Specific roles of a cluster instance ● Node Group Templates ● ● ○ Describes a group of nodes within a cluster Cluster Templates ○ Describes a group of Node Group templates to form a cluster Job binary ○ Job executable (jar, . py) Job template ○ Describes a job to be run Data Sources ○ Source used pull data into Sahara or store data out of Sahara
Sahara Overview Image Generation ● Sahara Image Elements ○ tox -e venv -- sahara-image-create -p spark -s [1. 3. 1|1. 6. 0|2. 1. 0|2. 2. 0] # spark standalone ○ tox -e venv -- sahara-image-create -p vanilla -v 2. 7. 1 -s [1. 6. 0|2. 1. 0|2. 2. 0] # spark on vanilla ● Sahara Image Pack ○ tox -e image -- sahara-image-pack --image Cent. OS. qcow 2 --config-file etc/sahara. conf cdh 5. 7. 0 [cdh 5. 7. 0 specific arguments, if any]
Architecture
Architecture
Sahara Repos Current ● ● ● sahara - https: //git. openstack. com/openstack/sahara-image-elements - https: //git. openstack. com/openstack/sahara-image-elements python-saharaclient - https: //git. openstack. com/openstack/python-saharaclient sahara-dashboard - https: //git. openstack. com/openstack/sahara-dashboard sahara-extra - https: //git. openstack. com/openstack/sahara-extra Planned ● sahara-plugins
Running Sahara # first terminal $ sahara-venv/bin/sahara-api --config-file sahara-venv/etc/sahara. conf # second terminal $ sahara-venv/bin/sahara-engine --config-file sahara-venv/etc/sahara. conf
Running Sahara Tests Unit Tests (from sahara) ● ● tox -e py 27 tox -e py 35 tox -e pep 8. . . Scenarios (From sahara-tests) ● sahara-scenario {posargs} Tempests ● ● Install sahara tempest plugin from sahara-tests Follow the default tempest process (see tempest docs)
Sahara Docs ● ● https: //docs. openstack. org/sahara/latest/ https: //docs. openstack. org/sahara-tests/latest/ https: //docs. openstack. org/sahara/latest/reference/restapi. html https: //developer. openstack. org/api-ref/data-processing/
- Slides: 10