Porting data workflow management code to Python 3

Official task: The candidate will work with the development teams to prepare dmwm code

1 st task: Create dependencies tree from. spec files: Read all *. spec files

2 nd task: Cheetah to Jinja 2 template converter Cheetah is no longer supported

Cheetah: Jinja 2: #from DAS. web. utils import quote {% set item = quote(item)

3 rd task: Get all dmwm dependencies 3 million lines of code Find all

Crab. Clients has ~ 500 imports. DAS ~ 1700 After analysis of DAS: 26

4 th task: code porting Started from mox -> mox 3 (more about in

5 th task: find unused code While porting mox found “weird” looking classes They

Failed attempts. . . Sequence. Matcher… Comparing code and if match is more than

6 th task: Write unit test for backends https: //github. com/dmwm/WMCore/tree/master/src/python/W MCore/Storage backends vs

7 th task: reqmgr 2 GUI improvement Clean CSS and add dropdowns Update templates

Slides: 13

Download presentation

Porting data & workflow management code to Python 3 Made by: Adelina Varatinskaite

Official task: The candidate will work with the development teams to prepare dmwm code for python 3. x, beginning with an audit of the suitability of the external software used and identifying replacements where needed. Automated tools will also be employed to help with this porting In other words: in 2020 python 2. x will be no longer supported

1 st task: Create dependencies tree from. spec files: Read all *. spec files from: tree_comp_gcc 493 Find what libraries can be imported Recursively check all dependencies Represent this tree in a nice way

2 nd task: Cheetah to Jinja 2 template converter Cheetah is no longer supported in py 3 (official website is down) Create tool that converts Cheetah syntax to Jinja 2 Find code that can’t be converted automaticaly Test it

Cheetah: Jinja 2: #from DAS. web. utils import quote {% set item = quote(item) %} #set item = $quote($item) {% set api = quote(api) %} #set api = $quote($api)  {% if api == "sites" %} #if $api == "sites" <a href="/sitedb/prod/sites/{{item}}">Site. DB</a> <a href="/sitedb/prod/sites/$item">Site. DB</a> {% elif api == "people" %} #else if $api == "people" <a href="/sitedb/prod/people/{{item}}">Site. DB</a> <a href="/sitedb/prod/people/$item">Site. DB</a> {% endif %} #end if

3 rd task: Get all dmwm dependencies 3 million lines of code Find all imports for all projects Sort them Get py 3 projects: https: //pypi. org/pypi/{project. Name}/json Find default libraries Analyze the results

Crab. Clients has ~ 500 imports. DAS ~ 1700 After analysis of DAS: 26 imports were not found/supported in py 3 16 of them were external libraries full list of dependencies for all projects and what libraries can replace them We have 11 unused projects in dmwm

4 th task: code porting Started from mox -> mox 3 (more about in in the next slide) popen 2 to subprocess set and String. IO c. Pickle and commands

5 th task: find unused code While porting mox found “weird” looking classes They were unused https: //pypi. python. org/pypi/vulture ~120 unused classes ~40+ are definitely not used

Failed attempts. . . Sequence. Matcher… Comparing code and if match is more than 0. 5 - code obsolete? Find oldest unupdated files using git logs Almost giving up… What about classes that have same names? Just sticking to manual labor…. Worked. . .

6 th task: Write unit test for backends https: //github. com/dmwm/WMCore/tree/master/src/python/W MCore/Storage backends vs plugins No tests, no documentation, debates over what are we using Using mock and writing unit test 177 tests, WMCore had ~1100

7 th task: reqmgr 2 GUI improvement Clean CSS and add dropdowns Update templates Add new libraries Test it My VM died…. .

Thank you for your attention!