[Git][reproducible-builds/debian-rebuilder-setup][ansible] 2 commits: Add basic scheduler deployment, waiting for bidb apis

[kpcyrd]

kpcyrd pushed to branch ansible at Reproducible Builds / debian-rebuilder-setup


Commits:
aee182be by kpcyrd at 2018-11-01T22:13:03Z
Add basic scheduler deployment, waiting for bidb apis

- - - - -
db755a7d by kpcyrd at 2018-11-02T21:44:43Z
Add buildinfo server monitor

Depends on:
https://github.com/lamby/buildinfo.debian.net/pull/54
https://github.com/Foxboron/buildinfo.debian.net/pull/1

- - - - -


4 changed files:

- playbook.yml
- + roles/schedulers/tasks/main.yml
- + scheduler/srebuild-monitor
- + scheduler/srebuild-worker


Changes:

=====================================
playbook.yml
=====================================
@@ -1,7 +1,7 @@
 - name: Debugging Stuff (this is needed to obtain the ipv4 addresses once)
   hosts: all
   tasks:
-    - debug: var=hostvars[inventory_hostname]['ansible_default_ipv4']['address']
+  - debug: var=hostvars[inventory_hostname]['ansible_default_ipv4']['address']
 
 - name: Setup Builders
   hosts: builders
@@ -9,11 +9,25 @@
   become_user: root
   become_method: sudo
   vars_files:
-    - external_vars.yml
+  - external_vars.yml
+  tags:
+  - builders
   roles:
-    - builders
-    # The gpgkey generation role had to be disabled because it was taking a lot of time. I'm not sure we are supposed to do this.
-    # - { role: juju4.gpgkey_generate, gpg_user: "{{ build_gpg_user }}", gpg_realname: "{{ build_gpg_realname }}", gpg_useremail: "{{ build_gpg_email }}" , gpg_generator_user: "root" }
+  - builders
+  # The gpgkey generation role had to be disabled because it was taking a lot of time. I'm not sure we are supposed to do this.
+  # - { role: juju4.gpgkey_generate, gpg_user: "{{ build_gpg_user }}", gpg_realname: "{{ build_gpg_realname }}", gpg_useremail: "{{ build_gpg_email }}" , gpg_generator_user: "root" }
+
+- name: Setup Scheduler
+  hosts: schedulers
+  become: yes
+  become_user: root
+  become_method: sudo
+  vars_files:
+  - external_vars.yml
+  tags:
+  - schedulers
+  roles:
+  - schedulers
 
 - name: Setup Visualizers
   hosts: visualizers
@@ -21,12 +35,14 @@
   become_user: root
   become_method: sudo
   vars_files:
-    - external_vars.yml
+  - external_vars.yml
+  tags:
+  - visualizers
   pre_tasks:
-    - name: Install dirmngr for nginx
-      apt:
-        name: "dirmngr"
+  - name: Install dirmngr for nginx
+    apt:
+      name: "dirmngr"
   roles:
-    - role: nginxinc.nginx
-    - visualizers
+  - role: nginxinc.nginx
+  - visualizers
 


=====================================
roles/schedulers/tasks/main.yml
=====================================
@@ -0,0 +1,24 @@
+- name: Install all dependencies
+  apt:
+    name:
+    - redis-server
+    - python3-redis
+    - python3-requests
+
+- name: Copy scheduler-worker
+  copy:
+    src: ../../../scheduler/srebuild-worker
+    dest: /usr/bin/srebuild-worker
+
+- name: Copy scheduler-monitor
+  copy:
+    src: ../../../scheduler/srebuild-monitor
+    dest: /usr/bin/srebuild-monitor
+
+- name: Set permissions
+  file:
+    path: "{{ item }}"
+    mode: 0755
+  with_items:
+  - /usr/bin/srebuild-worker
+  - /usr/bin/srebuild-monitor


=====================================
scheduler/srebuild-monitor
=====================================
@@ -0,0 +1,51 @@
+#!/usr/bin/env python3
+import requests
+import urllib.parse
+import redis
+import time
+import sys
+
+
+def main(server):
+    most_recent = 0
+    db = redis.StrictRedis(host='localhost', port=6379, db=0)
+
+    # try to load most_recent, default to 0
+    most_recent = db.get('rebuild-most-recent') or '0'
+    most_recent = int(most_recent)
+
+    while True:
+        print('[*] Requesting new buildinfo files')
+
+        url = urllib.parse.urljoin(server, '/api/buildinfo/since/%d' % most_recent)
+        r = requests.get(url)
+        r.raise_for_status()
+        buildinfos = r.json()['buildinfos']
+
+        for buildinfo in buildinfos:
+            raw_uri = buildinfo['raw-uri']
+            print('[+] Adding %r' % raw_uri)
+
+            # request buildinfo file
+            r = requests.get(raw_uri)
+            r.raise_for_status()
+
+            # add buildinfo to queue
+            db.rpush('rebuild-q', r.text)
+
+            # update most_recent
+            most_recent = buildinfo['created']
+
+        # store most_recent
+        db.set('rebuild-most-recent', most_recent)
+
+        if not buildinfos:
+            print('[*] Sleeping zZz')
+            time.sleep(10)
+
+
+if __name__ == '__main__':
+    if len(sys.argv) < 2:
+        print('Usage: %s http://buildinfo.nyu.wtf/' % sys.argv[0])
+    else:
+        main(sys.argv[1])


=====================================
scheduler/srebuild-worker
=====================================
@@ -0,0 +1,28 @@
+#!/usr/bin/env python3
+import redis
+import tempfile
+import subprocess
+
+
+def rebuild(buildinfo):
+    with tempfile.NamedTemporaryFile() as f:
+        f.write(buildinfo)
+        cmd = ['srebuild-runner', f.name]
+        print('[+] invoking %r' % cmd)
+        rc = subprocess.call(cmd)
+        if rc != 0:
+            # TODO: we should handle this properly
+            print('[!] srebuild returned an error')
+
+
+def main():
+    r = redis.StrictRedis(host='localhost', port=6379, db=0)
+    print('[*] monitoring queue')
+    while True:
+        q, item = r.blpop('rebuild-q')
+        print('[*] rebuilding %r' % item)
+        rebuild(item)
+
+
+if __name__ == '__main__':
+    main()



View it on GitLab: https://salsa.debian.org/reproducible-builds/debian-rebuilder-setup/compare/9714bf3b6fcb1ff90d8f81b14a0b3a71ee85967d...db755a7d68ece7f8fb55770fdd927c61d002bdd3

-- 
View it on GitLab: https://salsa.debian.org/reproducible-builds/debian-rebuilder-setup/compare/9714bf3b6fcb1ff90d8f81b14a0b3a71ee85967d...db755a7d68ece7f8fb55770fdd927c61d002bdd3
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.reproducible-builds.org/pipermail/rb-commits/attachments/20181102/6253a75f/attachment.html>