[Git][reproducible-builds/reprotest][master] 10 commits: Help on available values for verbosity
Holger Levsen
gitlab at salsa.debian.org
Mon Jan 18 17:42:34 UTC 2021
Holger Levsen pushed to branch master at Reproducible Builds / reprotest
Commits:
47516307 by Frédéric Pierret (fepitre) at 2021-01-18T18:01:55+01:00
Help on available values for verbosity
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
9f86fbb9 by Frédéric Pierret (fepitre) at 2021-01-18T18:01:55+01:00
logger: replace deprecated warn method
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
a8a7f248 by Frédéric Pierret (fepitre) at 2021-01-18T18:01:55+01:00
Update .gitignore
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
03991226 by Frédéric Pierret (fepitre) at 2021-01-18T18:01:55+01:00
presets: slightly reorganize functions
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
07bd29ff by Frédéric Pierret (fepitre) at 2021-01-18T18:01:55+01:00
Add support for building source RPM
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
3a284371 by Frédéric Pierret (fepitre) at 2021-01-18T18:01:55+01:00
presets: disable %clean stage when building RPM
Workaround issue when running with +fileordering on Fedora.
It fails to remove some directories probably seen as still in-use:
+ rm -rf /tmp/reprotest.bJhwNd/const_build_path/rpmbuild/BUILDROOT/qubes-utils-4.1.12-1.fc32.x86_64
rm: cannot remove '/tmp/reprotest.bJhwNd/const_build_path/rpmbuild/BUILDROOT/qubes-utils-4.1.12-1.fc32.x86_64/usr/lib/qubes': Directory not empty
rm: cannot remove '/tmp/reprotest.bJhwNd/const_build_path/rpmbuild/BUILDROOT/qubes-utils-4.1.12-1.fc32.x86_64/usr/lib/debug/.dwz': Directory not empty
error: Bad exit status from /var/tmp/rpm-tmp.pEJpn7 (%clean)
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
ff45fbf9 by Frédéric Pierret (fepitre) at 2021-01-18T18:01:55+01:00
presets: better formatting for RPM extra_build_command
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
188cdf4d by Frédéric Pierret (fepitre) at 2021-01-18T18:01:55+01:00
presets: clean rpmbuild folder
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
40899e2e by Frédéric Pierret (fepitre) at 2021-01-18T18:01:55+01:00
Update README for RPM support
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
6c00d047 by Frédéric Pierret (fepitre) at 2021-01-18T18:01:55+01:00
Add RPM spec file
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
5 changed files:
- .gitignore
- README.rst
- + reprotest.spec
- reprotest/__init__.py
- reprotest/presets.py
Changes:
=====================================
.gitignore
=====================================
@@ -6,3 +6,5 @@ __pycache__
/.reprotestrc
/doc/reprotest.1
/doc/reprotest.h2m
+.idea
+.vscode
=====================================
README.rst
=====================================
@@ -32,7 +32,14 @@ The easiest way to run reprotest is via our presets::
# See https://wiki.debian.org/sbuild for instructions on setting that up.
$ reprotest reprotest_0.3.3.dsc -- schroot unstable-amd64-sbuild
-Currently, we only support this for Debian packages, but are keen on
+ # Build the given RPM source package
+ # Only null server (/tmp) is currently supported.
+ $ reprotest reprotest-0.7.16.src.rpm
+
+ # Build the given RPM source package and automatically install dependencies
+ $ reprotest --testbed-build-pre 'dnf -y builddep ./*.src.rpm' reprotest-0.7.16.src.rpm
+
+Currently, we only support this for Debian and RPM based packages, but are keen on
adding more. If we don't have knowledge on how to build your file or
directory, you can send a patch to us on adding this intelligence - see
the reprotest.presets python module, and adapt the existing logic.
@@ -368,6 +375,12 @@ If you see a difference that you really think should not be there, try passing
https://tests.reproducible-builds.org/ which use a different (more reliable)
mechanism to vary the system time.
+Kernel
+------
+
+The "kernel" variation is currently not working for RPM based packages. While
+building with this variation enabled, the tool `/usr/lib/rpm/redhat/brp-ldconfig`
+compains about `FATAL: kernel too old` and aborts the build.
Avoid sudo(1) password prompts
==============================
=====================================
reprotest.spec
=====================================
@@ -0,0 +1,48 @@
+Name: reprotest
+Version: 0.7.15
+Release: 1%{?dist}
+Summary: Build packages and check them for reproducibility
+
+License: GPL-3+
+Source0: https://salsa.debian.org/reproducible-builds/%{name}/-/archive/%{version}/%{name}-%{version}.tar.gz
+BuildArch: noarch
+
+BuildRequires: python%{python3_pkgversion}-devel
+BuildRequires: python%{python3_pkgversion}-setuptools
+
+Requires: diffoscope
+Requires: disorderfs
+Requires: python%{python3_pkgversion}-rstr
+
+%description
+reprotest builds the same source code twice in different environments, and
+then checks the binaries produced by each build for differences. If any are
+found, then diffoscope (or if unavailable then diff) is used to display them
+in detail for later analysis.
+
+It supports different types of environment such as a "null" environment (i.e.
+doing the builds directly in /tmp) or various other virtual servers, for
+example schroot, ssh, qemu, and several others.
+
+reprotest is developed as part of the "reproducible builds" Debian project.
+
+%prep
+%autosetup -n %{name}-%{version}
+# Remove bundled egg-info
+rm -rf %{name}.egg-info
+
+%build
+%py3_build
+
+%install
+%py3_install
+
+%files
+%doc README.rst
+%{_bindir}/reprotest
+%{python3_sitelib}/%{name}
+%{python3_sitelib}/%{name}-%{version}-py%{python3_version}.egg-info
+
+%changelog
+* Mon Jan 04 2021 Frédéric Pierret (fepitre) <frederic.pierret at qubes-os.org> - 0.7.15-1
+- Initial RPM packaging.
=====================================
reprotest/__init__.py
=====================================
@@ -91,7 +91,7 @@ def start_testbed(args, temp_dir, no_clean_on_error=False, host_distro=None):
pass
except BaseException as e:
if no_clean_on_error:
- logger.warn("preserving temporary files in: %s", testbed.scratch)
+ logger.warning("preserving temporary files in: %s", testbed.scratch)
should_clean = False
raise
finally:
@@ -545,7 +545,8 @@ def cli_parser():
group1 = parser.add_argument_group('basic options')
group1.add_argument('--verbosity', type=int, default=0,
- help='An integer. Control which messages are displayed.')
+ help='An integer. Control which messages are displayed (0: quiet ('
+ 'warning/error only), 1: info, 2: debug).')
group1.add_argument('-v', '--verbose', dest='verbosity', action='count',
help='Like --verbosity, but given multiple times without arguments.')
group1.add_argument('--host-distro', default=None,
@@ -689,7 +690,7 @@ def command_line(parser, argv):
def get_main_spec(parsed_args):
variations = [parsed_args.variations] + parsed_args.vary
if parsed_args.dont_vary:
- logger.warn("--dont-vary is deprecated; use --vary=-$variation instead")
+ logger.warning("--dont-vary is deprecated; use --vary=-$variation instead")
variations += ["-%s" % a for x in parsed_args.dont_vary for a in x.split(",")]
return VariationSpec().extend(variations)
@@ -731,9 +732,9 @@ def run(argv, dry_run=None):
elif first_arg == "auto":
build_command = first_arg
if parsed_args.artifact_pattern:
- logger.warn("old CLI form `reprotest auto <source_root>` detected, "
+ logger.warning("old CLI form `reprotest auto <source_root>` detected, "
"setting source_root to the second argument: %s", parsed_args.artifact_pattern)
- logger.warn("to avoid this warning, use instead `reprotest <source_root>` "
+ logger.warning("to avoid this warning, use instead `reprotest <source_root>` "
"or (if really necessary) `reprotest -s <source_root> auto <artifact>`")
source_root = parsed_args.artifact_pattern
parsed_args.artifact_pattern = None
@@ -743,7 +744,7 @@ def run(argv, dry_run=None):
parts = shlex.split(first_arg)
if len(parts) == 1:
if shutil.which(parts[0]) is None:
- logger.warn("XXX")
+ logger.warning("XXX")
raise RuntimeError("Not found, neither as a file nor as a command: %s" % first_arg)
# if len(parts) > 1 then it could be something like '( command )'
# which is valid despite '(' not existing.
@@ -792,7 +793,7 @@ def run(argv, dry_run=None):
specs.append(spec.extend(extra_build))
check_func = check
if parsed_args.min_cpus is None and not dry_run:
- logger.warn("The control build runs on 1 CPU by default, give --min-cpus to increase this.")
+ logger.warning("The control build runs on 1 CPU by default, give --min-cpus to increase this.")
min_cpus = parsed_args.min_cpus or 1
build_variations = Variations.of(
*specs,
@@ -810,10 +811,10 @@ def run(argv, dry_run=None):
missing = [(var, tools) for var, tools in missing if tools]
for var, tools in missing:
if tools:
- logger.warn("Varying '%s' requires these program(s): %s", var, ", ".join(tools))
+ logger.warning("Varying '%s' requires these program(s): %s", var, ", ".join(tools))
if missing:
- logger.warn("Your build will probably fail, either install them or disable the variations.")
- logger.warn("(From a system package manager, simply install the 'optional' or 'recommended' "
+ logger.warning("Your build will probably fail, either install them or disable the variations.")
+ logger.warning("(From a system package manager, simply install the 'optional' or 'recommended' "
"dependencies of reprotest.)")
# Remaining args
=====================================
reprotest/presets.py
=====================================
@@ -63,15 +63,27 @@ class ReprotestPreset(collections.namedtuple('_ReprotestPreset',
PRESET_DEB_DIR = ReprotestPreset(
- build_command = 'dpkg-buildpackage --no-sign -b',
- artifact_pattern = '../*.deb',
- testbed_pre = None,
- testbed_init = None,
- testbed_build_pre = None,
- source_pattern = None,
- diffoscope_args = [],
+ build_command='dpkg-buildpackage --no-sign -b',
+ artifact_pattern='../*.deb',
+ testbed_pre=None,
+ testbed_init=None,
+ testbed_build_pre=None,
+ source_pattern=None,
+ diffoscope_args=[],
)
+PRESET_RPM_DIR = ReprotestPreset(
+ build_command='rpmbuild --rebuild',
+ artifact_pattern='*[^src].rpm',
+ testbed_pre=None,
+ testbed_init=None,
+ testbed_build_pre=None,
+ source_pattern='*.src.rpm',
+ diffoscope_args=[],
+)
+
+
+# DEB
def preset_deb_schroot(fn, preset):
return preset.set.testbed_init(
# need to symlink /etc/mtab to work around a fusermount(1) deficiency
@@ -81,12 +93,6 @@ def preset_deb_schroot(fn, preset):
).set.testbed_build_pre(
'apt-get -y --no-install-recommends build-dep ./"%s"' % fn)
-def parse_dsc_aux(path):
- dscfiles = subprocess.check_output(["egrep",
- # this regex comes from dcmd(1) from the devscripts package
- r"^ [0-9a-f]{32} [0-9]+ ((([a-zA-Z0-9_.-]+/)?[a-zA-Z0-9_.-]+|-) ([a-zA-Z]+|-) )?(.*)$",
- path])
- return [x.split()[-1].decode("utf-8") for x in dscfiles.splitlines()]
def preset_deb_dsc(fn, aux):
# we use "$(basename "$PWD")" so that the dirname varies iff we are varying the whole build path
@@ -95,6 +101,40 @@ def preset_deb_dsc(fn, aux):
).set.artifact_pattern("*.deb"
).set.source_pattern(" ".join(shlex.quote(a) for a in [fn] + aux))
+
+def parse_dsc_aux(path):
+ dscfiles = subprocess.check_output(["egrep",
+ # this regex comes from dcmd(1) from the devscripts package
+ r"^ [0-9a-f]{32} [0-9]+ ((([a-zA-Z0-9_.-]+/)?[a-zA-Z0-9_.-]+|-) ([a-zA-Z]+|-) )?(.*)$",
+ path])
+ return [x.split()[-1].decode("utf-8") for x in dscfiles.splitlines()]
+
+
+# RPM
+def preset_rpm_rpmbuild(fn):
+ # We ensure to have generated RPMs into temporary HOME folder and not in
+ # rpmbuild/RPMS/%%{ARCH}/
+ extra_build_command = [
+ '--define "_build_name_fmt %%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm"',
+ '--define "_topdir $PWD/rpmbuild"',
+ '--define "_rpmdir $PWD"',
+ '%s' % fn
+ ]
+
+ # rpmbuild needs to be told to use reproducible specific options
+ extra_build_command += ['--define "source_date_epoch_from_changelog Y"']
+ extra_build_command += ['--define "clamp_mtime_to_source_date_epoch Y"']
+ extra_build_command += ['--define "use_source_date_epoch_as_buildtime Y"']
+
+ # Disable %clean stage: workaround issue when running with +fileordering
+ extra_build_command += ['--noclean']
+ extra_build_command += ['; rm -rf $PWD/rpmbuild']
+
+ extra_build_command = ' '.join(extra_build_command)
+ return PRESET_RPM_DIR.append.build_command(' %s' % extra_build_command)
+
+
+###
def get_presets(buildfile, virtual_server):
fn = os.path.basename(buildfile)
parts = os.path.splitext(fn)
@@ -110,5 +150,12 @@ def get_presets(buildfile, virtual_server):
return preset_deb_dsc(fn, parse_dsc_aux(buildfile))
else:
return preset_deb_schroot(fn, preset_deb_dsc(fn, parse_dsc_aux(buildfile)))
+ elif parts[1] == '.rpm':
+ if os.path.splitext(parts[0])[1] != '.src':
+ raise ValueError('cannot determine .src.rpm')
+ if virtual_server == "null":
+ return preset_rpm_rpmbuild(fn)
+ else:
+ raise ValueError("only 'null' virtual_server is currently available for source RPM")
raise ValueError('unrecognised file type: "%s"; try giving '
'an appropriate --build-command' % buildfile)
View it on GitLab: https://salsa.debian.org/reproducible-builds/reprotest/-/compare/b6d3b44dd17b9962b89bdee5ada4160f10259ddd...6c00d047bb5a339188b4cdbfa5deb99045710579
--
View it on GitLab: https://salsa.debian.org/reproducible-builds/reprotest/-/compare/b6d3b44dd17b9962b89bdee5ada4160f10259ddd...6c00d047bb5a339188b4cdbfa5deb99045710579
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.reproducible-builds.org/pipermail/rb-commits/attachments/20210118/f44d5219/attachment.htm>
More information about the rb-commits
mailing list