[diffoscope] Tests on FreeBSD

[Ed Maste]

On Mon, 12 Jun 2023 at 17:09, Mattia Rizzolo <mattia at mapreri.org> wrote:
>
> Hey! :)
>
> On Mon, Jun 12, 2023 at 04:33:47PM -0400, Ed Maste wrote:
> > I'd like to get FreeBSD into diffoscope CI, either by using
> > Cirrus-CI's hosted service, or via GitLab CI.
>
> That's a great idea! ^^
> I'd probably stick with gitlab-ci (but I think it would need somebody to
> provide a runner?) if anything just so that we can have everything
> together…

Certainly having a runner connected to the existing gitlab-ci is
preferable if we can easily stand up a VM for the runner. Holger
mentions that there's already a FreeBSD 13.2 VM for jenkins.debian.net
-- perhaps the runner can execute there? Otherwise I can see about
standing something up on FreeBSD infrastructure.

Cirrus-Ci is convenient because it requires no infrastructure to start
- for example, here's a Cirrus run in my tree:
https://cirrus-ci.com/task/4669869195526144

> > But to start off I had a
> > look at the state of tests on FreeBSD and I see a few failures to sort
> > out. There are also a number of skipped tests; I'm leaving those aside
> > for now.
> >
> > Summary (run on my laptop):
>
> It's probably better if you open a bunch of (different!) bug reports on
> salsa for the several test failures (with, maybe, a catchall one to
> gather them as subtasks).

I was hoping to gather some more detail before submitting bug reports
(or patches). I would at least prefer to group identical issues into
common bugs.

> > A couple of the failures come from llvm_version() invoking
> > llvm-config; in FreeBSD we have Clang and other primary LLVM tools in
> > the base system but do not include llvm-config. Perhaps we can check
> > the version of the specific tool we're using (i.e., llvm-dis /
> > llvm-readobj / llvm-objdump)?
>
> llvm-config is used only in _1_ test, so I'm sure it's possible to
> change that, sure.

I couldn't find any cases where llvm-config is actually used, other
than to find the version of LLVM that's installed.

> > tests/comparators/test_device.py::test_diff and ::test_diff_reverse,
> > and tests/comparators/test_git.py::test_diff all fail because device
> > ID (major/minor) do not match. I'm not sure how best to handle this,
> > but I think we ought to have diffoscope elide them rather than having
> > the test be permissive.
>
> well, doesn't that change because of the different kernel?  If my
> assumption is correct, then I think we should rather have diffoscope
> recognize the running kernel type (linux vs bsd) and handle that
> accordingly.

The tests already do this -- for example we have separate output files
tests/data/device_expected_diff and
tests/data/device_expected_diff_freebsd. But the device node major and
minor numbers aren't really meaningful on FreeBSD, and might be
different depending on the environment (as seems to have happened
here).