Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve :all bottle coverage #191352

Open
carlocab opened this issue Sep 20, 2024 · 7 comments
Open

Improve :all bottle coverage #191352

carlocab opened this issue Sep 20, 2024 · 7 comments
Labels
good first issue A good candidate for your first contribution to Homebrew/homebrew-core help wanted Task(s) needing PRs from the community or maintainers

Comments

@carlocab
Copy link
Member

carlocab commented Sep 20, 2024

We currently have a few hundred :all bottles in this repository. I count another 200 or so more (see candidate formulae below) that could have an :all bottle but don't currently have one.

These are typically due to /usr/local references inside that bottles that are relocated on x86_64 macOS but not on arm64 macOS or Linux. To fix these, we typically only need a few inreplace lines. See below for example PRs.

Example PRs
Candidate Formulae
  • abi3audit
  • alluxio
  • angular-cli
  • apify-cli
  • appium
  • appwrite
  • arttime
  • asciidoc
  • asyncapi
  • autoconf
  • automake
  • autopsy
  • autorest
  • aws-amplify
  • aws-auth
  • awscli-local
  • b2-tools
  • bandcamp-dl
  • basedpyright
  • basti
  • bork
  • bower
  • braid
  • bundletool
  • cdk8s
  • censys
  • certbot
  • certifi
  • chruby
  • cimg
  • cloudflare-wrangler2
  • cmake-docs
  • cocoapods
  • commitlint
  • composer
  • contentful-cli
  • cpanminus
  • cpm
  • cppman
  • credstash
  • cubejs-cli
  • dbml-cli
  • ddclient
  • devcontainer
  • dicebear
  • diceware
  • diffoscope
  • dita-ot
  • doc8
  • dockerfile-language-server
  • druid
  • duply
  • easyengine
  • eleventy
  • eslint
  • extra-cmake-modules
  • fades
  • fail2ban
  • fanyi
  • fauna-shell
  • firebase-cli
  • flake8
  • flamebearer
  • flawfinder
  • flit
  • fnt
  • fonts-encodings
  • foreman
  • fred
  • geoip2fast
  • getmail6
  • git-big-picture
  • git-cola
  • git-number
  • git-remote-codecommit
  • git-review
  • git-subrepo
  • gitlab-ci-local
  • gitmoji
  • gitter-cli
  • go
  • goenv
  • goolabs
  • gorilla-cli
  • gost
  • gprof2dot
  • grails
  • graphql-cli
  • graphqurl
  • grokmirror
  • groovysdk
  • grunt-cli
  • gsettings-desktop-schemas
  • gtop
  • gulp-cli
  • hadoop
  • haste-client
  • hexo
  • http-server
  • httpyac
  • hyfetch
  • insect
  • instalooter
  • intltool
  • inxi
  • ios-sim
  • ipython
  • itstool
  • jruby
  • js-beautify
  • jscpd
  • json2ts
  • jtbl
  • kafka
  • keyring
  • kubevious
  • lando-cli
  • latexdiff
  • lerna
  • liquibase
  • luarocks
  • lynis
  • libclc
  • libunwind-headers
  • mailsy
  • markdownlint-cli
  • marp-cli
  • maven
  • maxwell
  • mjml
  • mysqltuner
  • n
  • nativefier
  • naturaldocs
  • nave
  • nb
  • nbdime
  • neonctl
  • netsurf-buildsystem
  • newman
  • nicotine-plus
  • nikto
  • nox
  • npm-check-updates
  • osc-cli
  • osc
  • pandemics
  • pass-git-helper
  • pass
  • patch-package
  • pedump
  • phoronix-test-suite
  • phpbrew
  • phpstan
  • pip-tools
  • pipdeptree
  • pipenv
  • pipgrip
  • pipx
  • platformio
  • pnpm
  • pnpm@8
  • postgraphile
  • powerlevel10k
  • principalmapper
  • psalm
  • psysh
  • ptpython
  • px
  • pycparser
  • pyenv-virtualenv
  • pylint
  • pyqt-builder
  • pyright
  • python-argcomplete
  • pythran
  • quicktype
  • rbtools
  • redo - redo: build an :all bottle #191545
  • release-it
  • rich-cli
  • rollup
  • saf-cli
  • scons
  • seam
  • search-that-hash
  • serveit
  • serverless
  • shallow-backup
  • shodan
  • sip
  • slimerjs
  • snyk-cli
  • solhint
  • sonarqube-lts
  • sonarqube
  • spack
  • spectral-cli
  • speedtest-cli
  • spotbugs
  • sql-lint
  • sqlite-utils
  • sqlmap
  • sslmate
  • standard
  • stepci
  • stylelint
  • subliminal
  • sugarjar
  • supervisor
  • tailwindcss-language-server
  • tartufo
  • teku
  • terrahub
  • tmuxinator-completion
  • tmuxinator
  • tox
  • twarc
  • ungit
  • unoconv
  • unum
  • urlscan
  • util-macros
  • vcs
  • vercel-cli
  • virtualenv
  • vite
  • vue-cli
  • waybackpy
  • web-ext
  • webpack
  • wikibase-cli
  • wildfly-as
  • wiremock-standalone
  • wp-cli
  • xmltoman
  • xorgproto
  • yaml-language-server
  • yapf
  • yo
  • yuicompressor
  • z
  • zim
  • zinit
How do I identify the files to `inreplace`?

diffoscope is your best friend here (install it with brew intall diffoscope).

If you're trying to fix formula foo, you'll want to first check foo for which bottles have different checksums, e.g., with brew cat foo (if you have homebrew/core tapped), or with

brew info --json=v2 foo | jq '.formulae[].bottle.stable.files'

Typically, the Intel macOS bottles will all have the same checksums, but these will be different from the arm64 macOS bottles and Linux bottles (which often all have the same checksums as well).

Pull down the relevant bottles with:

brew fetch --bottle-tag arm64_sonoma foo
brew fetch --bottle-tag sonoma foo

and then examine them using diffoscope:

diffoscope "$(brew --cache --bottle-tag arm64_sonoma foo)" "$(brew --cache --bottle-tag sonoma foo)"

diffoscope will tell you which files differ, and these are the ones you'll want to inreplace.

Why do we care about `:all` bottles?

An :all bottle can be installed by any Homebrew user, regardless of their HOMEBREW_PREFIX or OS version, so having more :all bottles mean that more users can benefit from bottle usage.

Moreover, :all bottles are a signal that our builds are (somewhat) reproducible, which is important when providing pre-built packages. See https://reproducible-builds.org for more on this.

@carlocab carlocab added help wanted Task(s) needing PRs from the community or maintainers good first issue A good candidate for your first contribution to Homebrew/homebrew-core labels Sep 20, 2024
@Bo98
Copy link
Member

Bo98 commented Sep 20, 2024

Alternative take: inreplace feels like a massive hack and we should perhaps have a DSL that excludes certain files from prefix-scanning.

@cho-m
Copy link
Member

cho-m commented Sep 21, 2024

Alternative take: inreplace feels like a massive hack and we should perhaps have a DSL that excludes certain files from prefix-scanning.

This seems like a per-formula/file decision

  • In some cases, inreplace may improve formula if loading/reading stuff we install inside HOMEBREW_PREFIX, e.g. installed by other formulae, plugins, etc.
  • In others, /usr/local should be kept, e.g. when a program needs strict compliance with XDG specification.

@Bo98
Copy link
Member

Bo98 commented Sep 21, 2024

Yeah it will totally depend on the formula. There's definitely a fair number though that already read HOMEBREW_PREFIX and use /usr/local as a fallback or are checking specifically for non-Homebrew installs.

@cho-m
Copy link
Member

cho-m commented Sep 24, 2024

Other situations I recall for missing :all bottles:

  • Perl documentation generation often end up with Perl version (default for pod2man --release) and tool versions
  • Python formulae can have diff due to build tool used (version of wheel or setuptools), e.g. pycparser
  • Python formulae also have the .pth file we auto-create. Sometimes this is unnecessarily encompassing, e.g. formulae that use uses_from_macos "libxml2" usually aren't using the Python binding as macOS doesn't provide it. An alternative could be handling some of this during bottle pour depending on how speed of file generation.

There are also formulae that are only :all per-OS/platform. These may have been :all before Linuxbrew merge. If there are enough of these, it may be worth exploring a :macos bottle.

@carlocab
Copy link
Member Author

Alternative take: inreplace feels like a massive hack and we should perhaps have a DSL that excludes certain files from prefix-scanning.

I guess these are hacks to the extent that parts of our existing prefix-relocation scheme is a hack. I actually wouldn't mind relocating automatically a /usr/local reference to HOMEBREW_PREFIX if that's what we would already have done on Intel macOS.

There are also formulae that are only :all per-OS/platform. These may have been :all before Linuxbrew merge. If there are enough of these, it may be worth exploring a :macos bottle.

I've noticed this too. Some Go formulae might even work as :arm64_macos and :x86_64_macos -- e.g. the go formula itself. Would not be surprised if our existing Intel bottles for go work as far back as :catalina.

@cho-m
Copy link
Member

cho-m commented Sep 30, 2024

I actually wouldn't mind relocating automatically a /usr/local reference to HOMEBREW_PREFIX if that's what we would already have done on Intel macOS.

Having brew automatically pick one and perhaps having an easy override for other may be best way to reduce the manual diff-ing and fix ups which add maintenance overhead. Maybe using a DSL similar to skip_clean for overriding.

@GIgako19929

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue A good candidate for your first contribution to Homebrew/homebrew-core help wanted Task(s) needing PRs from the community or maintainers
Projects
None yet
Development

No branches or pull requests

4 participants