As sometime I am trying to compile a full version (with all options) of rsync on MacOS (please, do not ask why I want to do it - if you can help me clear and directly, thank you so much. Otherwise, do not waste your time).
I found a really helpful script made by "junsionzhang" (https://gist.github.com/junsionzhang), which in my option is simple and direct. Thank you junsionzhang!
Even being a good script, clear, step-by-step, there are some steps that does not work for me (and I tried a lot and for a while).
Here is the script (at Oct, 16, 2022):
#Compile rsync 3.0.7
#Follow these instructions in Terminal on both the client and server to download and compile rsync 3.0.7:
#Download and unarchive rsync and its patches
cd ~/Desktop
curl -O http://rsync.samba.org/ftp/rsync/src/rsync-3.1.2.tar.gz
tar -xzvf rsync-3.1.2.tar.gz
rm rsync-3.1.2.tar.gz
curl -O http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.1.2.tar.gz
tar -xzvf rsync-patches-3.1.2.tar.gz
rm rsync-patches-3.1.2.tar.gz
cd rsync-3.1.2
#Apply patches relevant to preserving Mac OS X metadata
patch -p1 <patches/fileflags.diff
patch -p1 <patches/crtimes.diff
patch -p1 <patches/hfs-compression.diff
#Configure, make, install
./prepare-source
./configure
make
sudo make install
#Verify your installation
/usr/local/bin/rsync --version
#By default, rsync will be installed in /usr/local/bin.
#If that isn't in your path, you will need to call your new version of rsync by its absolute path (/usr/local/bin/rsync).
The three patches lines does not work for me. After patching the fileflags, the rsync patched can not me "prepared", and of course, not be configured. The others, crtimes.diff and hfs-compression.diff, does not exist on the TAR package.
So, questions:
Trying to compile the 3.2.6 version os MacOS Big Sur (11.7), what do I need and which is the right/correct way and steps to patch, update and have the "correct"version?
How do I (correctly) compile and install the all the libraries to have a real full rsync version, with all features available (ACL support, Xattr support, xxhash, zstd, lz4, openssl crypto, and so on...)?
I would like to update and contribute to a new "junsionzhang" script version, making options to install a simple/standard version (rsync only) and options to install the libraries and choose for a "more complete" version, and help another Mac users and the community. How can I make this bash script?
How to install gawk, mawk, nawk, awk ( and where from (what are the differences): gawk, mawk, nawk?
Some libraries I already have installed (which I do not know if I did them right) seems to be outdated. How to I update them?
When running "./prepare-source", i get this: "make: Nothing to be done for `conf'.". Does this is right?
Thank you all! I really appreciate for all help I can get!
Completely untested (I don't run MacOS).
URLs to access the source code:
Source Version Tarballs: https://rsync.samba.org/ftp/rsync/src/
Git repository: https://github.com/WayneD/rsync
might need these pre-requisites (from INSTALL.md):
brew install automake
brew install xxhash
brew install zstd
brew install lz4
brew install openssl
code to download and extract (to ~/Desktop/rsync-3.2.6):
# Download the relevant version of Rsync and the same version of
# Rsync patches, extract them and apply the "suggested"
# patches from the original script:
cd ~/Desktop
# Rsync
curl -O https://rsync.samba.org/ftp/rsync/src/rsync-3.2.6.tar.gz
tar -xzvf rsync-3.2.6.tar.gz
rm rsync-3.2.6.tar.gz
# Patches
curl -O https://rsync.samba.org/ftp/rsync/src/rsync-patches-3.2.6.tar.gz
tar -xzvf rsync-patches-3.2.6.tar.gz
rm rsync-patches-3.2.6.tar.gz
apply the "suggested" patches, light the blue touch paper and stand back:
cd ~/Desktop/rsync-3.2.6
# Apply patches relevant to preserving Mac OS X metadata
patch -p1 <patches/fileflags.diff
patch -p1 <patches/crtimes.diff
patch -p1 <patches/hfs-compression.diff
# Configure, make, install
./prepare-source
./configure
make
sudo make install
# Verify your installation
/usr/local/bin/rsync --version
Related
I can't load packages in R because the file libpq.5.dylib is not in /usr/lib/libpq.5.dylib. It is in /usr/local/Cellar/libpq/13.0/lib/libpq.5.dylib
I tried this line: sudo ln -s /usr/local/Cellar/libpq/13.0/lib/libpq.5.dylib /usr/lib/libpq.5.dylib but I get this response: ln: /usr/lib/libpq.5.dylib: Operation not permitted
What can I do to get the file in /usr/lib/libpq.5.dylib without causing issues? This solution suggests that I may face problems down the line so I don't understand what to do.
You really don't want it in /usr/lib. Apple declared that as off-limits, and on newer macOS versions it lives on a read-only volume. Unless you're willing to go into recovery mode and manually tamper with the volume (and possibly repeat that on future OS updates), this is not the way to go.
Instead, let's address the core issue:
Dynamic libraries on macOS embed their own install path inside the binary, and the linker copies that into binaries linking against them. This information can be changed with install_name_tool (see man install_name_tool).
Examine the install name of the dylib:
otool -l /usr/local/Cellar/libpq/13.0/lib/libpq.5.dylib | fgrep -A2 LC_ID_DYLIB
If the printed path already points to the dylib itself (or a path that is symlinked to it), use this path as [new_path] below, and skip step 2.
If the dylib's install name does not point back to itself, run this:
sudo install_name_tool -id /usr/local/Cellar/libpq/13.0/lib/libpq.5.dylib /usr/local/Cellar/libpq/13.0/lib/libpq.5.dylib
And use /usr/local/Cellar/libpq/13.0/lib/libpq.5.dylib for [new_path] below.
For binaries that link against the dylib, run:
sudo install_name_tool -change /usr/lib/libpq.5.dylib [new_path] [path_to_binary]
I had the same issue building a container through docker for API use : RPostgres was installed but the library couldn't load, same error message.
Since I had installed Postgres on my machine, I figure the problem was worked around therefore I had no such message on local ; but here's how I solved this in my dockerfile, 100% verified on a machine with nothing related to R installed :
RUN apt-get update && apt-get install libpq5 -y
So executing apt-get update && apt-get install libpq5 -y on your terminal should do the trick. Light and efficient.
It tried to load libpq.5.dylib from the symlink /opt/homebrew/opt/postgresql/lib/libpq.5.dylib but could not find the file, so you need to update it:
# TODO: get this from the error, after "Library not loaded:"
SYMLINK_PATH="/opt/homebrew/opt/postgresql/lib/libpq.5.dylib"
# TODO: find this in your machine. The version maybe different than mine
DESTINATION_PATH="/opt/homebrew/opt/postgresql/lib/postgresql#14/libpq.5.dylib"
sudo mv $SYMLINK_PATH $SYMLINK_PATH.old
sudo ln -s $DESTINATION_PATH $SYMLINK_PATH
I am trying to create a completely portable version of R for Mac that I can send to users with no R on their system and they can essentially double click a command file and it launches a Shiny application. I'll need to be able to install packages including some built from source (and some from GitHub).
I am using the script from this GitHub repository (https://github.com/dirkschumacher/r-shiny-electron/blob/master/get-r-mac.sh) as a starting point (it's also pasted below), creating a version of R, but (A) I find that when I try to launch R it gives me an error not finding etc/ldpaths and (B) when I try to launch Rscript it runs my system version -- I run `Rscript -e 'print(R.version)' and it prints out 4.0 which is my system version of R rather than the version 3.5.1 which the shell script has downloaded and processed.
I've experimented with editing the "R" executable and altering R_HOME and R_HOME_DIR but it still runs into issues when I try to install packages to the 3.5.1 directory.
Can anyone provide some guidance?
(By the way docker is not an option, this needs to be as simple as possible end-users with limited technical skills. So having them install docker etc won't be an option)
#!/usr/bin/env bash
set -e
# Download and extract the main Mac Resources directory
# Requires xar and cpio, both installed in the Dockerfile
mkdir -p r-mac
curl -o r-mac/latest_r.pkg \
https://cloud.r-project.org/bin/macosx/R-3.5.1.pkg
cd r-mac
xar -xf latest_r.pkg
rm -r r-1.pkg Resources tcltk8.pkg texinfo5.pkg Distribution latest_r.pkg
cat r.pkg/Payload | gunzip -dc | cpio -i
mv R.framework/Versions/Current/Resources/* .
rm -r r.pkg R.framework
# Patch the main R script
sed -i.bak '/^R_HOME_DIR=/d' bin/R
sed -i.bak 's;/Library/Frameworks/R.framework/Resources;${R_HOME};g' \
bin/R
chmod +x bin/R
rm -f bin/R.bak
# Remove unneccessary files TODO: What else
rm -r doc tests
rm -r lib/*.dSYM
Happy to help you get this working for your shiny app. You can use this github repo for Electron wrapping R/Shiny... just clone, and replace the app.R (for your other packages you need to install them in the local R folder after cloning and then running R from the command line out of the R-Portable-Mac/bin folder...
Try it with the Hello World app.R that is included first
https://github.com/ColumbusCollaboratory/electron-quick-start
And, then installing your packages in the local R-Portable-Mac folder runtime. Included packages by default...
https://github.com/ColumbusCollaboratory/electron-quick-start/tree/master/R-Portable-Mac/library
Your packages will show up here after install.packages() from the command line using the local R-Mac-Portable runtime.
We have been working on a R Addin for this also...
https://github.com/ColumbusCollaboratory/photon
But, note the add-in is still a work in progress and doesn't work with compiled R packages; still have to go into the local R folder and runtime on the command line and install the packages directly into the local R folder libpath as discussed above.
Give it a try and let us know through Github issues if you have any questions and issues. And, if you've already posted out there, sorry we haven't responded as of yet. Would love to communicate through the photon Add-In for this to get it working with compiling packages (into the libPath)--if you have the time to help. Thanks!
while i tried to install packages on Rstudio,I kept encountering the ld warning like this:
ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation are out of sync. Falling back to library file for linking.
Does someone know how to fix this?
Reinstalling CommandLineTools doesn't work for me.
These warnings were caused by the frameworks in /System/Library/Frameworks/.
The solution
Links the recent frameworks from MacOSX.sdk to /Library/Frameworks/.
sudo ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreFoundation.framework /Library/Frameworks/
sudo ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework /Library/Frameworks/
First, you can safely ignore those messages. This is a known condition after the installation of Mojave (macOS 10.14) or the macOS 10.13.6+ updates.
You can try to remove them by having the OS regenerate the compatibility files (Apple uses a different library format for macOS app development but has a compatibility layer for "normal" development). To do that, set aside abt 8 minutes on a fast internet connection, open a command-line prompt (Terminal/iTerm) and do:
sudo mv /Library/Developer/CommandLineTools /Library/Developer/CommandLineTools.old
then do:
xcode-select --install
and follow the instructions and wait.
Re-try your package/source compilations and if the messages go away, you should be able to safely do:
sudo rm -rf /Library/Developer/CommandLineTools.old
but leaving that directory tree there should not do any harm (provided you have the free space to spare).
The setup I have is like this: I have two sets of libraries that are compiled for amd64 (pc) and armelx (ARM). They are both used to cross-compile some software on a build machine.
The first ones (amd64) can be updated without hassle by updating the apt-repository and using apt-get install on the build machine. The packages for ARM however, I don't want to install with apt, because it does not support installing to different directory. If I installed to default directories, the versions could not coexist. Right?
So far, the build machine was updated manually each time there was a new version of the packages, simply by extracting with dpkg -x to a dedicated "fake" footfs directory. This is where the compiler would also look when cross compiling other SW. The problem is, there is no information about these extracted packages or their versions anywhere on the system, right? It should have been in the status file.
My thought was to have these packages installed on this footfs dir with dpkg -i <package.deb> --root=<rootfs>. Would this work? I have a feeling it will not, because the deb packages have no post/pre-remove/install scripts, so it may work for a virgin install somehow, but not for upgrading? Also, what must the rootfs directory structure look like and what must it contain in order for this to work even the first time? Is there a tool to help with this?
Thanks.
Once you have a base armel Debian system, you can actually enter it and run the armel code inside it using something like QEMU. The qemu-arm-static tool (in the qemu-user-static package) can make use of the binfmt_misc capability in Linux to make it so ARM executables are directly run under the QEMU ARM system emulator. So you can run dpkg, apt-get, and so on inside the armel "rootfs" while running on amd64 hardware.
Example:
my_arm_system=/mnt/arm_system
sudo cp /usr/bin/qemu-arm-static "$my_arm_system/usr/bin/"
sudo chroot "$my_arm_system" apt-get update
sudo chroot "$my_arm_system" apt-get install $somepkg
sudo chroot "$my_arm_system" /bin/bash
As for setting up the base armel system in the first place: Debootstrap is the typical method for setting up a Debian base system, whether in a chroot or otherwise. You can use it for installing a base system of a different architecture, but it takes a few extra steps:
distro=jessie # or whatever
echo "Debootstrap phase 1"
sudo mkdir "$my_arm_system"
sudo debootstrap --arch=armel --verbose --foreign "$distro" "$my_arm_system"
sudo cp /usr/bin/qemu-arm-static "$my_arm_system"/usr/bin/
echo "Debootstrap phase 2"
sudo chroot "$my_arm_system" /debootstrap/debootstrap --second-stage
Multistrap is another tool that might be useful; it is intended for setting up Debian environments of one architecture on a host of a different architecture, or for using more complicated APT source combinations. It's not perfect, as it doesn't follow all the deb installation "rules" exactly. It takes some shortcuts/deviations in order to make its job reasonably possible.
Is there any way to automatically update R on Mac OS X to the latest patched version (R-Patched) on a daily basis or some predetermined intervals?
My impression is that compiling from source is the most (only?) reliable way to get the most recent patched version, but I could be wrong about this. A simple shell script to download the latest patched version and recompile would be:
curl -o /tmp/R-patched.tar.gz ftp://ftp.stat.math.ethz.ch/Software/R/R-patched.tar.gz
tar xzvf /tmp/R-patched.tar.gz
cd /tmp/R-patched
./configure
make
cp bin/R <old_R_binary_location>
You could then use crontab to run this at regular intervals. I don't find the crontab man page to be very helpful, so I always end up referring back to guides such as this one.
I have a bash script that installs the daily patched build from http://r.research.att.com. Installed libraries remain untouched, except for those in core.
I update manually, but you could set up a cron job as #bnaul suggests. I'm not sure how it will handle the need for sudo'ing, however. You might have to move your R out of /Library/Frameworks and then change the script accordingly.
#!/bin/bash
curl -s http://r.research.att.com/R-2.13-branch-leopard-universal.tar.gz | sudo tar fvxz - -C /