Reverse Engineering macOS 11.0
New in macOS Big Sur 11 beta, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to
dlopen()the path, which will correctly check for the library in the cache.
The only impact is if you are doing runtime detection/search of library by path yourself. Which is a terrible idea for perf anyway.
iOS has been like that for a decade already.
The goal was optimization, but unfortunately it does make reverse engineering more difficult.
The shared cache isn’t encrypted or anything, and dyld is in the Darwin source dumps. The shared cache format may not be stable, but isn’t secret either
The data is there, but there currently aren’t tools that can get it into a useful format like we had before.
Incidentally, the new stripped framework cache on macOS 11 is horrendous for disassembly. If you’re trying to track down why there’s a bug in your app, or how a system implementation works, you are screwed. This is going to hurt developers more than the ARM transition
Jeff Johnson (tweet, also: zhuowei):
If the libraries are no longer present on the filesystem, that makes it awfully hard to disassemble them! Fortunately, there are ways to extract the system libraries from the cache. One way is provided by Apple itself: the
dyld_shared_cache_utilcommand-line tool. Unfortunately, this tool does not come installed with macOS Big Sur. However, the tool is open source, so we can build it ourselves.
Let’s take a look at an example from my favorite framework, AppKit.
[…]
It seems that prior to Big Sur, Objective-C references in a Mach-O file are offsets from the beginning on the file, whereas on Big Sur, Objective-C references in a Mach-O file are offsets from the beginning of the dyld shared cache. Roughly speaking.
You can also point Hopper at the shared cache in the folder /System/Library/dyld/, and it will let you choose which library to load. But, as with dyld_shared_cache_util, what you end up with is difficult to work with because the tools don’t know how to find the Objective-C selector information.
Big Sur also adds another optimization that gets in the way of reverse engineering. Leo Natan:
A lot of Apple’s private APIs are now peppered with direct and can no longer be swizzled.
This makes it harder to debug and work around bugs. Unlike with the shared cache, this can’t be worked around with better tools. The information (and indirection) have been removed from the library entirely.
Previously:
Update (2020-07-06): Anton Sotkov:
Modifications to Apple’s dyld project to fix Objective-C information when extracting dyld_shared_cache from macOS Big Sur to help Hopper generate readable pseudocode.
Update (2020-07-27): Jeff Johnson:
Static disassembly tools such as otool and llvm-objdump have not been updated to handle the dyld shared cache on Big Sur. However, one tool that does handle it is lldb, the debugger.
[…]
I hope that my little hack helps you to disassemble system libraries on Big Sur. It’s a bid tedious, but it mostly works, and you only have to do it once for each library you’re interested in.
See also: Hopper for Apple Silicon and Big Sur
Update (2020-11-25): nevyn Bengtsson:
Remember when I complained months ago how bad it would be for everyone if Big Sur really did ship without system library binaries? Less open, harder to develop, etc etc. Well, now even the latest version of CMake doesn’t understand how to make Big Sur apps, it seems.
Here’s the solution. Replace
-framework AVFoundationwithFIND_LIBRARIES. Now it finds the .tbd and links with that INSTEAD.But I find it absurd to break decades-old foundations in every build tool that isn’t Xcode >12.2.
Update (2021-09-08): Steve Troughton-Smith:
Apple’s changes to how its system frameworks are packaged in macOS 11+, rendering them stripped & extremely difficult to reverse-engineer, is proving disastrous for finding & fixing framework-level bugs 😪
Update (2024-02-01): Wade Tregaskis:
The good news for Hopper is that it has since been updated to work around this – you can access the Apple framework binaries through File > Read File from DYLD Cache… There’s also tools like dyld-shared-cache-extractor which can resurrect the binaries from the cache.
Note also that in Sonoma, at least, the cache lives at /System/Volumes/Preboot/Cryptexes/OS/System/Library/dyld/ (in previous macOS releases it was apparently in /System/Library/dyld/).
7 Comments RSS · Twitter
This is not actually something news that optimisation does not cope well with debugging and reverse engineering. Even without "direct", with Apple doing most new devs in Swift, the result would have been the same anyway.
And for the disassembly thing, there is nothing that prevent tools author to update their product to support this change.
This introduces an interesting problem for me, as I want to link agains some libraries, that are present in the system, but for quite some time Apple doesn't want you to link against them (like libcrypto.dylib and libssl.dylib). For example, I want to build libssh2.dylib (https://www.libssh2.org) shared library from source, using built in libcrypto.dylib and libssl.dylib as backend cryptography backend. Before Big Sur, it was possible by, e.g. creating symbolic links "/usr/local/lib/libcrypto.dylib -> /usr/lib/libcrypto.35.dylib" and "/usr/local/lib/libssl.dylib -> /usr/lib/libssl.35.dylib". The build script would find the libraries and link against them without a problem. But now, such attempt ends up with the message "No suitable cryptography backend found".
I could surely build both libcrypto.dylib and libssl.dylib from source as well, but then I'd have to ship them as well in the application bundle, thus increasing its size, while that's completely unnecessary, since the libraries are present in the system.
Does anyone know a way to overcome this problem. All my attempts to modify CMake and/or make/configure files ended up in a same failure.
Dragan, I don't know whether this would work, but perhaps you could link against the libraries extracted from the shared cache, and then use install_name_tool to change the path to /usr/lib
Thanks for suggestion Jeff. I was already thinking in that direction, as other binaries, already built on Catalina, contain paths to “/usr/lib/*” in LC_LOAD_DYLIB Mach-O commands, and they work fine. So I thought your approach may work. Unfortunately, that’s not going to be that easy. It seems those extracted libraries (btw, thanks a lot for outlining how to build dyld_shared_cache_util, it surely saved me a couple of hours) cannot be linked with. They don’t seem to export the list of symbols, so whatever library function you try to use, the linking fails with error “ld: symbol(s) not found for architecture…”.
For a test, I’ve created a very simple dummy tool, which links with system provided and allowed to link with (still :-)) libz.dylib shared library. If I use the system provided library (symlink “/usr/lib/libz.dylib -> /usr/lib/libz.1.dylib”, but actually pointing to the cached library), the tool builds and runs fine. If I then extract libz.1.dylib using dyld_shared_cache_util, copy it into “/usr/local/lib/libz.1.dylib” and try to build the tool using symlink “/usr/local/lib/libz.dylib -> /usr/local/lib/libz.1.dylib”, linking fails with error:
Undefined symbols for architecture x86_64:
“_inflateInit_” referenced from:
_main in main.o
ld: symbol(s) not found for architecture x86_64
and inflateInit_() was the only zlib specific function I used in the tool. If I then go to Catalina box, copy “/usr/lib/libz.1.dylib” from there into “/usr/local/lib/libz.1.dylib” on Big Sur (BS :-)) box, thus effectively replacing extracted version of the library with the old Catalina version, the tool builds and runs fine again.
I tried this with a couple of other (allowed) libraries, using both Xcode and automake to build the tool and the results described above are consistent. Perhaps there is some flag to pass to dyld_shared_cache_util to extract library’s list of symbols? I don’t know, I didn’t look into the code yet.
Besides, this whole “cached system libraries” thing seems to break automake/cmake builds of a lot of known tools out there. I do most of my work in Xcode, so I’m totally unexperienced with automake/cmake, apart from creating and tweaking very simple scripts. But I suspect that, once the required library is found in a form of symlink in “/usr/lib”, at certain moment scripts tend to check for a presence in the file system of an actual target of the symlink, either in configuration phase while checking for various symbols and functions availability, or later in the linking phase.
For example, using cmake I tried to build a very well known libarchive (https://libarchive.org). It links with some (still allowed) system libraries, such are libiconv.dylib, libz.dylib, libbz2.dylib and libxml2.dylib. Whenever any such library is needed to check for certain functionality, an error is generated. For example:
Performing C SOURCE FILE Test HAVE_ICONV_libiconv_ failed with the following output:
……..
make[1]: *** No rule to make target `/usr/lib/libiconv.dylib', needed by target `cmTC_73fde'. Stop.
Or:
Determining if the BZ2_bzCompressInit exist failed with the following output:
……..
make[1]: *** No rule to make target `/usr/lib/libbz2.dylib', needed by target `cmTC_ed8d5'. Stop.
Finally, the build definitely fails at linking stage with the similar error:
make[2]: *** No rule to make target `/usr/lib/libz.dylib', needed by `libarchive/libarchive.17.dylib'. Stop.
I don’t know yet if only build scripts require modifications to work with cached libraries, of even automake/cmake tools themselves, but it’s clear the problem exists and it needs solving.
The dynamic linker cache is real headache when I try to use old Makefile(s) on BigSur. The only solution I found is to extract the system libraries from dynamic linker cache using dyld_shared_cache_util utility.
Thanks for this! Every now and again I forget Apple did this, and have to suffer websearching for ages looking for an explanation until I finally rediscover this post.
