1# Android changes for NDK developers 2 3This document details important changes related to native code 4loading in various Android releases. 5 6Required tools: the NDK has an _arch_-linux-android-readelf binary 7(e.g. arm-linux-androideabi-readelf or i686-linux-android-readelf) 8for each architecture (under toolchains/), but you can use readelf for 9any architecture, as we will be doing basic inspection only. On Linux 10you need to have the “binutils” package installed for readelf, 11and “pax-utils” for scanelf. 12 13 14## How we manage incompatible changes 15 16Our general practice with dynamic linker behavior changes is that they 17will be tied to an app's target API level: 18 19* Below the affected API level we'll preserve the old behavior or issue 20a warning, as appropriate. 21 22* At the affected API level and above, we’ll refuse to load the library. 23 24* Warnings about any behavior change that will affect a library if you 25increase your target API level will appear in logcat when that library 26is loaded, even if you're not yet targeting that API level. 27 28* On a developer preview build, dynamic linker warnings will also show up 29as toasts. Experience has shown that many developers don’t habitually 30check logcat for warnings until their app stops functioning, so the 31toasts help bring some visibility to the issues before it's too late. 32 33 34## Changes to library search order 35 36We have made various fixes to library search order when resolving symbols. 37 38With API 22, load order switched from depth-first to breadth-first to 39fix dlsym(3). 40 41Before API 23, the default search order was to try the main executable, 42LD_PRELOAD libraries, the library itself, and its DT_NEEDED libraries 43in that order. For API 23 and later, for any given library, the dynamic 44linker divides other libraries into the global group and the local 45group. The global group is shared by all libraries and contains the main 46executable, LD_PRELOAD libraries, and any library with the DF_1_GLOBAL 47flag set (by passing “-z global” to ld(1)). The local group is 48the breadth-first transitive closure of the library and its DT_NEEDED 49libraries. The M dynamic linker searches the global group followed by 50the local group. This allows ASAN, for example, to ensure that it can 51intercept any symbol. 52 53 54## RTLD_LOCAL (Available in API level >= 23) 55 56The dlopen(3) RTLD_LOCAL flag used to be ignored but is implemented 57correctly in API 23 and later. Note that RTLD_LOCAL is the default, 58so even calls to dlopen(3) that didn’t explicitly use RTLD_LOCAL will 59be affected (unless they explicitly used RTLD_GLOBAL). With RTLD_LOCAL, 60symbols will not be made available to libraries loaded by later calls 61to dlopen(3) (as opposed to being referenced by DT_NEEDED entries). 62 63 64## GNU hashes (Availible in API level >= 23) 65 66The GNU hash style available with --hash-style=gnu allows faster 67symbol lookup and is now supported by the dynamic linker in API 23 and 68above. (Use --hash-style=both if you want to build code that uses this 69feature >= Android M but still works on older releases.) 70 71 72## Correct soname/path handling (Available in API level >= 23) 73 74The dynamic linker now understands the difference 75between a library’s soname and its path (public bug 76https://code.google.com/p/android/issues/detail?id=6670). API level 23 77is the first release where search by soname is implemented. Earlier 78releases would assume that the basename of the library was the soname, 79and used that to search for already-loaded libraries. For example, 80`dlopen("/this/directory/does/not/exist/libc.so", RTLD_NOW)` would 81find `/system/lib/libc.so` because it’s already loaded. This also meant 82that it was impossible to have two libraries `"dir1/libx.so"` and 83`"dir2/libx.so"` --- the dynamic linker couldn’t tell the difference 84and would always use whichever was loaded first, even if you explicitly 85tried to load both. This also applied to DT_NEEDED entries. 86 87Some apps have bad DT_NEEDED entries (usually absolute paths on the build 88machine’s file system) that used to work because we ignored everything 89but the basename. These apps will fail to load on API level 23 and above. 90 91 92## Symbol versioning (Available in API level >= 23) 93 94Symbol versioning allows libraries to provide better backwards 95compatibility. For example, if a library author knowingly changes 96the behavior of a function, they can provide two versions in the same 97library so that old code gets the old version and new code gets the new 98version. This is supported in API level 23 and above. 99 100 101## Opening shared libraries directly from an APK 102 103In API level 23 and above, it’s possible to open a .so file directly from 104your APK. Just use `System.loadLibrary("foo")` exactly as normal but set 105`android:extractNativeLibs="false"` in your `AndroidManifest.xml`. In 106older releases, the .so files were extracted from the APK file 107at install time. This meant that they took up space in your APK and 108again in your installation directory (and this was counted against you 109and reported to the user as space taken up by your app). Any .so file 110that you want to load directly from your APK must be page aligned 111(on a 4096-byte boundary) in the zip file and stored uncompressed. 112Current versions of the zipalign tool take care of alignment. 113 114Note that in API level 23 and above dlopen(3) will open a library from 115any zip file, not just your APK. Just give dlopen(3) a path of the form 116"my_zip_file.zip!/libs/libstuff.so". As with APKs, the library must be 117page-aligned and stored uncompressed for this to work. 118 119 120## Private API (Enforced for API level >= 24) 121 122Native libraries must use only public API, and must not link against 123non-NDK platform libraries. Starting with API 24 this rule is enforced and 124applications are no longer able to load non-NDK platform libraries. The 125rule is enforced by the dynamic linker, so non-public libraries 126are not accessible regardless of the way code tries to load them: 127System.loadLibrary, DT_NEEDED entries, and direct calls to dlopen(3) 128will all work exactly the same. 129 130Users should have a consistent app experience across updates, 131and developers shouldn't have to make emergency app updates to 132handle platform changes. For that reason, we recommend against using 133private C/C++ symbols. Private symbols aren't tested as part of the 134Compatibility Test Suite (CTS) that all Android devices must pass. They 135may not exist, or they may behave differently. This makes apps that use 136them more likely to fail on specific devices, or on future releases --- 137as many developers found when Android 6.0 Marshmallow switched from 138OpenSSL to BoringSSL. 139 140In order to reduce the user impact of this transition, we've identified 141a set of libraries that see significant use from Google Play's 142most-installed apps, and that are feasible for us to support in the 143short term (including libandroid_runtime.so, libcutils.so, libcrypto.so, 144and libssl.so). In order to give you more time to transition, we will 145temporarily support these libraries; so if you see a warning that means 146your code will not work in a future release -- please fix it now! 147 148In O and later, the system property `debug.ld.greylist_disabled` can be 149used to deny access to the greylist even to an app that would normally 150be allowed it. This allows you to test compatibility without bumping the 151app's `targetSdkVersion`. Use `setprop debug.ld.greylist_disabled true` 152to turn this on (any other value leaves the greylist enabled). 153 154``` 155$ readelf --dynamic libBroken.so | grep NEEDED 156 0x00000001 (NEEDED) Shared library: [libnativehelper.so] 157 0x00000001 (NEEDED) Shared library: [libutils.so] 158 0x00000001 (NEEDED) Shared library: [libstagefright_foundation.so] 159 0x00000001 (NEEDED) Shared library: [libmedia_jni.so] 160 0x00000001 (NEEDED) Shared library: [liblog.so] 161 0x00000001 (NEEDED) Shared library: [libdl.so] 162 0x00000001 (NEEDED) Shared library: [libz.so] 163 0x00000001 (NEEDED) Shared library: [libstdc++.so] 164 0x00000001 (NEEDED) Shared library: [libm.so] 165 0x00000001 (NEEDED) Shared library: [libc.so] 166``` 167 168*Potential problems*: starting from API 24 the dynamic linker will not 169load private libraries, preventing the application from loading. 170 171*Resolution*: rewrite your native code to rely only on public API. As a 172short term workaround, platform libraries without complex dependencies 173(libcutils.so) can be copied to the project. As a long term solution 174the relevant code must be copied to the project tree. SSL/Media/JNI 175internal/binder APIs should not be accessed from the native code. When 176necessary, native code should call appropriate public Java API methods. 177 178A complete list of public libraries is available within the NDK, under 179platforms/android-API/usr/lib. 180 181Note: SSL/crypto is a special case, applications must NOT use platform 182libcrypto and libssl libraries directly, even on older platforms. All 183applications should use GMS Security Provider to ensure they are protected 184from known vulnerabilities. 185 186 187## Missing Section Headers (Enforced for API level >= 24) 188 189Each ELF file has additional information contained in the section 190headers. These headers must be present now, because the dynamic linker 191uses them for sanity checking. Some developers strip them in an 192attempt to obfuscate the binary and prevent reverse engineering. (This 193doesn't really help because it is possible to reconstruct the stripped 194information using widely-available tools.) 195 196``` 197$ readelf --header libBroken.so | grep 'section headers' 198 Start of section headers: 0 (bytes into file) 199 Size of section headers: 0 (bytes) 200 Number of section headers: 0 201``` 202 203*Resolution*: remove the extra steps from your build that strip section 204headers. 205 206## Text Relocations (Enforced for API level >= 23) 207 208Starting with API 23, shared objects must not contain text 209relocations. That is, the code must be loaded as is and must not be 210modified. Such an approach reduces load time and improves security. 211 212The usual reason for text relocations is non-position independent 213hand-written assembler. This is not common. Use the scanelf tool as 214described in our documentation for further diagnostics: 215 216``` 217$ scanelf -qT libTextRel.so 218 libTextRel.so: (memory/data?) [0x15E0E2] in (optimized out: previous simd_broken_op1) [0x15E0E0] 219 libTextRel.so: (memory/data?) [0x15E3B2] in (optimized out: previous simd_broken_op2) [0x15E3B0] 220 ... 221``` 222 223If you have no scanelf tool available, it is possible to do a basic 224check with readelf instead, look for either a TEXTREL entry or the 225TEXTREL flag. Either alone is sufficient. (The value corresponding to the 226TEXTREL entry is irrelevant and typically 0 --- simply the presence of 227the TEXTREL entry declares that the .so contains text relocations). This 228example has both indicators present: 229 230``` 231$ readelf --dynamic libTextRel.so | grep TEXTREL 232 0x00000016 (TEXTREL) 0x0 233 0x0000001e (FLAGS) SYMBOLIC TEXTREL BIND_NOW 234``` 235 236Note: it is technically possible to have a shared object with the TEXTREL 237entry/flag but without any actual text relocations. This doesn't happen 238with the NDK, but if you're generating ELF files yourself make sure 239you're not generating ELF files that claim to have text relocations, 240because the Android dynamic linker trusts the entry/flag. 241 242*Potential problems*: Relocations enforce code pages being writable, and 243wastefully increase the number of dirty pages in memory. The dynamic 244linker has issued warnings about text relocations since Android K 245(API 19), but on API 23 and above it refuses to load code with text 246relocations. 247 248*Resolution*: rewrite assembler to be position independent to ensure 249no text relocations are necessary. The 250[Gentoo Textrels guide](https://wiki.gentoo.org/wiki/Hardened/Textrels_Guide) 251has instructions for fixing text relocations, and more detailed 252[scanelf documentation](https://wiki.gentoo.org/wiki/Hardened/PaX_Utilities). 253 254 255## Invalid DT_NEEDED Entries (Enforced for API level >= 23) 256 257While library dependencies (DT_NEEDED entries in the ELF headers) can be 258absolute paths, that doesn't make sense on Android because you have 259no control over where your library will be installed by the system. A 260DT_NEEDED entry should be the same as the needed library's SONAME, 261leaving the business of finding the library at runtime to the dynamic 262linker. 263 264Before API 23, Android's dynamic linker ignored the full path, and 265used only the basename (the part after the last ‘/') when looking 266up the required libraries. Since API 23 the runtime linker will honor 267the DT_NEEDED exactly and so it won't be able to load the library if 268it is not present in that exact location on the device. 269 270Even worse, some build systems have bugs that cause them to insert 271DT_NEEDED entries that point to a file on the build host, something that 272cannot be found on the device. 273 274``` 275$ readelf --dynamic libSample.so | grep NEEDED 276 0x00000001 (NEEDED) Shared library: [libm.so] 277 0x00000001 (NEEDED) Shared library: [libc.so] 278 0x00000001 (NEEDED) Shared library: [libdl.so] 279 0x00000001 (NEEDED) Shared library: 280[C:\Users\build\Android\ci\jni\libBroken.so] 281``` 282 283*Potential problems*: before API 23 the DT_NEEDED entry's basename was 284used, but starting from API 23 the Android runtime will try to load the 285library using the path specified, and that path won't exist on the 286device. There are broken third-party toolchains/build systems that use 287a path on a build host instead of the SONAME. 288 289*Resolution*: make sure all required libraries are referenced by SONAME 290only. It is better to let the runtime linker to find and load those 291libraries as the location may change from device to device. 292 293 294## Missing SONAME (Enforced for API level >= 23) 295 296Each ELF shared object (“native library”) must have a SONAME (Shared 297Object Name) attribute. The NDK toolchain adds this attribute by default, 298so its absence indicates either a misconfigured alternative toolchain 299or a misconfiguration in your build system. A missing SONAME may lead 300to runtime issues such as the wrong library being loaded: the filename 301is used instead when this attribute is missing. 302 303``` 304$ readelf --dynamic libWithSoName.so | grep SONAME 305 0x0000000e (SONAME) Library soname: [libWithSoName.so] 306``` 307 308*Potential problems*: namespace conflicts may lead to the wrong library 309being loaded at runtime, which leads to crashes when required symbols 310are not found, or you try to use an ABI-incompatible library that isn't 311the library you were expecting. 312 313*Resolution*: the current NDK generates the correct SONAME by 314default. Ensure you're using the current NDK and that you haven't 315configured your build system to generate incorrect SONAME entries (using 316the -soname linker option). 317 318 319## Writable and Executable Segments (Enforced for API level >= 26) 320 321Each segment in an ELF file has associated flags that tell the 322dynamic linker what permissions to give the corresponding page in 323memory. For security, data shouldn't be executable and code shouldn't be 324writable. This means that the W (for Writable) and E (for Executable) 325flags should be mutually exclusive. This wasn't historically enforced, 326but is now. 327 328``` 329$ readelf --program-headers -W libBadFlags.so | grep WE 330 LOAD 0x000000 0x00000000 0x00000000 0x4c01d 0x4c01d RWE 0x1000 331``` 332 333*Resolution*: we're aware of one middleware product that introduces these 334into your app. The middleware vendor is aware of the problem and has a fix 335available. 336 337## Invalid ELF header/section headers (Enforced for API level >= 26) 338 339In API level 26 and above the dynamic linker checks more values in 340the ELF header and section headers and fails if they are invalid. 341 342*Example error* 343``` 344dlopen failed: "/data/data/com.example.bad/lib.so" has unsupported e_shentsize: 0x0 (expected 0x28) 345``` 346 347*Resolution*: don't use tools that produce invalid/malformed 348ELF files. Note that using them puts application under high risk of 349being incompatible with future versions of Android. 350 351## Enable logging of dlopen/dlsym and library loading errors for apps (Available in Android O) 352 353Starting with Android O it is possible to enable logging of all dlsym/dlopen calls 354for debuggable apps. Here is short instruction on how to do that: 355``` 356adb shell setprop debug.ld.app.com.example.myapp dlsym,dlopen,dlerror 357adb logcat 358``` 359 360Any subset of (dlsym,dlopen,dlerror) can be used. 361 362On userdebug and eng builds it is possible to enable tracing for the whole system 363by using debug.ld.all system property instead of app-specific one: 364``` 365adb shell setprop debug.ld.all dlerror,dlopen 366``` 367 368enables logging of all errors and dlopen calls 369