Mobile App Data Extraction Guide

Mobile apps often surface richer data than their web counterparts. The Uber app shows surge pricing logic the website doesn't expose. A retail app serves real-time inventory and location-based pricing the desktop site aggregates away. A food delivery app exposes restaurant-level supply signals the web page never shows.

The technique for extracting this data is different from web scraping. There's no HTML to parse. Mobile apps communicate with backend APIs over HTTP — so the approach is to intercept those requests, understand the API, and automate calls to it directly.

This post walks through how that works, what defences you'll hit, and what makes it hard at scale.

All techniques here are for educational purposes and authorised security research. Reverse engineering apps in violation of their terms of service may carry legal risk. Evaluate your use case and jurisdiction before proceeding.

The Core Idea: Apps Are Just API Clients

A mobile app is a client that makes HTTP requests to backend services. When you open a product page, the app calls something like:

http

GET https://api.example.com/v2/products/12345
Authorization: Bearer eyJhbGci...

It gets back a JSON response and renders it into a UI.

If you can capture that API call — endpoint, auth headers, request parameters — you can reproduce it in code. Instead of parsing a rendered UI, you call the API directly and get clean structured data, exactly what the app uses.

Step 1: Intercept Traffic with mitmproxy

mitmproxy is an open-source HTTPS proxy that sits between the app and the internet, recording every request and response.

Set it up in three steps:

Install and run:

bash

pip install mitmproxy
mitmweb --listen-host 0.0.0.0 --listen-port 8080

This opens a web UI at http://localhost:8081 where you can watch traffic in real time.

Configure the Android emulator to route through it: Go to Settings → Wi-Fi → Modify Network → Advanced → Manual Proxy, set the host to 10.0.2.2 and port to 8080.

Install mitmproxy's certificate inside the emulator by navigating to http://mitm.it in the emulator's browser. Without this, TLS connections will fail because the emulator won't trust mitmproxy's certificate.

Open the app and watch API calls appear in the mitmproxy UI. For most apps, this is enough to start capturing traffic.

Step 2: The Certificate Trust Problem (Android 7+)

Android 7 changed the rules: apps only trust system certificates by default, not user-installed ones. mitmproxy's certificate installs as a user cert, so many apps will refuse connections even after you've installed it.

Fix 1 — Push the cert to the system store (emulator only):

bash

adb root && adb remount
adb push ~/.mitmproxy/mitmproxy-ca-cert.cer /system/etc/security/cacerts/
adb reboot

After reboot, the cert is trusted by all apps.

Fix 2 — Patch the APK directly: Decompile the app with apktool, add a network_security_config.xml that enables user certificate trust, rebuild, and reinstall. This works on production apps where you can't root the device.

Step 3: SSL Pinning — The Real Defence

User certificate trust gets you past basic TLS inspection. But most apps with sensitive data use SSL pinning — they verify not just that the certificate is valid, but that it matches a specific key they've hardcoded.

When you MITM a pinned connection, the app sees mitmproxy's certificate, fails the pin check, and silently refuses the connection.

How to detect it: In mitmproxy, connections that establish and then immediately fail with a TLS error are pinning signals.

How to bypass it — Frida:

Frida is a dynamic instrumentation tool that injects JavaScript into a running app to intercept and modify function calls. The most common use case is patching the certificate validation functions at runtime:

bash

# Install Frida and push the server binary to the emulator
pip install frida-tools
adb push frida-server /data/local/tmp/frida-server
adb shell chmod 755 /data/local/tmp/frida-server
adb shell /data/local/tmp/frida-server &

# Run the universal SSL unpinning script against the target app
frida --codeshare sowdust/universal-android-ssl-pinning-bypass-2 -f com.target.app

This patches the most common pinning implementations (Android's TrustManager, OkHttp's CertificatePinner, and several others) without modifying the APK.

If the generic script doesn't work, the app is using a custom or native pinning implementation. At that point you decompile with JADX, search for certificate validation code, and write a targeted Frida hook:

javascript

Java.perform(function() {
  var CertPinner = Java.use('okhttp3.CertificatePinner');
  CertPinner.check.overload('java.lang.String', 'java.util.List').implementation = function() {
    return; // bypass — do nothing instead of checking
  };
});

Step 4: Capture and Understand the API

With pinning bypassed, use the app normally and watch the traffic in mitmproxy. You're looking for:

Endpoint patterns — usually /v1/products/{id} or /api/search?q=...
Authentication — Authorization: Bearer TOKEN, X-API-Key, custom headers
Request parameters — query strings, JSON body fields
Response format — JSON (easy), Protobuf (binary, harder)

Save a session for offline analysis:

bash

mitmdump -w capture.mitm   # record
mitmproxy -r capture.mitm  # replay and inspect

Protobuf responses look like binary garbage. They're identifiable by Content-Type: application/x-protobuf or binary response bodies. You can decode them without a schema using protoc --decode_raw, which shows field numbers and values. If the app's APK contains .proto files (many do), you can decode with full field names.

Step 5: Automate the API Calls

Once you understand the endpoints and authentication, skip the app entirely and call the API directly:

python

import requests, time, random

AUTH_TOKEN = "eyJhbGci..."  # captured from mitmproxy

def get_product(product_id):
    return requests.get(
        f"https://api.example.com/v2/products/{product_id}",
        headers={
            "Authorization": f"Bearer {AUTH_TOKEN}",
            "X-App-Version": "4.2.1",
            "User-Agent": "ExampleApp/4.2.1 (Android 13; Pixel 6)",
        },
        timeout=10,
    ).json()

for product_id in product_ids:
    data = get_product(product_id)
    time.sleep(random.uniform(0.5, 2.0))

What Makes This Hard at Scale

Token expiry. Mobile tokens expire — sometimes hourly. You need to detect 401 responses and re-authenticate. If re-auth requires a device challenge (SMS, TOTP), you may need a real device in the loop.

Device fingerprinting. Many apps send device identifiers (X-Device-ID, hardware fingerprints) with every request. Sending the same ID for millions of requests gets flagged. You need to spoof identifiers consistently — and rotate them.

API-layer bot detection. Mobile APIs increasingly run the same behavioral analysis as websites: request frequency, inter-request timing, user-agent consistency. The countermeasures are the same — realistic timing, rotating identifiers, avoiding automation patterns.

App updates. An update can change endpoint URLs, authentication schemes, response formats, or pin new certificates. Everything you've reverse-engineered can break overnight. Production pipelines need monitoring that detects response structure changes immediately.

When Mobile Scraping Is Worth It

Most data needs are better served by web scraping. Mobile app extraction makes sense when:

The data only exists in the mobile API — no equivalent web endpoint
The mobile API returns significantly richer data (real-time pricing, granular inventory, location-based signals the website aggregates)
You need data the app shows live that the web page delays or omits

It's meaningfully more complex to build and maintain than web scraping. The expertise required — traffic analysis, APK decompilation, dynamic instrumentation — is specialised. Budget accordingly, and build monitoring into the pipeline from day one.

Mobile App Data Extraction: HTTP Interception, SSL Pinning Bypass, and API Replay

The Core Idea: Apps Are Just API Clients

Step 1: Intercept Traffic with mitmproxy

Step 2: The Certificate Trust Problem (Android 7+)

Step 3: SSL Pinning — The Real Defence

Step 4: Capture and Understand the API

Step 5: Automate the API Calls

What Makes This Hard at Scale

When Mobile Scraping Is Worth It

Related posts

How Anti-Bot Systems Like PerimeterX Detect Web Scrapers: A Technical Breakdown

Web Scraping Proxies Explained: Residential, Datacenter, and ISP Proxies Compared