How to cross-compile Python for Android

When it comes to cross-compiling Python for Android, I've followed Gabriel's blog post @ http://mdqinc.com/blog/2011/09/cross-compiling-python-for-android/ and I was successful in creating an Arm-based Python executable (and related libraries) in little or no time.
Gabriel has a lot of the initial steps just verbally described, but I've come up with a shell script that allows you to automate the entire process. Of course you need to replace <path-to-android-ndk> with the directory where you have installed the NDK.
Here it goes.

ANDROID_NDK=<path-to-android-ndk>
ANDROID_ABI="armeabi-v7a"
ANDROID_NATIVE_API_LEVEL="android-8"
PYTHON_VERSION="2.6.2"

output_dir=$1
mkdir -p $output_dir
cd $output_dir
# get Python source tarball
wget http://www.python.org/ftp/python/$PYTHON_VERSION/Python-$PYTHON_VERSION.tgz

# create Python Host version
tar zxvf Python-$PYTHON_VERSION.tgz
mv Python-$PYTHON_VERSION Host-Python-$PYTHON_VERSION-src
cd Host-Python-$PYTHON_VERSION-src
./configure --prefix=$output_dir/Host-Python-$PYTHON_VERSION
make
make install

cd $output_dir
# create Python Cross-compiled version for Android
tar zxvf Python-$PYTHON_VERSION.tgz
mv Python-$PYTHON_VERSION Android-Python-$PYTHON_VERSION-src
cd Android-Python-$PYTHON_VERSION-src

# get and apply Python patch
wget -o Python-2.6.2-android.patch https://sites.google.com/site/dgtechblogscripts/Python-2.6.2-android.patch
patch -p0 < Python-2.6.2-android.patch
# fix setup.py
mv setup.py setup.py.orig
cat setup.py.orig | awk '{ if (NR==316) {print "    " $0} else {print $0}}' > setup.py

MY_HOSTPYTHON=$output_dir/Host-Python-$PYTHON_VERSION/bin/python
MY_HOSTPGEN=$output_dir/Host-Python-$PYTHON_VERSION-src/Parser/pgen

export ANDROID_NDK
export PATH="$ANDROID_NDK/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/:$ANDROID_NDK:$ANDROID_NDK/tools:/usr/local/bin:/usr/bin:/bin"
export ARCH=$ANDROID_ABI
export CFLAGS="-DANDROID -mandroid -fomit-frame-pointer --sysroot $ANDROID_NDK/platforms/$ANDROID_NATIVE_API_LEVEL/arch-arm"
export CXXFLAGS="$CFLAGS"
export CC="arm-linux-androideabi-gcc $CFLAGS"
export CXX="arm-linux-androideabi-g++ $CXXFLAGS"
export AR="arm-linux-androideabi-ar"
export RANLIB="arm-linux-androideabi-ranlib"
export STRIP="arm-linux-androideabi-strip --strip-unneeded"
export MAKE="make -j4 install HOSTPYTHON=$MY_HOSTPYTHON HOSTPGEN=$MY_HOSTPGEN CROSS_COMPILE=arm-eabi- CROSS_COMPILE_TARGET=yes"

./configure LDFLAGS="-Wl,--allow-shlib-undefined" CFLAGS="-mandroid -fomit-frame-pointer --sysroot $ANDROID_NDK/platforms/$ANDROID_NATIVE_API_LEVEL/arch-arm" HOSTPYTHON=$MY_HOSTPYTHON HOSTPGEN=$MY_HOSTPGEN --host=arm-eabi --build=i686-pc-linux-gnu --enable-shared --prefix="$output_dir/Android-Python-$PYTHON_VERSION"
sed -i "s|^INSTSONAME=\(.*.so\).*|INSTSONAME=\\1|g" Makefile
$MAKE
 

Read More

Installing the Android NDK

These instructions refer to a Ubuntu 12.04.1 LTS system running on an Intel processor.

• Head to the http://developer.android.com/tools/sdk/ndk/index.html web site and download the latest version of the Android NDK for Linux (release r8b for example):
  

  $ wget http://dl.google.com/android/ndk/android-ndk-r8b-linux-x86.tar.bz2

  
  
• Unzip and untar the downloaded file in your preferred directory:

  $ bunzip2 android-ndk-r8b-linux-x86.tar.bz2
  $ tar zxvf android-sdk_r20.0.3-linux.tgz

You should now have a directory called  android-ndk-r8b containing release 8b of the Android NDK.

Read More

Process Attribution In Network Traffic

Author: Phil -at- DigitalOperatives

Overview

Digital Operatives recently completed a DARPA Cyber Fast Track (CFT) contract called Process Attribution in Network Traffic (PAINT). The goal of this project was to build an easy to use software product that is capable of attributing originating and receiving Process IDs (PIDs) and Process Names for TCP/IP packets on a Windows system in real-time. We chose Wireshark as the implementation platform since it is a very popular and one of the most useful open-source network analysis tools available today.

Our challenge in designing a packet-to-process attribution scheme was to take process-related metadata available at the TCP/IP layer and correlate it with discrete frames of network activity available at the bottom of the network stack (the NDIS layer). We considered several potential approaches to solving this problem, including implementing kernel-mode drivers, but we ultimately found that it was possible using a novel application of Event Tracing for Windows (ETW). Beginning with Windows 7, the various components in the network stack are able to generate trace messages via ETW that provide a way to collect correlated end-to-end network stack activity.

Motivation

We believe network traffic analysis and visualization are regarded as fundamental pillars of cyber defense. Wireshark is one of the most well-known and widely used network analysis tools. It’s useful for dynamic malware analysis and reverse engineering due to its powerful system of filters that allow the analyst to focus on network packets between selected hosts or using only certain protocols. However, due to the way it traces network traffic, it cannot determine the process on the host system that is responsible for sending or receiving that traffic. Thus, malware is able to hide its network communications semi-anonymously in the noise of legitimate network activity using similar protocols and data flow patterns. By PAINT-ing Wireshark, we are able to attribute all TCP/IP traffic to responsible processes and allow human and machine analysts to detect anomalies and potential malware traffic easily.

Comparison to Other Process Attribution Software Products

We believe PAINT/Wireshark is one of a kind but there are a couple software products that partially achieve PAINT/Wireshark’s real-time TCP/IP process attribution and packet capture functionalities on the Windows platform: Network Monitor and Process Monitor.

• Microsoft Network Monitor (NetMon) is a free network packet capturing software from Microsoft. It looks up the TCB (TCP Control Block) in semi-real-time to attribute network packets to processes. It can, however, only attribute TCP traffic and not UDP traffic. Another NetMon limitation is that it only periodically queries the TCB due to performance constraints. It’s possible for a TCP connection to exist between the TCB polls and cause a false-negative.

• Process Monitor is another Microsoft (SysInternals) tool. It logs virtually all attributable application activities including network activities. However, it can only attribute network connection activity and does not capture the traffic content like PAINT/Wireshark or Network Monitor.

• On Linux, there is a tool that partially achieves what we are doing with PAINT, which is a program called NetHogs. It can attribute network traffic usage totals to individual processes, but it lacks the ability to attribute data at a per-packet level, and is limited to TCP only.

• On the Mac OS X platform there is Private Eye. It is a simplified and user-friendlier tool similar to the Process Monitor. It does not attribute traffic at a per-packet level.

Event Tracing for Windows

Event Tracing for Windows (ETW) is a logging mechanism provided by Windows. It enables any user or kernel processes to log an event and lets any user or kernel processes subscribe to and receive events as broadly or as selectively as desired. Each ETW event contains an Activity ID that is used to perform a call trace within a single thread of control. ETW was first introduced in Windows 2000 and has been improving with subsequent Windows releases as more components of the system provide tracing information via ETW. PAINT/Wireshark requires the end-to-end ETW tracing information from the network stack components, which was first introduced in Windows 7.

Technical Overview

We identified 3 ETW providers built-in to Windows that gave us raw network packet capture, TCPIP events with PID, and a way to correlate the two: Microsoft-Windows-NDIS-PacketCapture (referred to as “ETW-NDIS” henceforth for brevity), Microsoft-Windows-TCPIP (referred to as “ETW-TCPIP” henceforth), and Microsoft-Windows-Networking-Correlation (referred to as “ETW-Correlation” henceforth), respectively.

While both the ETW-TCPIP and ETW-NDIS events contain the Activity ID field, these fields cannot be correlated in the traditional ETW event correlation fashion because they occur in different OS threads. Thankfully the ETW-Correlation provider sends out Activity Transfer events that correlates the ETW-TCPIP Activity IDs to the ETW-NDIS Activity IDs. Correlating these events in real-time proved to be non-trivial as it originally seemed, and we believe there’s at least a small room for improvement in the correlation state-machine for both attribution and run-time performance.

We modified Wireshark 1.6.5 to introduce PID and process name columns. We introduced a new binary file (.process) that pairs with the .pcap file. The current version of PAINT looks for the .process file whenever a .pcap file is opened and reads the process information to display (functions normally when it’s not found). We created our own dumpcap.exe implementation that replaces the original dumpcap.exe to provide the packet captures and the .process file using the ETW-NDIS provider.

The Result

As you can see the result looks great! Process information seamlessly integrates into Wireshark 1.6.5 in real-time and file-open operation modes.

Performance wise, we believe we achieve 0% false-positive (no wrongly attributed traffic) and 0% false-negative (no missing correct attribution) rates for UDP traffic and 0% false-positive and ~20% false-negative rates for TCP traffic. Currently, we can attribute about 100 packets per second on a typical Core2 machine.

We plan on making our custom Wireshark code available so that other pcap/PID providers can follow the same design paradigm enabling network traffic process attribution in Wireshark. Wireshark is not affected by our code if the original dumpcap.exe is used.

Please inquire at contact -at- digitaloperatives -dot- com for more information. Also stay tuned for details on PAINT Enterprise, an enterprise tool enabling easy-to-manage whole network process attribution.

Read More

Problems with new version of rpmbuild

The Problem

With the new version of rpmbuild installed on CentOS 6.x, if you try to use an old RPM spec file, you will get an error like the following:

error: File not found: <path>/BUILDROOT/<product>-6.7.x86_64/<filename>

Previously rpmbuild would install and look for files under the rpm_top/BUILD directory, while now it looks under the new BUILDROOT directory.

The Solution

This is what I ended up doing to solve the problem. When I call rpmbuild I now define a new variable, for example:

$ centos_release=$(lsb_release -rs | sed 's/\.//')

$ rpmbuild --define "centos_release $centos_release" ... -bb specfile 

then I changed my spec file to contain something like the following:

%if %{centos_release} >= 60

%setup -q -c -n %{buildroot}/usr/local/ 
cp -a -r %{buildroot}/usr/local/ /usr/
 

%else
 

%setup -q -c -n usr/local/
%install
cp -a -r $PWD /usr/
 

%endif

Another related problem

Suppose you have a single tar ball and you want to create two or more RPMs using different spec files. The new version of rpmbuild automatically cleans the BUILDROOT directory after completing the targets for a given spec file. If you want to save time and have the second spec file just look for the BUILDROOT created by the first spec it won't find it because of the automatic clean. So it seems like you are forced to untar and install under the BUILDROOT your file over and over again.
A simple solution to this problem is to add a %clean directive to your spec file and do nothing to disable to automatic cleaning done by rpmbuild. Add the following line at the bottom of your spec file:

%clean

Read More

Installing the Android SDK

These instructions refer to a Ubuntu 12.04.1 LTS system running on an Intel processor.

• Head to the developer.android.com web site and download the latest version of the Android SDK for Linux (release 20.0.3 for example): http://dl.google.com/android/android-sdk_r20.0.3-linux.tgz 
• Untar the downloaded file in your preferred directory:

  $ tar zxvf android-sdk_r20.0.3-linux.tgz

You should now have a directory called  android-sdk-linux containing release 20.0.3 of the Android SDK.

Read More

Porting your Legacy C/C++ project to Android

This is a recurring problem people have often: trying to port a big C/C++ project to the Android platform. You have thousands of lines of tested and working C/C++ code that you want to reuse and access from your Android application. So far Android only provides an SDK for Java applications, only supports devices with ARM, MIPS and x86 architectures, and several steps are necessary if you want to port your code and call it from Java as a library. By large the biggest deployment is for ARM processor devices, so we are initially focused on porting our code to this architecture.
Porting your C/C++ project to the Android platform entails then the following steps:

1. Cross-compile the C/C++ code for the ARM processor (or any other supported) architecture.
2. Develop a JNI module allowing an Android application to natively call the C/C++ code from Java.
3. Develop an Android application using the JNI module.
4. Run and debug the app on an Android device.

Developing an Android application requires the Android Software Development Kit (SDK), while cross-compiling your C/C++ code and developing the JNI module for Android requires the Android Native Development Kit (NDK). Both these kits run on Windows XP/Win7, Mac OS 10.5.8 or later (Intel), or Ubuntu Linux 8.04 or later.

My intention is to publish on this blog several posts with detailed instructions on how to accomplish the above mentioned steps. The instructions are based on other posts you can search on the internet and on my experience with Android. All the instructions are based on the development and testing environments described below.

Development Environment

• Processor: Intel(R) Core(TM) 2 Duo CPU     E8600  @ 3.33GHz
• RAM: 4GB DDR2 SDRAM DIMM 240-pin
• OS: Ubuntu Linux 12.04
• Eclipse IDE Juno
• Android SDK r20
• Android NDK r8

Testing Environment

• LG Thrill 4G - Android 2.2
• Samsung Galaxy Tab 8.9 - Android 3.2
• Samsung Galaxy Tab 2 7.0 - Android 4.01
• Samsung Galaxy S III - Android 4.0.4

Read More

Transferring Files from a computer to your Android device

Android devices have file systems similar to regular computers. Subject to permissions restrictions, we can transfer files from a computer to a device (connected to the computer by USB) or emulator, or from the device or emulator to the computer. There are several ways to transfer files to an Android device:

1. Using Eclipse
   Select the DDMS perspective for Eclipse and use the File Explorer tab to select the directory where you want to tranfer a file to (a drawback of using this method is that you can only tranfer one file at a time).
2. Using the ADB
   You can tranfer entire directories using the ADB command line interface. For example to transfer the entire content of a director to an Android device connected to a Ubuntu computer via the USB port:
   

   $ adb push <path to directory to copy> /mnt/sdcard/<target directory>

3. Install an ssh daemon on the Android device and then use sftp from your workstation. For example you can download the free SSHDroid app from the Google market and install it on your device, run the app an find out about the IP address to connect to. Then from a terminal window in Linux you can connect like this (works on Ubuntu 12.04):

   $ ssh -p 2222 root@ip-address-mentioned-in-SSHDroid

   
   

Read More

Welcome

After many years of searching the web for tips and tricks about computers, operating systems and programming languages, it's finally time to give back and post solutions to common problems I'm facing during my professional career.

I hope you will find this blog useful.

Read More