Install Python and SQLite from Source#

Introduction#

I was writing some Python to pull text from pdf files and put them into a sqlite database so that I could perform full text searches for various keywords and phrases. I was able to extract the text and put it in the database. I was using Anaconda on windows to do this fully expecting to be able to do the same on Ubuntu (14.04). I had to replace the sqlite3.dll with the latest one from here because the sqlite3.dll included with Anaconda didn’t have FTS4 or FTS5 enabled. This was as simple as copying the new dll over top of the old one and running this script to verify the changes:

#!/usr/bin/env python3
#-*- coding:utf-8 -*-

import sqlite3, platform

print('sqlite {}'.format(sqlite3.version_info))
print('sqlite {}'.format(sqlite3.sqlite_version_info))
print('sqlite {}'.format(sqlite3.__file__))

db = sqlite3.connect(':memory:') # memory stream
cursor = db.cursor()

results = cursor.execute("pragma compile_options")

for r in results:
    print(r)

print()
print("Operating System: {} {}".format(platform.system(), platform.release()))
print("Platform: {} {}".format(platform.python_implementation(),platform.python_version()))
print("SQLite: {}".format(sqlite3.sqlite_version))

When I tried to run it on Ubuntu with Anaconda installed in my home folder, I couldn’t get it to work as expected. I replaced the sqlite3 binary with the pre-complied version from the sqlite website. That didn’t work. I came to the conclusion that I would have to compile sqlite from source, including the proper switches for the full text searching and compile python from source making sure that it used the new sqlite library.

Most of the information that I found on the net relating to this was for installing python system wide. I didn’t want to do this. I wanted an installation that was similar to what Anaconda does. I wanted a local installation that didn’t require sudo privileges and was easy to upgrade when new code was available.

Dependencies#

Here are the required dependencies to build both sqlite and python:

$ sudo apt-get install build-essential
$ sudo apt-get install bzip2 libbz2-dev
$ sudo apt-get install libncursesw5-dev
$ sudo apt-get install lzma-dev liblz-dev liblzma-dev
$ sudo apt-get install tk8.5-dev
$ sudo apt-get install libreadline6 libreadline6-dev
$ sudo apt-get install libssl-dev
$ sudo apt-get install libgdbm-dev
$ sudo apt-get install libc6-dev
$ sudo apt-get install tk-dev

Sqlite#

References#

Download#

The latest version of the source code can be here (I used the autoconfig version): https://www.sqlite.org/download.html

Create a folder to download and extract the source code to:

$ mkdir ~/tmp/compile
$ cd ~/tmp/compile

Download the source code (version 3.12 was the current one as of this writing):

$ wget https://www.sqlite.org/2016/sqlite-autoconf-3120200.tar.gz

Extract the source code and navigate to the new directory:

$ tar xf ./sqlite-autoconf-3120200.tar.gz
$ cd ./sqlite-autoconf-3120200.tar.gz

Configure Make#

We have to run configure before we can do anything else:

$ ./configure --prefix=/home/troy/opt/sqlite/sqlite3 --disable-static --enable-fts5 --enable-json1 CFLAGS="-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1"

The switches breakdown as follows:

–prefix=/home/troy/opt/sqlite/sqlite3 <- This is where we want the binaries built. I have an opt folder in ~ where I will be placing custom built software.
–disable-static <- disable static linking.
–enable-fts5 <- This is what I really want, full text search 5.
–enable-json1 <- Since we are building from scratch I may as well enable this.
CFLAGS=”-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1”
- -g
- -O2 <- Favor fast execution
- -DSQLITE_ENABLE_FTS3=1 <- enable FTS3
- -DSQLITE_ENABLE_FTS4=1 <- enable FTS4
- -DSQLITE_ENABLE_RTREE=1 <- enable spatial index

I didn’t have to enable FTS3 and FTS4, but I figured since I am building it, it may be useful for backwards compatibility some point. You can also run configure with the –help switch to list the available configuration options:

$ ./configure --help

Compile#

To compile the code, run the following commands:

$ make
$ make install

Shortcut#

Once sqlite has finished compiling and the binary is created we can create a symbolic link to it so that it is available on your path. I have bin folder in my home folder that is on my path. This is a nice place to put links to binaries on my system so I don’t have to add new paths all the time:

$ mkdir ~/bin
$ ln -s ~/opt/sqlite/sqlite3/bin/sqlite3 ~/bin/sqlite

Test#

Test everything out by running sqlite at the CLI:

$ sqlite

Issue the pragma compile_options to see what it was built with:

SQLite version 3.12.2 2016-04-18 17:30:31
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.

sqlite> pragma compile_options;

ENABLE_FTS3
ENABLE_FTS4
ENABLE_FTS5
ENABLE_JSON1
ENABLE_RTREE
SYSTEM_MALLOC
THREADSAFE=1

Everything looks great at this point and seems to be working as expected.

Python#

References#

Link a different version of sqlite to python during compile:

Notes#

You will need to make sure that you have sqlite compiled and running properly before going ahead with this portion.

Download#

Create a temporary directory to download and compile the source code if it doesn’t already exist:

$ mkdir ~/tmp/compile
$ cd ~/tmp/compile

Download the python source code from here:

$ wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz

Extract the source code and change to that directory:

$ tar xf ./Python-3.5.1.tgz
$ cd ./Python-3.5.1

Install Python#

Configure the build ensuring that the prefix switch is pointing to the location where you want the binaries built:

$ ./configure --prefix=/home/troy/opt/python/python3.5

The switch --prefix is set to the build location. This will build python in that particular folder. I could have let it go to a system wide folder, but I wanted something that mimicked what Anaconda does. I just need a separate python 3 installation that doesn’t interfere with the version that the system uses. I will probably go this route after the upgrade to 16.04 LTS.

Issue the make command:

$ make

Note: After running make, check to see if it complains about any dependencies that are missing. If there are missing dependencies or issue, resolve them before proceeding. Once the dependencies are installed, run make again to confirm. The dependencies can usually be installed from the repos.

Issue the rest of the commands:

$ make test
$ make install

Unfortunately, the preceding steps will not link the python build to the sqlite that we want to use. We have to make some changes to the commands:

$ LD_RUN_PATH=$HOME/opt/sqlite/sqlite3/lib ./configure --prefix=/home/troy/opt/python/python3.5 LDFLAGS="-L$HOME/opt/sqlite/sqlite3/lib" CPPFLAGS="-I $HOME/opt/sqlite/sqlite3/include"
$ LD_RUN_PATH=$HOME/opt/sqlite/sqlite3/lib make
$ LD_RUN_PATH=$HOME/opt/sqlite/sqlite3/lib make test
$ LD_RUN_PATH=$HOME/opt/sqlite/sqlite3/lib make install

At this point you can run the small python script to verify that the new python build works as expected. At this point you’ll have to navigate to the bin folder of your new python installation for things to work. We’ll need to setup shortcuts.

Python Shortcut#

$ mkdir ~/bin
$ ln -s ~/opt/python/python3.5/bin/python3 ~/bin/python3
$ ln -s ~/opt/python/python3.5/bin/pip3 ~/bin/pip3

The first link is directly to the python3 symbolic link and we also link directly to pip3 as well to make software installation easier.

PIP#

Install setuptools and distribute:

$ pip3 install setuptools distribute

General form for Installing a module:

$ sudo pip3 install numpy

General form for Upgrading a module:

$ sudo pip3 install --upgrade numpy

Install Jupyter#

Installing Jupyter will cover a lot of the scientific stack that I like to use. So it is one of the first things that I install:

$ pip3 install jupyter

One of the odd things is that creating a symbolic link to the jupyter executable in our ~/bin folder doesn’t work for the jupyter notebook command. To get that working we have to add the path to the python bin folder to our $PATH. We need to modify the .bashrc:

$ vi .bashrc

We need to prefix our $PATH:

export PATH="/home/troy/bin:/home/troy/opt/python/python3.5/bin:$PATH"

Install Numpy#

Install:

$ pip3 install numpy

Upgrade:

$ pip3 install --upgrade numpy

Install Nose#

Let’s install nose so that we can run the numpy unit tests.

Install:

$ pip3 install nose

Upgrade:

$ pip3 install --upgrade nose

Running Numpy’s Unit Tests#

After installation, we can run the numpy unit tests:

$ python3 -c 'import numpy; numpy.test()'

Install Scipy#

Install:

$ pip3 install scipy

Upgrade:

$ pip3 install --upgrade scipy
```bash

### Testing Scipy

Launch a python shell:
```bash
$ python3

Import scipy and run the unit tests:

>>> import scipy
>>> scipy.__version__
>>> scipy.test('full')

Use Winscp to sync files from Windows to Linux Docker - A Cool Option

2016-04-24

Recent Posts

Categories

Archives

Install Python and SQLite from Source#

Introduction#

Dependencies#

Sqlite#

References#

Download#

Configure Make#

Compile#

Shortcut#

Test#

Python#

References#

Notes#

Download#

Install Python#

Python Shortcut#

PIP#

Install Jupyter#

Install Numpy#

Install Nose#

Running Numpy’s Unit Tests#

Install Scipy#