Wrapper around the OpenTrep library
Table of contents generated with markdown-toc
OpenTREP aims at providing a clean API, its corresponding C++ implementation and Python extension, for parsing travel-/transport-focused requests. It powers the https://transport-search.org Web site (as well as its newer version, https://www2.transport-search.org).
As part of the OpenTREP releases,
there is a Python extension, named
opentrep and published on PyPi.
That Python extension comes as a so-called binary wheel only for MacOS
(which makes it very easy to install on that platform, then).
Contributions are welcome
to also build wheels for other platforms (e.g., Linux, including WSL
on MS Windows). So far, for those other platforms, the Python extension
has to be built from the source distribution, also
available on PyPi in the same place.
OpenTrepWrapper (this project) provides a simple Python wrapper around the OpenTREP Python extension, and is also released on PyPi.
pyenv and pipenvpyenv and pipenvSee https://github.com/machine-learning-helpers/induction-python/tree/master/installation/virtual-env for the details.
$ pushd ~/.pyenv && git pull && popd # if pyenv is in ~/.pyenv
$ pyenv install 3.8.5 && pyenv global 3.8.5 && \
pip install -U pip pipenv && pyenv global system
$ mkdir -p ~/dev/geo/trep && \
git clone https://github.com/trep/wrapper.git ~/dev/geo/trep-wrapper
$ cd ~/dev/geo/trep-wrapper
$ pipenv --rm && rm -f Pipfile.lock && \
pipenv install && pipenv install --dev
This module does not install OpenTREP itself; the overview gives the general procedure on how to do it.
Also, on RedHat/CentOS/Fedora, OpenTREP is packaged and can therefore easily
be installed with the native packager manager (dnf or yum).
Alternatively, on most of the platforms, it can be installed by following
the instructions in the
OpenTREP README.md file.
Basically, a few C++-, Python- and database-related development tools
have to be installed. The
Docker images for C++ projects GitHub repository
gives all the details in the Dockerfile of the corresponding Linux
distribution folders, e.g.,
ubuntu2004/Dockerfile
for Ubuntu 20.04 LTS (Focal Fossa).
And then the OpenTREP Python extension may simply be installed with pip
(either invoked through the native Python interpreter or through pipenv
with a Python interpreter installed by pyenv).
Note that the OpenTREP Python extension is not referred to in the
Pipfile file,
in order to have a minimum viable Python virtual environment installed
by pipenv. If opentrep were referred to in the Pipfile file,
the installation of the Python virtual environment may just stop in
error after an attempt to compile the OpenTREP Python extension,
and it would be hard to debug, without evenhaving a minimal viable
Python virtual environment.
pipenv,
The installation of the OpenTREP Python extension should go something like:
user@laptop$ pipenv shell
Launching subshell in virtual environment…
(trep-wrapper-FkiBIXof) ✔ python -V
Python 3.8.5
(trep-wrapper-FkiBIXof) ✔ pip install -U opentrep
Collecting opentrep
Using cached opentrep-0.7.7.post6-cp38-cp38-macosx_10_15_x86_64.whl (10.3 MB)
Installing collected packages: opentrep
Successfully installed opentrep-0.7.7.post6
(trep-wrapper-FkiBIXof) ✔ exit
opentrep) has been installed
properly:
user@laptop$ pipenv run python -mpip freeze|grep opentrep
opentrep==0.7.7.post6
$ pipenv run python -mpip show opentrep
Name: opentrep
Version: 0.7.7.post6
Summary: Simple Python wrapper for OpenTREP
Home-page: https://github.com/trep/opentrep
Location: ~/.local/share/virtualenvs/trep-wrapper-FkiBIXof/lib/python3.8/site-packages
$ export TREP_VENV="${HOME}/.local/share/virtualenvs/trep-wrapper-FkiBIXof"
$ export PYTHONPATH="${TREP_VENV}/lib:${TREP_VENV}/lib/python3.8/site-packages/pyopentrep"
$ export LD_LIBRARY_PATH="${TREP_VENV}/lib"
$ export PATH="${TREP_VENV}/bin:${PATH}"
$ export TREP_DIR="${HOME}/dev/deliveries/opentrep-latest"
$ export PYTHONPATH="${TREP_DIR}/lib/python3.8/site-packages/pyopentrep:${TREP_DIR}/lib"
$ export LD_LIBRARY_PATH="${TREP_DIR}/lib"
$ export PATH="${TREP_DIR}/bin:${PATH}"
pip, without any virtual environment
(as it does not play well with the MacOS Python so-called framework):
$ python3 -mpip uninstall -y opentrep
$ python3 -mpip install -U opentrep
$ export PYTHONPATH=/usr/local/lib/python3.8/site-packages/pyopentrep:/usr/local/lib
$ export DYLD_LIBRARY_PATH=/usr/local/lib
pip, either with the native Python framework (MacOS)
or with pipenv (Linux, including WSL on MS Windows)>>> import opentraveldata
>>> myOPTD = opentraveldata.OpenTravelData()
>>> myOPTD.downloadFilesIfNeeded()
>>> myOPTD
OpenTravelData:
Local IATA/ICAO POR file: /tmp/opentraveldata/optd_por_public_all.csv
Local UN/LOCODE POR file: /tmp/opentraveldata/optd_por_unlc.csv
>>> myOPTD.fileSizes()
(44079255, 4888752)
>>>
-i option of pyopentrep.py,
so as to index the full OpenTravelData (OPTD) POR (points of reference)
data file
$ pipenv run python ${TREP_VENV}/lib/python3.8/site-packages/pyopentrep/pyopentrep.py -p /tmp/opentraveldata/optd_por_public_all.csv -i
$ DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib ASAN_OPTIONS=detect_container_overflow=0 /usr/local/Cellar/python\@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python /usr/local/lib/python3.8/site-packages/pyopentrep/pyopentrep.py -p /tmp/opentraveldata/optd_por_public_all.csv -i
OPTD-maintained list of POR (points of reference): '/tmp/opentraveldata/optd_por_public_all.csv'
Xapian-based travel database/index: '/tmp/opentrep/xapian_traveldb0'
SQLite database: '/tmp/opentrep/sqlite_travel.db'
Perform the indexation of the (Xapian-based) travel database.
That operation may take several minutes on some slow machines.
It takes less than 20 seconds on fast ones...
Number of actually parsed records: 1,000, out of 103,394 records in the POR data file so far
...
Number of actually parsed records: 20,000, out of 122,394 records in the POR data file so far
Done. Indexed 20335 POR (points of reference)
$ pipenv run python
$ DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib ASAN_OPTIONS=detect_container_overflow=0 /usr/local/Cellar/python\@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python
>>> import pyopentrep
>>> openTrepLibrary = pyopentrep.OpenTrepSearcher()
>>> initOK = openTrepLibrary.init ('/tmp/opentraveldata/optd_por_public.csv', '/tmp/opentrep/xapian_traveldb', 'sqlite', '/tmp/opentrep/sqlite_travel.db', 0, False, True, True, 'pyopentrep.log')
>>> initOK
True
>>> openTrepLibrary.search('S', 'nce sfo')
'NCE/0,SFO/0'
$ pipenv run python
Python 3.8.5 (default, Jul 24 2020, 13:35:18)
>>>
$ ASAN_OPTIONS=detect_container_overflow=0 \
DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib \
/usr/local/Cellar/python\@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python
Python 3.8.5 (default, Jul 24 2020, 13:35:18)
>>>
>>> import OpenTrepWrapper
>>> otp = OpenTrepWrapper.OpenTrepLib()
>>> otp.init_cpp_extension(por_path=None,
xapian_index_path="/tmp/opentrep/xapian_traveldb",
sql_db_type="nodb",
sql_db_conn_str=None,
deployment_nb=0,
log_path="test_OpenTrepWrapper.log",
log_level=5)
>>>
>>> otp.index()
>>> otp.search(search_string="nce sfo", outputFormat="S")
([(89.8466, 'NCE'), (357.45599999999996, 'SFO')], '')
------------------
otp.search(search_string=”nce sfo”, outputFormat=”F”) search_string: nce sfo Raw result from the OpenTrep library: 1. NCE-A-6299418, 8.16788%, Nice Côte d’Azur International Airport, Nice Cote d’Azur International Airport, LFMN, , FRNCE, , 0, 1970-Jan-01, 2999-Dec-31, , NCE|2990440|Nice|Nice|FR|PAC, PAC, FR, , France, 427, France, EUR, NA, Europe, 43.6584, 7.21587, S, AIRP, 93, Provence-Alpes-Côte d’Azur, Provence-Alpes-Cote d’Azur, 06, Alpes-Maritimes, Alpes-Maritimes, 062, 06088, 0, 3, 5, Europe/Paris, 1, 2, 1, 2018-Dec-05, , https://en.wikipedia.org/wiki/Nice_C%C3%B4te_d%27Azur_Airport, 43.6627, 7.20787, nce, nce, 8984.66%, 0, 0 2. SFO-C-5391959, 32.496%, San Francisco, San Francisco, , , USSFO, , 0, 1970-Jan-01, 2999-Dec-31, , SFO|5391959|San Francisco|San Francisco|US|CA, CA, US, , United States, 91, California, USD, NA, North America, 37.7749, -122.419, P, PPLA2, CA, California, California, 075, City and County of San Francisco, City and County of San Francisco, Z, , 864816, 16, 28, America/Los_Angeles, -8, -7, -8, 2019-Sep-05, SFO, https://en.wikipedia.org/wiki/San_Francisco, 0, 0, sfo, sfo, 35745.6%, 0, 0
+ JSON output format:
```python
>>> otp.search(search_string="nce sfo", outputFormat="J")
search_string: nce sfo
Raw (JSON) result from the OpenTrep library:
{
"locations": [
{
"iata_code": "NCE",
"icao_code": "LFMN",
"geonames_id": "6299418",
"feature_class": "S",
"feature_code": "AIRP",
...
},
{
"iata_code": "SFO",
"icao_code": "",
"geonames_id": "5391959",
"feature_class": "P",
"feature_code": "PPLA2",
...
}
]
}
------------------
>>> otp.search(search_string="nce sfo", outputFormat="I")
search_string: nce sfo
JSON format => recognised place (city/airport) codes:
NCE-LFMN-6299418 (8.1678768123135352%) / NCE: 43.658411000000001 7.2158720000000001; SFO--5391959 (32.496021550940505%) / SFO: 37.774929999999998 -122.41942;
------------------
>>> quit()
====================== 3 passed in 1.35s =======================
# Release OpenTrepWrapper to PyPi
* Build the Python artifacts for OpenTrepWrapper:
```bash
$ rm -rf dist build */*.egg-info *.egg-info .tox MANIFEST
$ pipenv run python setup.py sdist bdist_wheel
$ ls -lFh dist
total 56
-rw-r--r-- 1 user staff 7.7K Mar 2 11:14 OpenTrepWrapper-0.7.7.post1-py3-none-any.whl
-rw-r--r-- 1 user staff 7.3K Mar 2 11:14 OpenTrepWrapper-0.7.7.post1-py3.8.egg
-rw-r--r-- 1 user staff 8.4K Mar 2 11:14 OpenTrepWrapper-0.7.7.post1.tar.gz
$ pipenv run twine upload -u __token__ dist/*