Roadmap

An incomplete roadmap for Conc.

Conc is in active development and support. The current releases of Conc are Beta releases.

It is currently released as a pip-installable Python package. For the latest release, install from Pypi (pip install conc). Add the -U flag to upgrade if you are already running Conc.

The code available on Github may be pre-release and may be ahead of the Pypi release. The code on Github may include bug fixes and new functionality, including development that is in-progress, potentially incomplete or subject to change. For the latest pre-release functionality install from Github (pip install git+https://github.com/polsci/conc.git).

The documentation reflects the most recent functionality. See the Roadmap below for upcoming features and the CHANGELOG for notes on previous releases.

Short-term

• add tutorial/walkthrough/ getting started notebook
• add citation information
• document use of Conc results in other Python libraries (e.g. Pandas) via tutorial/walkthrough, results module (include link on README principles)
• Corpus tokenize support for functionality from earlier versions of Conc for wildcards, multiple strings, case insensitive tokenization
• extend caching support to all intensive reports, revise storage of cached results for in-memory/disk option
• relegate some logger warnings to debug level and audit logger messages for consistency and clarity for users
• add support for build from datasets library
• ngrams method - implement case handling
• get_ngrams_by_index - implement case handling
• improve concordance ordering so not fixed options e.g. include 3R1R2R
• improve ngram support for ngram token position beyond LEFT/RIGHT (i.e. define positions relative to ngram, or ANY)
• concordancing - add in ordering by metadata columns or doc
• annotations support for spaCy POS, TAG, SENT_START, LEMMA
• shift more processing from in-memory to polars with support for streaming or in-memory processing
• revisit polars streaming - potentially implement a batched write for very large files i.e. splitting vocab/tokens files into smaller chunks to reduce memory usage.

Medium-term

• Support for processing backends other than spaCy (i.e. other tokenizers)