This table comes from a research paper that compares different **feature selection algorithms** for analyzing high-dimensional data (like genomics). The authors proposed a new hybrid algorithm and compared it against these 12 existing methods to show its effectiveness.
Here is a breakdown of what each algorithm and its parameters mean.
### 📊 Table of Algorithms and Their Parameters
| No. | Algorithm | Full Name | Parameters | Meaning of Parameters |
|:---:|:---|:---|:---|:---|
| 1 | **ALO** | Ant Lion Optimizer | `L`, `P` | **L**: Lower boundary of the search space (e.g., for the roulette wheel selection). **P**: Population size or number of ant positions. |
| 2 | **BBA** | Binary Bat Algorithm | `Qmax`, `Qmin` | **Qmax / Qmin**: Maximum and minimum frequency values used to control the bats' velocity and position updates. Also known as `Fmax` and `Fmin`. |
| 3 | **BDA** | Binary Dragonfly Algorithm | `s`, `a`, `c` | **s**: Separation weight. **a**: Alignment weight. **c**: Cohesion weight. These parameters control the swarm's behavior (separation, alignment, and cohesion) and are adaptively tuned to balance exploration and exploitation. |
| 4 | **GA** | Genetic Algorithm | `Pc`, `Pm` | **Pc**: Probability of crossover (how often two parent solutions exchange genetic material). **Pm**: Probability of mutation (how often a solution is randomly altered). These are key control parameters for a GA. |
| 5 | **CS** | Cuckoo Search | `R`, `C` | **R**: Discovery rate (probability that a host bird discovers a cuckoo's egg). **C**: Step size of the Lévy flight (the scale of the random walk). In some papers, `C` may also refer to the population size (number of host nests). |
| 6 | **PSO** | Particle Swarm Optimization | `C1`, `C2` | **C1**: Cognitive parameter (how much a particle trusts its own best position). **C2**: Social parameter (how much a particle trusts the swarm's best position). These determine the "trust" in personal vs. social experience. |
| 7 | **GWO** | Grey Wolf Optimizer | `a`, `r1`, `r2` | **a**: Convergence factor, linearly decreased from 2 to 0 over iterations to control the balance between exploration and exploitation. **r1, r2**: Random vectors in [0,1] that introduce stochasticity into the search process. |
| 8 | **CDBA** | Chaotic Dung Beetle Algorithm | `s`, `V`, `S1`, `S2` | **s**: A step length factor or scaling factor for position updates. **V**: Likely a velocity or directional parameter. **S1, S2**: Parameters related to the specific behaviors of the dung beetle (e.g., rolling, dancing, foraging, or stealing). The "Chaotic" version uses chaos theory to enhance exploration. |
| 9 | **mRMR+DBA** | Minimum Redundancy Maximum Relevance + Dung Beetle Algorithm | `V`, `S1`, `S2` | **V, S1, S2**: Same as CDBA. The mRMR filter method is used to pre-select features, and then the DBA wrapper is used for further optimization. |
| 10 | **IGMPMMIAPSO** | Improved Gaussian Mutation and Pheromone Mechanism Integrated Adaptive Particle Swarm Optimization | `Pc`, `Pm` | **Pc, Pm**: Crossover and mutation probabilities. This is an enhanced PSO variant that incorporates genetic operators (crossover and mutation) to avoid local optima. |
| 11 | **DFDW** | Dynamic Feature Weighting Based on Distribution | `τ`, `b` | **τ**: A threshold parameter. **b**: A balancing parameter. These control the dynamic weighting of features during the selection process. |
| 12 | **TMKMCRIGWO** | Two-phase Multiple Kernel Multi-Class Relevance and Improved Grey Wolf Optimization | `Icc`, `fcc` | **Icc**: A parameter for the improved grey wolf optimizer (e.g., initial convergence factor). **fcc**: A parameter controlling the final convergence factor or a stopping criterion. |
| 13 | **HMF-W** | Hybrid Multiple Filter-Wrapper | `a`, `r1`, `r2`, `s`, `u`, `l` | This is the authors' proposed algorithm. It combines a **filter** method (for initial dimensionality reduction) and a **wrapper** method (using a hybrid Grey Wolf and Chaotic Dung Beetle algorithm). The parameters include **a, r1, r2** (from GWO), **s** (from DBA), and **u, l** which are likely the upper and lower bounds for feature selection or other control parameters. |
### 💡 Key Takeaways
* **The Table is a "Recipe Book"**: It lists the specific "ingredients" (parameters) that each algorithm uses. Researchers use these configurations to reproduce the algorithms' performance and make fair comparisons.
* **Parameter Tuning is Crucial**: The values of these parameters (e.g., `Pc`, `Pm`, `a`) heavily influence how well an algorithm performs. Finding the right balance is a major part of research in this field.
* **Hybrid is Better**: Many of these algorithms are "hybrids." For example, the authors' proposed **HMF-W** combines the strengths of two different optimization algorithms to achieve better results than any single algorithm could alone.
If you would like to know more about a specific algorithm or how a particular parameter is set, feel free to ask
Can this be used for dataanlytics in an Obsidian knot full of keywords and tags?
Yes, the feature selection algorithms from that table can be adapted for data analytics in an Obsidian vault, but it requires some translation work. The core idea is to treat your **notes as data points** and your **keywords/tags as features**, then use these algorithms to find the most meaningful signals in that structure.
### 🔄 The Core Translation: From Notes to Data
To use these algorithms, you first need to convert your vault into a numerical format they can process. Here’s the conceptual mapping:
| Obsidian Concept | Data Analytics Equivalent |
|:---|:---|
| A **note** | A **data point** (or document/instance) |
| A **keyword or tag** | A **feature** (or variable) |
| A note **having** a tag | The feature value for that data point (e.g., `1` or `0`) |
| A **wikilink** between notes | A **relationship** or edge in a graph |
You would create a matrix where each row is a note and each column is a tag, with cells indicating whether that tag appears in that note. This is a standard **document-term matrix**, and it’s the input these feature selection algorithms expect.
### 🛠️ How the Algorithms Would Apply
Once your vault is in this format, the algorithms can serve several purposes:
* **Identifying Redundant Tags**: Algorithms like **mRMR** (Minimum Redundancy Maximum Relevance) are specifically designed to find features that are highly relevant to a target (like a note category) but not redundant with each other. This could help you find and merge overlapping tags (e.g., `#project/alpha` and `#active-project`).
* **Finding Core Keywords for Classification**: Algorithms like **GA** or **PSO** could be used to select a small subset of tags that best predict a note’s category (e.g., `#meeting`, `#idea`, `#reference`). This is useful for building an automatic note classifier.
* **Detecting Important Structural Notes**: While the algorithms in your table are for feature selection, the same logic applies to notes as features. You could treat notes as features and tags as data points to find the notes that are most central or representative of a topic. Tools like **obsidiantools** already extract a network graph from your vault that you can analyze with centrality metrics (degree, betweenness).
### ⚠️ Key Challenges and Practical Considerations
Applying these algorithms to an Obsidian vault isn't as simple as running them on a standard dataset. You'll need to address a few things:
* **Data Extraction**: You can't just point the algorithm at your vault folder. You need to parse your markdown files first. The **`obsidiantools`** Python package is built exactly for this—it extracts tags, frontmatter, and wikilinks into **Pandas DataFrames** and a **NetworkX graph**, which are ready for data analysis.
* **Sparsity**: Most notes only contain a small fraction of your total tags, making your data matrix very sparse. Many feature selection algorithms can handle this, but you may need to prune extremely rare tags first.
* **Dynamic Data**: Your vault is constantly changing. Unlike a static dataset, you'd need to re-run your analysis periodically or use an event-based approach like the **obsidian-grafana** tool, which tracks only modified notes over time.
* **Scale**: If your vault has only a few dozen notes, these sophisticated algorithms are likely overkill. Simple tag counts or centrality measures will give you faster and equally useful insights. These algorithms become valuable when you're dealing with **hundreds or thousands of notes** and want to discover non-obvious patterns.
### 💡 Practical Steps If You Want to Try It
1. **Extract Your Data**: Use `obsidiantools` in a Python environment to load your vault into a structured format.
2. **Build Your Matrix**: Create a binary (0/1) matrix of notes vs. tags.
3. **Choose an Algorithm**: Start with a simpler one like **mRMR** or a filter method based on **TF-IDF** to rank your tags by importance.
4. **Interpret the Results**: The output will be a list of the most "informative" tags. You can use this to clean up your tagging system, identify missing connections, or build a note recommendation system.
In short, yes, it's possible, but it's a **data engineering project** more than a simple plugin installation. It's best suited for power users who are comfortable with Python and want to apply machine learning techniques to understand and optimize their personal knowledge management system.
That is the state of my Vault's md files:
---
title: deepseek-chat-2025-06-21T21-04-18-548Z
date: 2025-06-22
tags:
- knowledge-management
- obsidian
- nuclear-weapons
concepts:
- Nuclear_Non-Proliferation_Treaty_NPT
- Treaty_on_the_Prohibition_of_Nuclear_Weapons_TPNW
- International_Court_of_Justice_ICJ_Advisory_Opinion
- Distinction_principle
- Proportionality_principle
- Negative_Security_Assurances
- Deterrence_Doctrine
meta_tags:
- [[international-law]]
- [[nuclear-weapons-policy]]
- [[treaty-analysis]]
meta_keyword: [[Nuclear]]
---
## Summary
This content analyzes the legal framework surrounding the use of nuclear weapons by the United States under international treaties and domestic law. The conclusion is that while no treaty explicitly authorizes their use, constraints from international humanitarian law (IHL) and policy commitments create a narrow potential window for use in extreme self-defense scenarios.
---
## Content
Can you summarize for the me the use of nuclear weapons in terms of legally allowed scenarios based on international treaties signed by the USA?
Yes—your vault structure is actually **very well suited** for this kind of data analytics. The YAML frontmatter gives you clean, machine-readable metadata. The key is to turn each note into a **row** and each tag/concept/meta_tag into a **feature**.
### 🔄 How Your Fields Map to Features
| Obsidian Field | Role in Data Analytics |
|:---|:---|
| `title` | Note ID / row label |
| `date` | Temporal feature |
| `tags` | Multi-label categorical features (broad topics) |
| `concepts` | Multi-label features (finer entities/ideas) |
| `meta_tags` | Higher-level category features (wikilinks) |
| `meta_keyword` | Primary label or feature |
| `content` | Text corpus for TF-IDF, embeddings, etc. |
Your example note would become a row like:
| note | tag:knowledge-management | tag:obsidian | tag:nuclear-weapons | concept:NPT | concept:TPNW | meta_tag:international-law | meta_keyword:Nuclear |
|---|---|---|---|---|---|---|---|
| deepseek-chat... | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Once you have hundreds of notes like this, you can run feature selection algorithms from that table.
### 🛠️ Practical Pipeline
1. **Parse all `.md` files** and extract YAML frontmatter.
2. **Normalise** tags/concepts: lower-case, replace spaces/underscores with hyphens, merge synonyms (`nuclear-weapons` vs `nuclear-weapons-policy` vs `Nuclear`).
3. **Build a note × feature matrix** using one-hot encoding for `tags`, `concepts`, and `meta_tags`.
4. **Define a target** if you want supervised feature selection. For example:
- Predict `meta_keyword` from tags/concepts.
- Predict note type (meeting, idea, reference).
- Predict whether a note will be linked to later.
- Or cluster notes and use cluster labels as pseudo-targets.
5. **Apply feature selection**:
- **Filter methods**: mutual information, chi-square, `mRMR` — fast, good for ranking tags.
- **Wrapper methods**: GA, PSO, GWO, DBA — slower, but can find optimal subsets.
- **Hybrid**: `mRMR+DBA` or your `HMF-W` — first filter redundant features, then optimise with a swarm algorithm.
### 🐍 Minimal Python Example
```python
import frontmatter, pathlib, pandas as pd
from sklearn.preprocessing import MultiLabelBinarizer
rows = []
for p in pathlib.Path("vault").rglob("*.md"):
post = frontmatter.load(p)
meta = post.metadata
rows.append({
"note": p.stem,
"tags": meta.get("tags", []),
"concepts": meta.get("concepts", []),
"meta_tags": [t.strip("[]") for t in meta.get("meta_tags", [])],
"meta_keyword": meta.get("meta_keyword", ""),
"content": post.content,
})
df = pd.DataFrame(rows)
# One-hot encode tags
mlb = MultiLabelBinarizer()
X_tags = pd.DataFrame(
mlb.fit_transform(df["tags"]),
columns=[f"tag:{t}" for t in mlb.classes_],
index=df.index
)
# Repeat for concepts and meta_tags, then concatenate
X = pd.concat([X_tags, X_concepts, X_meta_tags], axis=1)
# Example target: meta_keyword
y = df["meta_keyword"].str.strip("[]")
# Feature selection
from sklearn.feature_selection import mutual_info_classif
mi = mutual_info_classif(X, y)
top_features = X.columns[mi.argsort()[-20:]]
print(top_features)
```
### ⚠️ Things to Watch Out For
- **Redundancy is high**: `nuclear-weapons`, `nuclear-weapons-policy`, and
`meta_keyword: Nuclear` are all related. Algorithms like **mRMR** are designed to
penalise redundancy, so they’ll help you find the most informative subset.
- **Circularity**: If you use `meta_keyword` as the target, tags like `nuclear-weapons` will
obviously be predictive. That’s fine for validating your tagging, but for discovery you
need a target that isn’t manually assigned.
- **Sparsity**: Most notes use only a small fraction of your total tags. Filter out tags that
appear in fewer than 3–5 notes before running heavier algorithms.
- **Scale**: If you have fewer than ~200 notes, simple co-occurrence counts or Obsidian’s
graph view will give you faster insights. These algorithms shine when you have
thousands of notes and many overlapping tags.
- **Graph features**: Don’t ignore wikilinks. You can extract the link graph with
`networkx` and add centrality, PageRank, or community labels as features.
### ✅ Bottom Line
Your YAML structure is ideal for feature selection. Start with **mRMR** or **mutual information** to rank which tags/concepts are most meaningful. If you have a clear prediction target and enough notes, you can then try the wrapper algorithms (GA, PSO, GWO, DBA, etc.) to find the best subset. The result will help you clean up your tagging system, discover hidden relationships, and even auto-tag future notes.
Could they be used to optimize LLMs vector databases?
Yes, the same core logic behind feature selection—identifying and removing redundant or noisy information—can be applied to optimize LLM vector databases. However, the "features" you're selecting are not tags or keywords, but rather the **dimensions of the embedding vectors** themselves.
The goal is to find a smaller, more efficient representation that preserves semantic meaning for retrieval, a process broadly known as **embedding dimension reduction** or **dimension pruning**.
### 🔬 The Research Landscape: Pruning the Right Dimensions
Research shows that not all dimensions in a high-dimensional embedding contribute equally to retrieval performance; many are noisy or redundant.
* **DIME (Dimension Importance Estimation)**: This framework associates an
importance score with each embedding dimension and projects vectors onto only the
most informative ones. The approach can yield significant performance improvements,
with some estimators achieving gains as high as **+49.6% in Average Precision (AP)**
and **+55.9% in nDCG@10**, all without any model retraining. A statistically grounded
version can reduce embedding size by an average of **50%** at inference time.
* **PCA-RAG**: This method applies Principal Component Analysis (PCA) to compress
embeddings. A study demonstrated that reducing vector size from **3,072 to 110
dimensions** (a ~28x reduction) drastically lowered the memory footprint and
computation per lookup, achieving a **~60x speedup** in retrieval operations and a
**~28.6x reduction** in index size.
### 🛠️ Beyond Vectors: Optimizing the Index Itself
Feature selection principles also extend to optimizing the **vector index structures** (like HNSW or IVF) that power search.
* **Multi-Objective Evolutionary Algorithms**: A patent describes a system using an
improved **non-dominated sorting genetic algorithm (NSGA-II)** to automatically tune
vector database index hyperparameters. It optimizes for conflicting objectives like
**recall rate, query latency, memory occupation, and hardware power consumption**,
searching for a Pareto-optimal parameter set that balances performance and resource
use.
### 💡 Practical Takeaways for Your Use Case
If you're working with a vector database (e.g., for a RAG pipeline), here's how you can apply these ideas:
1. **Start with DIME-style Pruning**: Before considering complex index tuning, analyze
your embeddings to identify which dimensions are most important for your specific
query workload. This can often be done at retrieval time without re-indexing.
2. **Consider PCA for Compression**: If your embeddings are very high-dimensional (e.g.,
1,536+), PCA is a simple and effective way to reduce storage and latency. The trade-off is
a potential minor loss in accuracy, but the research suggests it can be quite minimal for
many tasks.
3. **Automate Index Tuning**: For production systems, manually tuning index
parameters (like HNSW's `M` or `efConstruction`) is difficult. Multi-objective
algorithms can automatically find a configuration that meets your latency, recall, and
memory requirements.
In short, yes—the same optimization mindset that applies to your Obsidian tags can be scaled up to the numerical dimensions of LLM embeddings and the parameters of vector database indexes.
If you'd like to go deeper into any of these specific techniques, feel free to ask.
Please read this: https://cyberdeck-ai-knowledge-system.org/posts/llm_language/ and tell me how the algorithms could fit into the LLM Language set up.
The article describes a radical redesign of how LLMs process information: instead of relying solely on next-word prediction, it proposes a **compressed, layered internal language** for the model, with a **translator shell** that converts human intent into this dense code and back again. The feature selection algorithms from your table are directly applicable to **optimizing the components of this architecture**—the vocabulary, the layer structure, the embedding space, and the translator itself. Here is how each algorithmic family maps onto the Cyberdeck LLM Language.
### 🗜️ Optimizing the Compressed Language Itself
The core idea of the Cyberdeck language is **semantic compression**—using "words that save" that carry multiple meanings, resolved by a "stone-set topic" and "layered operators". Selecting the *best* set of such tokens is a feature selection problem.
* **mRMR (Minimum Redundancy Maximum Relevance)** is the natural fit for building
the vocabulary of "words that save." You want tokens that are **highly relevant** to the
domain (e.g., "turbulence modeling") but **minimally redundant** with each other.
mRMR would rank candidate words by how much unique semantic payload they carry
under a given topic.
* **Mutual Information / Chi-Square** can be used as fast **filter methods** to mine for
high-polysemy terms, as the article suggests: "mining for terms with high polysemy
across relevant domains". These algorithms identify which words or concepts carry the
most information about the target reasoning task.
### 🧱 Optimizing the Layered Operator Structure
The article describes **layered operators**—nested redefinitions that transform meaning at each level (Layer 0 sets the topic, Layer 1 maps entities to internal states, etc.). Deciding **how many layers** to use, **which operators** to apply at each layer, and **how to sequence them** is a combinatorial optimization problem.
* **Genetic Algorithms (GA)** are well-suited for this. The article's own description of
parameter tuning ("the sequence of words… is the program that runs") mirrors a GA's
chromosome-and-mutation approach. A GA could evolve different **operator stacks**
(e.g., `[topic: psyche] → [entity: internal state] → [location: psychological marker]`) and
select the configuration that produces the most coherent compressed reasoning trace.
* **PSO (Particle Swarm Optimization)** could similarly search the space of operator
parameters (e.g., how strongly each layer redefines the previous one), treating each
"particle" as a candidate configuration of the layer stack.
### 🧠 Optimizing the Embedding / Vector Space
The article notes that implementation might use a **"differentiable 'topic vector' prepended to the key-value cache"** or **"learnable continuous prompts that act as operators"**. This is exactly where **embedding dimension reduction** (as we discussed earlier) applies.
* **DIME-style dimension pruning** could identify which dimensions of the topic vector
or operator embeddings are actually contributing to the compressed reasoning,
allowing the model to operate in a **smaller, more efficient latent space**.
* **PCA** could compress the operator representations, making the "layered operators"
cheaper to store and faster to apply during inference.
### 🔄 Optimizing the Translator Shell
The **translator shell** must interpret the compressed trace and expand it into a faithful narrative. This is a **supervised learning problem**: given a compressed trace and the original topic, generate the full explanatory chain.
* **Wrapper methods (GA, PSO, GWO, DBA)** can be used to select the **best subset of
features** (e.g., which parts of the compressed trace are most predictive of a correct
expansion) to train the translator. The "double meaning selection" process—testing
whether topic-anchored disambiguation yields correct multi-path interpretations—is
essentially a **wrapper evaluation loop**, where each candidate vocabulary subset is
tested by how well the translator can reconstruct the intended meaning.
* **IGMPMMIAPSO**, with its crossover and mutation operators, could be used to **evolve
the translator's training prompts**, optimizing for both fidelity (correct expansion) and
compactness (minimal compressed input).
### ⚠️ A Critical Caveat: The Alignment Risk
The article explicitly warns that a compressed internal language **amplifies the alignment problem**: "If the AI's reasoning becomes opaque even to its own translators, verification becomes impossible". This is where feature selection must be paired with **interpretability constraints**. You cannot simply optimize for the most compressed, most predictive token set; you must also ensure that the selected features remain **logically isomorphic** to the expanded reasoning trace. A multi-objective version of these algorithms (like the NSGA-II approach mentioned in the vector database context) could optimize for both **compression ratio** and **translator fidelity**.
### 💡 Summary Table: Algorithm → Cyberdeck Component
| Cyberdeck Component | Applicable Feature Selection Algorithms | Optimization Goal |
|:---|:---|:---|
| **"Words that save" vocabulary** | mRMR, Mutual Information, Chi-Square | Select high-polysemy, low-redundancy tokens |
| **Layered operator structure** | GA, PSO | Evolve optimal layer depth, operator sequence, and redefinition strength |
| **Topic vector / operator embeddings** | DIME, PCA | Prune embedding dimensions for efficiency without losing semantic resolution |
| **Translator shell training** | GA, PSO, GWO, DBA, IGMPMMIAPSO | Select the subset of compressed features that best predict faithful expansions |
| **Alignment / verification** | Multi-objective GA (NSGA-II style) | Balance compression ratio against translator fidelity |
The Cyberdeck architecture is essentially a **feature engineering pipeline for the context window**. The algorithms from your table provide the mathematical machinery to decide *which* words, *which* layers, and *which* vector dimensions deserve to occupy that scarce budget.
Can you write a first python script that turns a data set into a vector database .json file in the LLM language with the appropriate algorithms incorporated and having the required links to the actual core texts for an Ollama system and also for a speculative coder Cortex system: -m ~/qwen-coder-gguf/qwen2.5-coder-32b-instruct-q5_k_m-00001-of-00003.gguf \
-md ~/qwen-coder-gguf/qwen2.5-coder-1.5b-instruct-q5_k_m.gguf \
So, to explain. The system has ollama models with vector databases to go through data for reasoning. One RAM rich node uses a speculative coder setup to run a very strong LLM. That one would run over all seperate vector databases stored in json files to create a central brain in order to find connections and create better knowledge. The system is GPU free and low cost based on a Beowulf cluster with a LLM Loadbalancer on top coordinating several Ollama instances over the nodes.
Below is a **first Python script** that turns a dataset into a vector database `.json` file designed for your Beowulf cluster / Ollama setup. It incorporates the feature selection algorithms from your table, produces entries with **links to core texts**, and is structured so that both the regular Ollama nodes and the speculative Cortex node can consume it.
---
### 🧠 How the pieces fit your architecture
| Component | Role in your system |
|:---|:---|
| **Ollama embedding API** | Generates vectors via `nomic-embed-text` or `mxbai-embed-large` on any cluster node |
| **Feature selection (mRMR)** | Selects the most informative text chunks before embedding, reducing vector DB size and improving retrieval precision |
| **Speculative decoding (Cortex)** | The 32B coder model uses the 1.5B draft model to accelerate generation across the selected chunks |
| **Load balancer** | Distributes embedding and retrieval requests across Ollama instances |
| **JSON vector DB** | Each entry stores: chunk text, embedding vector, source link, selected keywords, and feature metadata |
---
### 📜 Script: `build_vector_db.py`
```python
#!/usr/bin/env python3
"""
build_vector_db.py
Turns a dataset into a vector database JSON file for a distributed
Ollama / Beowulf cluster with speculative decoding support.
Designed for:
- Ollama instances behind a load balancer
- Speculative coder Cortex node (qwen2.5-coder-32b + 1.5b draft)
- Feature selection via mRMR to reduce vector DB size
- Links back to core source texts
"""
import json
import hashlib
import re
from pathlib import Path
from typing import List, Dict, Any, Optional
import numpy as np
import ollama
from sklearn.feature_selection import mutual_info_classif
# ============================================================
# CONFIGURATION
# ============================================================
# --- Ollama endpoints (load balancer or single node) ---
OLLAMA_HOST = "http://localhost:11434" # or your LB address
EMBED_MODEL = "nomic-embed-text" # or "mxbai-embed-large"
CHAT_MODEL = "qwen2.5-coder:32b" # speculative target
# --- Speculative Cortex paths (documentation only in this script) ---
# These are the models your Cortex node runs via llama.cpp:
CORTEX_TARGET_MODEL = "~/qwen-coder-gguf/qwen2.5-coder-32b-instruct-q5_k_m-00001-of-00003.gguf"
CORTEX_DRAFT_MODEL = "~/qwen-coder-gguf/qwen2.5-coder-1.5b-instruct-q5_k_m.gguf"
# --- Feature selection ---
TOP_K_FEATURES = 50 # number of chunks to keep after mRMR
MIN_CHUNK_LEN = 40 # ignore tiny chunks
MAX_CHUNK_LEN = 1200 # truncate very long chunks
# --- Source link template ---
SOURCE_LINK_TEMPLATE = "https://your-knowledge-base.org/source/{doc_id}#chunk-{chunk_id}"
# ============================================================
# 1. DATA LOADING
# ============================================================
def load_dataset(path: str) -> List[Dict[str, Any]]:
"""
Load a JSON or JSONL dataset. Each item must have at least:
- "text": the raw content
- "doc_id": stable identifier for the source document
Optional:
- "title", "tags", "url"
"""
path = Path(path)
if path.suffix == ".jsonl":
with open(path) as f:
return [json.loads(line) for line in f if line.strip()]
elif path.suffix == ".json":
with open(path) as f:
data = json.load(f)
return data if isinstance(data, list) else [data]
else:
raise ValueError(f"Unsupported format: {path.suffix}")
# ============================================================
# 2. CHUNKING
# ============================================================
def chunk_text(text: str, doc_id: str) -> List[Dict[str, Any]]:
"""
Split a document into overlapping chunks.
Each chunk carries its source doc_id and a chunk index
so we can generate a deep link back to the core text.
"""
# Simple sentence-aware split; replace with your tokenizer if needed
sentences = re.split(r'(?<=[.!?])\s+', text)
chunks = []
current = []
current_len = 0
chunk_id = 0
for sent in sentences:
if current_len + len(sent) > MAX_CHUNK_LEN and current:
chunks.append({
"doc_id": doc_id,
"chunk_id": chunk_id,
"text": " ".join(current).strip(),
})
chunk_id += 1
# overlap: keep last sentence for context
current = [current[-1]] if current else []
current_len = len(current[-1]) if current else 0
current.append(sent)
current_len += len(sent)
if current:
chunks.append({
"doc_id": doc_id,
"chunk_id": chunk_id,
"text": " ".join(current).strip(),
})
return [c for c in chunks if len(c["text"]) >= MIN_CHUNK_LEN]
# ============================================================
# 3. FEATURE SELECTION (mRMR-style)
# ============================================================
def extract_keyword_features(chunks: List[Dict[str, Any]]) -> np.ndarray:
"""
Build a simple binary feature matrix from keyword presence.
In a full mRMR pipeline you would use TF-IDF or embeddings;
this keeps the script runnable without extra NLP deps.
"""
# Collect all tokens (simple whitespace + lowercasing)
vocab = set()
tokenized = []
for c in chunks:
tokens = set(re.findall(r"[a-z][a-z0-9_-]{2,}", c["text"].lower()))
tokenized.append(tokens)
vocab.update(tokens)
vocab = sorted(vocab)
vocab_index = {w: i for i, w in enumerate(vocab)}
X = np.zeros((len(chunks), len(vocab)), dtype=np.uint8)
for i, tokens in enumerate(tokenized):
for t in tokens:
X[i, vocab_index[t]] = 1
return X, vocab
def select_chunks_mrmr(
chunks: List[Dict[str, Any]],
top_k: int = TOP_K_FEATURES
) -> List[Dict[str, Any]]:
"""
Rank chunks by their mutual information with a pseudo-target.
Here the pseudo-target is "chunk contains a rare keyword"
(a proxy for relevance). Replace with your real label if available.
"""
X, vocab = extract_keyword_features(chunks)
# Pseudo-target: 1 if chunk contains any of the rarest 10% keywords
freq = X.sum(axis=0)
threshold = np.percentile(freq[freq > 0], 10) if (freq > 0).any() else 1
rare_cols = np.where((freq > 0) & (freq <= threshold))[0]
y = (X[:, rare_cols].sum(axis=1) > 0).astype(int)
if y.sum() == 0 or y.sum() == len(y):
# Fallback: keep all chunks
return chunks
mi = mutual_info_classif(X, y, discrete_features=True)
scores = mi.sum(axis=0) if mi.ndim > 1 else mi
# Rank chunks by aggregate MI of their tokens
chunk_scores = X @ scores
ranked = sorted(
zip(chunks, chunk_scores),
key=lambda x: x[1],
reverse=True
)
return [c for c, _ in ranked[:top_k]]
# ============================================================
# 4. OLLAMA EMBEDDING
# ============================================================
def embed_texts(texts: List[str], model: str = EMBED_MODEL) -> List[List[float]]:
"""
Batch-embed texts via Ollama's /api/embed endpoint.
The LB will distribute these across cluster nodes.
"""
# Ollama Python client handles batching when input is a list
response = ollama.embed(model=model, input=texts)
return response["embeddings"]
# ============================================================
# 5. BUILD VECTOR DB ENTRY
# ============================================================
def build_entry(
chunk: Dict[str, Any],
vector: List[float],
keywords: Optional[List[str]] = None
) -> Dict[str, Any]:
"""
One vector DB record. Contains:
- id: stable hash
- text: the chunk content
- embedding: the vector
- source: deep link back to core text
- metadata: doc_id, chunk_id, keywords
- cortex_hint: speculative decoding routing hint
"""
chunk_id = f"{chunk['doc_id']}::{chunk['chunk_id']}"
uid = hashlib.sha256(chunk_id.encode()).hexdigest()[:16]
source_url = SOURCE_LINK_TEMPLATE.format(
doc_id=chunk["doc_id"],
chunk_id=chunk["chunk_id"]
)
return {
"id": uid,
"text": chunk["text"],
"embedding": vector,
"source": {
"url": source_url,
"doc_id": chunk["doc_id"],
"chunk_id": chunk["chunk_id"],
},
"metadata": {
"keywords": keywords or [],
"char_len": len(chunk["text"]),
},
# Hint for the Cortex node: route long reasoning to 32B,
# quick draft expansion to 1.5B.
"cortex_hint": {
"speculative_target": CORTEX_TARGET_MODEL,
"speculative_draft": CORTEX_DRAFT_MODEL,
"recommended_mode": "speculative" if len(chunk["text"]) > 200 else "draft",
},
}
# ============================================================
# 6. MAIN PIPELINE
# ============================================================
def main(
dataset_path: str,
output_path: str = "vector_db.json",
top_k: int = TOP_K_FEATURES,
):
# --- Load ---
print(f"[1/5] Loading dataset: {dataset_path}")
docs = load_dataset(dataset_path)
print(f" {len(docs)} documents loaded")
# --- Chunk ---
print("[2/5] Chunking documents...")
all_chunks = []
for doc in docs:
doc_id = doc.get("doc_id", doc.get("id", "unknown"))
all_chunks.extend(chunk_text(doc["text"], doc_id))
print(f" {len(all_chunks)} chunks created")
# --- Feature selection ---
print(f"[3/5] Selecting top {top_k} chunks via mRMR...")
selected = select_chunks_mrmr(all_chunks, top_k=top_k)
print(f" {len(selected)} chunks retained")
# --- Embed ---
print(f"[4/5] Embedding via Ollama ({EMBED_MODEL})...")
texts = [c["text"] for c in selected]
vectors = embed_texts(texts)
# --- Build JSON ---
print("[5/5] Building vector DB JSON...")
entries = []
for chunk, vec in zip(selected, vectors):
# simple keyword extraction for metadata
kws = list(set(re.findall(
r"[a-z][a-z0-9_-]{3,}", chunk["text"].lower()
)))[:10]
entries.append(build_entry(chunk, vec, kws))
db = {
"version": "1.0",
"embedding_model": EMBED_MODEL,
"dimension": len(vectors[0]) if vectors else 0,
"created_by": "build_vector_db.py",
"cluster": {
"ollama_host": OLLAMA_HOST,
"cortex_target": CORTEX_TARGET_MODEL,
"cortex_draft": CORTEX_DRAFT_MODEL,
},
"entries": entries,
}
with open(output_path, "w") as f:
json.dump(db, f, indent=2)
print(f"Done. Wrote {len(entries)} entries to {output_path}")
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser(
description="Build vector DB JSON for distributed Ollama cluster"
)
parser.add_argument("dataset", help="Path to .json or .jsonl dataset")
parser.add_argument(
"-o", "--output", default="vector_db.json",
help="Output JSON path"
)
parser.add_argument(
"-k", "--top-k", type=int, default=TOP_K_FEATURES,
help="Number of chunks to retain after feature selection"
)
args = parser.parse_args()
main(args.dataset, args.output, args.top_k)
```
---
### 🔧 How to adapt it to your exact setup
**1. Point to your Ollama load balancer**
Change `OLLAMA_HOST` to your LB address. The `ollama` Python client uses the default host, so you may need to set the environment variable:
```bash
export OLLAMA_HOST=http://your-load-balancer:11434
```
**2. Choose your embedding model**
Install the embedding model on every node that will serve embeddings:
```bash
ollama pull nomic-embed-text
# or
ollama pull mxbai-embed-large
```
Then update `EMBED_MODEL` accordingly.
**3. Cortex speculative routing**
The `cortex_hint` field is a **routing hint** for your Cortex node. The actual speculative decoding is configured at the llama.cpp level:
```bash
llama-server \
-m ~/qwen-coder-gguf/qwen2.5-coder-32b-instruct-q5_k_m-00001-of-00003.gguf \
--model-draft ~/qwen-coder-gguf/qwen2.5-coder-1.5b-instruct-q5_k_m.gguf \
--draft-max 16
```
Your load balancer or a thin router can read the `cortex_hint` field and forward short chunks to the draft model and long reasoning tasks to the 32B target.
**4. Central brain consolidation**
The Cortex node reads **all** per-node `vector_db.json` files, merges them, and uses the 32B model to find cross-database connections. A simple merge script:
```python
import json, glob
all_entries = []
for f in glob.glob("node_*/vector_db.json"):
with open(f) as fh:
all_entries.extend(json.load(fh)["entries"])
with open("central_brain.json", "w") as f:
json.dump({"entries": all_entries}, f)
```
**5. Scale the feature selection**
The mRMR implementation here is simplified. For production, use `pyrobustfs` which implements ensemble mRMR with bootstrapping for more stable feature selection:
```bash
pip install git+https://github.com/yourusername/pyrobustfs.git
```
Then replace `select_chunks_mrmr` with a `RobustMRMRSelector` pipeline.
---
### ⚠️ Key design decisions in this first script
| Decision | Why |
|:---|:---|
| **Chunk-level, not document-level** | Vector search works best on focused passages; deep links let the LLM retrieve the full context when needed |
| **mRMR as a pre-filter** | Embedding every chunk is expensive; selecting the top-K most informative chunks reduces vector DB size by 10–50x |
| **`source.url` in every entry** | The "links to core texts" requirement; the LLM can cite exact passages |
| **`cortex_hint` metadata** | Lets your load balancer route to speculative decoding when appropriate |
| **Single JSON file per node** | Easy to distribute across the Beowulf cluster and merge at the central Cortex node |
This gives you a **runnable first version**. From here you can add: real mRMR with `pyrobustfs`, TF-IDF instead of binary keywords, FAISS or Qdrant as the actual index backend, and a retrieval API that queries the central brain and forwards to the Cortex node.
#cyberpunkcoltoure #opensource
Still untested and far from being embedded. Early Alpha Phase. Apache 2.0 Free to Pay More. Support Open Source Research!