Graph-Powered Knowledge Base

Into the Matrix

Embeddings are a marvel. They reveal how matrices are powerful since you can imitate the activation pattern of the brain's neurons by stacking multiple layers of matrices that represent meaning.

Brilliant as this is, it has significant limits

• Converting meaning through numerical representation can only be as detailed as the possible dimensions.
• The underlying logic behind representing an object in a certain dimension is, in essence, reductive and would require near-endless dimensions to represent. Think of how a cat can be an animal, a pet, a living being, a symbol of Egyptian heritage, an emoji, a companion, and so much more. The final cause can give us a stable meaning of an object, but a subject's attribution of meaning to an object exposes us to a complex problem.
• Embeddings can indicate how similar two or more objects are based on the geometric relationships among the points. However, they cannot indicate how they are related. We cannot effectively discover the intersections between knowledge. A lot is left to an LLM to infer through its feed-forward layer - since this is where it "stores" its facts about the real world.

I will focus on the last issue. The second one is a fascinating one that I will explore in another experiment.

Objective

So, first things first, we need to prove that embeddings cannot identify the relationships between concepts unless they use the inference layer. This is important since Garbage In, Garbage Out is a universal principle that governs the quality of computation. No inference layer can have the entirety of knowledge, so a gap will always persist. Think about it as having a library with very few books on botany. You may have the right idea of what you want to plant, but no book will supply the right instructions.

1. Let us push embeddings to the limit by introducing parent-child relationships between chunks.
2. We will then evaluate what causal relationships we can extract from a query without relying on the inference layer (I do not know how we will do this, but there is always a way, since I have Will with me)

I will see you on the next log with the results of this objective.