Synthetic Beings

A Synthetic Being ('synth') is comprised of many components - just like a Human Being. Let's consider it analogously.

Just like in humans, autonomic systems keep the being alive. These are the behind the scenes tasks fundamental to running - like operations performed by the operating system itself. We'll discuss those later - for now, lets focus on the 'conscious' part. But let us be clear here - it is NOT really conscious, it is performing calculations.

MIND

The foremost part of a synth is its mind. This thought experiment uses Hermes Agent, but you can use any other Agentic AI Framework.

-THOUGHTS

Just like in humans, thoughts drive the entire process. Thoughts are output from a LLM Server - Ollama, vLLM, etc running a LLM Model. We can download and use different models easily. Changing out the model swaps out the 'thinking' part of its brain, not its entire 'self'. It just changes how smart it is, or how good it can perform various jobs.

-SHORT TERM MEMORY

Just like in humans, the synth must be able to make and recall what jist happened - without short term memory we just exist in the moment. The most recent conversation exchanges are recorded in the Context Window.

-LONG TERM MEMORY

Having long term memories is what makes us effective over long periods of time. Memories types include Episodic (historical logs), Semantic (verified facts), Procedural (how to do things) and Knowledge (reference materials). We store them in a vector database, so we can semantically search. Integrating a vector database to inform the thinking is called Retrival Augmented Generation (RAG).

CAPABILITES

Now that the synth's mind is structured, it needs to be able to do things! To perform actions, 'tools' are used. The synth 'thinks' (with the LLM) that it is going to call a tool, and then (if the model is actually capable of calling tools), organizes the tool call in JSON format as part of its responses. There are a few basic tools that are essential for anything else, like read/write of files - those are generally going to be natively a part of whatever framework you are using.

Beyond the Native Tools that are built-in, we extend the synth's capabilities using MCPs. Here, we are using Engram MCP to bridge the gap between the thinking and memory parts of its mind - letting the synth make/recall memories.

Additional MCPs can be added, like in the Personal Assistant and in the Project Management layers we will add later. This will let the synth use email and calendars, plan and decompose tasks, and be generally useful.

BODY

The synth is a being even without a body, just as amputees are still human. But if we embody it - that is, run the synth's mind inside a robot body, we can achieve a complete synth. Interfacing is simple if we expect ROS2 (Robot Operating System) as the standard. Various manufacturers will have their own versions, but ROS2 is the clear winner here.

PROCEDURE (WIP, not a guide):

Install Ollama (local llm ai)

Download local models (ollama pull):

- qwen3.5:35B-a3b is good for general utility agentic orchestration

// We require a local solution to generate and edit images - we are going to use ComfyUI. ComfyUI will run our image models, and provide an easy to use UI to graphically ties it all into useable workflows. The basic pipeline is Diffusion Model > LoRAs > Text Encoders > VAE Decoder.

// The diffusion model is the core of the pipeline - it will 'generate' the image. The LoRAs help that diffusion model maintain order (like keeping the same subject or background). Text Encoders allow positive and negative prompts to be considered. A node called a KSampler then assembles all that information together with the model to define what the image is going to be using latent space. Finally, we will use a VAE Decoder to turn that latent space image into a real-deal pixel-space image.

Install ComfyUI (local image ai)

We are going to create a custom workflow to use Flux.2-dev

Download local models:

https://huggingface.co/Comfy-Org/flux2-dev/

- flux2_dev_fp8mixed.safetensors | Diffusion Core (Processing/Editing) | Base architecture that modifies/reconstructs image latents based on analysis input from the encoder above. Handles edits/generation with understanding of what's in source images!

- mistral_3_small_flux2_fp8.safetensors | Image Encoder (Input Understanding) | Multimodal vision-language encoder - interprets uploaded images, performs OCR/extraction on text content visible in screenshots, outputs embeddings/clues for downstream processing

- flux2-vae.safetensors | VAE Decoder (Output Reconstruction) | Re-encodes processed latent representations back to visible pixels - specifically optimized for end-to-end Flux workflows where the encoder's features need proper decoding that matches its architectural assumptions

- Flux2TurboComfyv2.safetensors

- Flux_2-Turbo-LoRA_comfyui.safetensors

- qwen_image_vae.safetensors | VAE Decoder (Output Reconstruction) | Re-encodes processed latent representations back to visible pixels - specifically optimized for end-to-end Qwen VL workflows where the encoder's features need proper decoding that matches its architectural assumptions

- Flux2-Klein-9B-consistency-V2.safetensors | LoRA

- V3_flux_klein.safetensors | LoRA | InstaPic-Ultrareal

- Flux Klein - NSFW v2.safetensors | LoRA

// install the Mind

Install Hermes (agentic harness)

hermes model

- Point it to Custom Endpoint, and point it to Ollama

hermes setup

-In Hermes, set up each skill

// install the Long Term Memory

Install Engram (memory management)

// Automonic sysyems

Install PM2

Install codebase-memory MCP?

watch-ollama OR dofek (system monitor)

Open WebUI (optional frontend)