I’m testing out msty as a way to compare some different AI tools. The ability to create “knowledge stacks” (and, moreover, use different models for embedding that knowledge for RAG) is really amazing.

Here I took some VW training materials and couple of YouTube links and created a “Volkswagen Vanagon” knowledge stack using 2 different embedding models: Snowflake Arctic and OpenAI’s Text Embedding. The tradeoff here is that by using Snowflake, you’re keeping the embedding local as you’re running that model locally, the openai text embedding requires you send the content via API to OpenAI for embedding.

Next, I used GPT-4o and Claude3 Opus to interrogate the training materials. The ability to see this output side by side is helpful.

SO here we see from left to right:

GPT w/ Snowflake, GPT w/ OpenAI Embedding, Claude with Snowflake and Claude with OpenAI Embedding

As I know a bit about Vanagon idle issues, I’d say that if I had to rank these responses, Claude with OpenAI embedding barely beats out GPT’s Open AI embedding simply because it communicates the same info as GPT but slightly more user-friendly.

Next would be GPT using the Snowflake embedding which gave accurate though not thorough or complete information. Finally, Claude with Snowflake was a total fail.

That said, I could probably tweak the prompting a bit and get a slightly different set of responses.

I then tried the same query using a local LLM, TinyDolphin, which msty installs as a default option and had it query both the Snowflake embedding and the open AI embedding.

The TinyDolphin + Snowflake, importantly, is an entire local process. All data, processing and embedding happens locally. Here is the response:

To troubleshoot the idle issue on your 1986 Vanagon, follow these steps:

Inspect Vacuum Lines: Check for any leaks or disconnections in the vacuum lines and hoses.
Clean the Idle Control Valve: Remove and clean the idle control valve to ensure it isn't clogged.
Check the Throttle Body: Inspect and clean the throttle body and throttle position sensor.
Test the Idle Control Unit: Ensure that the idle control unit (ICU) is functioning correctly.
Check Fuel System: Examine the fuel filter, fuel injectors, and fuel pump for blockages or malfunctions.
Evaluate Electrical Components: Evaluate the condition of the spark plugs, ignition wires, distributor cap, and rotor for damage or wear.
Remember to allow time for the readings to stabilize when checking or making adjustments.

Not bad, but not as good as the response that TinyDolphin gives when using the OpenAI embedding, which, here remember that the source material for the knowledge stack is embedded using the API so it’s not an entirely local process.

To troubleshoot the idle issue on your 1986 Vanagon, follow these steps based on the provided information:

1. Check Engine Operating Temperature:
   • Ensure the engine is at operating temperature with an oil temperature of at least 80°C (176°F).
2. Inspect Throttle Position Switch:
   • Ensure that the throttle position switch is connected and functioning properly.
   • Move the throttle linkage to full throttle position and listen for a clicking noise.
   • Return the throttle linkage to the idle position and listen for a clicking noise.
   • Adjust if necessary to ensure the switch clicks at both full throttle and idle positions.
3. Idle Stabilization Valve (ISV) Check:
   • Turn off all electrical consumers and ensure the ignition coil is switched on at the connector for the idle stabilization control unit.
4. If Readings are Off:
   • Refer to “Idle Speed/CO Content – Checking and Adjusting” as indicated on page 37 of your manual.
5. If Readings are Fixed (Not Fluctuating):
   • Check for a signal from terminal 1 of the ignition coil at the connector for the idle stabilization control unit.
   • Use the US 1115 LED tester between terminals 5 and 17 of the connector to ensure proper signal.

By following these steps, you should be able to identify and potentially correct the idle issue on your 1986 Vanagon.

So the weak link here really seems to be the local embedding process. This embedding process is still opaque to me and there are many parameters that may improve the quality so that is going to be my next step here.