Editing AMD Geode/Troubleshooting

[[Category:Projects]]
This page contains my troubleshooting notes for getting the video working with the Geode.

== Problem description ==
The eBay page listed the issue as "I was able to boot Windows 98SE from a Compact Flash card and everything worked fine. It currently refuses to output a video signal that any of my monitors can recognise" which sounds like it could be anything.

It's a little difficult to tell what's wrong with an x86 computer without video output, so I just checked what I could.

== Power supply ==
Connecting the board's 5V rail to my [[Gophert NPS-1601]] power supply caused it to draw up to 1A of current then beep.

There were no short circuits or smoke. I'm assuming the beeping is from the board POSTing.

Looking at the voltage drop with an oscilloscope, there was jitter from current consumption but no huge drops that would cause a brownout.

== Video output ==
Video output caused both my monitors to display a 'Input not supported' message twice during boot:

* The first was quick just after boot and lasted a second or so, followed by a second of nothing
* The second message lasts until you reset the board or pull power 

This is different to a 'No input' message, so some signal must be generated that the monitor can recognize but still give a failure for.

== RAM ==
In case some memory error was causing the machine to hang during video initialization I swapped the stick with a test one.

No difference at all, so I don't think anything's really wrong with the RAM.

Removing the RAM altogether caused long beeps, presumably by the BIOS.

The original RAM that came with the machine is a Transcend 512M DDR333 stick, and the test one is Kingston 256M DDR333 stick.

== Serial output ==
I connected both a makeshift and proper null modem cable to the serial port on the device. There was no output on it, but my oscilloscope showed the transmit line being held low.

This doesn't mean necessarily mean anything, most BIOSes on consumer hardware (old and modern) don't output on the serial port. This is horrible on many levels ranging from accessibility to troubleshooting or just configuring things if you don't have a monitor.

== Clock battery ==
I removed the clock battery and checked it with my multimeter. It was around 200mV, much lower than the 3V it should provide.

I figured this may have caused some kind of under-voltage data corruption, but booting without the battery doesn't do anything.

The battery is a Rayovac BR1632 (see [https://www.rayovac.com/wp-content/uploads/2021/08/R6099_OEMReferenceGuide.pdf Rayovac 6099 OEM Reference Guide] for more details) coin battery. This is a size smaller than the standard 2032 battery you'd see in a computer so I haven't replaced it yet.

Visual inspection doesn't show any battery leakage on the battery, and I can't really check under the battery holder without de-soldering it. I'm going to assume there wasn't any damage from a leakage event.

== Keyboard ==
I plugged in a standard PS/2 keyboard (a black Acer KB-2971 covered in grime). The system would allow toggling caps lock and num lock as well as make some beeps when pressing keys. 

This confirms that the system is active and not physically dead. 

== Ethernet ==
I connected the machine to my computer using an Ethernet cable to see if some kind of network boot was being attempted.

No packets arrived at my machine, so it's definitely not trying to boot over the network.

== BIOS ==
I removed the BIOS chip to see how this would affect things.

The system didn't respond to any keypresses and made no noise.

This confirms that the BIOS is responsible for all the behaviour seen so far.

== VGA signals ==
Probing the VGA signal showed the following:

* Vertical sync was a 60Hz square wave lasting for 40 microseconds
* Horizontal sync was a 75.02KHz square wave lasting for 1.45 microseconds
* Red green and blue were all flat signals around 200mV

This seems like the machine is outputting no graphics signal.

I didn't see any visible damage on the board's VGA section so this seems like a software issue.

== USB powering ==
I measured powering over USB using my [[Color TFT USB Tester]] and it confirmed earlier measurements. The current hangs around 1A and peaks around 1.5A.

== Measuring the DAC ==
The [https://www.amd.com/system/files/TechDocs/33234H_LX_databook.pdf AMD Geode LX Processors Data Book] explains that after video has been rendered it's output through an 8-bit DAC.

The following signals are used by the DAC:

* DVREF - A 1.235V voltage reference
* DRSET - A 1.21K current setting resistor
* DAVdd - Four 3.3V analog power connections
* DAVss - Four 0.0V analog ground connections
* RED, GREEN, BLUE - Analog DAC output
* DOTREF - 48MHz input clock

After looking at the [https://wiki.laptop.org/mediawiki/images/8/83/XO-1_Schematics.pdf OLPC XO-1 Schematics] and [https://manualzz.com/doc/29656964/amd-geode%E2%84%A2-lx-epic-rdk-reference AMD Geode LX EPIC RDK Schematics] which both use LX processors and inspecting the PCM-9375 board itself I figured out the following:

* The OLPC XO-1 uses a resistor voltage divider to generate DVREF
* The AMD EPIC RDK and PCM-9375 use a shunt to generate DVREF
* The OLPC XO-1 sends the DAC signals to the connector directly
* The AMD EPIC RDK and PCM-9375 put inductors and ESD protection in front of the signals
* The PCM-9375 has some capacitors in front of the signals
* The OLPC XO-1 and AMD EPIC RDK connect the DOTREF to the CS5536 chip
* I presume the PCM-9375 does that too

Most of the signals above are physically exposed balls under the Geode chip. Probing them shows:

* DVREF hangs around 1.22V
* DAVdd hangs around 3.3V
* DOTREF is a 48MHz sine wave (possibly scope artifacts)

Everything looks fine here.

== Booting FreeDOS ==
I ran QEMU with my CompactFlash card passed through as the hard drive.

Installing FreeDOS was fairly easy.

I added 'CTTY CON' to the autoexec and the command shell talked over serial.

Unfortunately all the DOS programs that use video graphics don't work.

== Booting Linux ==
I ran QEMU with my CompactFlash card passed through as the hard drive again.

I installed Alpine Linux and configured it to use the serial port for communication.

I recompiled Linux enabled the Geode LX video driver and set Alpine to use that.

Somehow VGA worked and displayed a login prompt, but degraded over time until it was unreadable.

Probing the signals showed very weak square waves on the DAC analog outputs.

== Probing desoldered analog signals ==
I figured the analog outputs were being pulled low somehow.

I desoldered the inductors and capacitors so the analog signals weren't loading any current.

Probing the signals showed no difference.

I attempted to solder things back together but it ended with ripped pads. I had a webcam acting as a microscope that didn't magnify well and blocked my motion, and too large of a soldering iron tip that I couldn't position properly. I also lacked a lot of experience and didn't know what to do.

In short I managed to:

* Lose three capacitors
* Desolder two inductors
* Rip a set of pads for an inductor
* Rip a pad on one of the analog output capacitors
* Burn a lot of plastic

Not a great move on my part.

== Probing desoldered signals again ==
After a long hiatus I took a look at the signals again. At this point I don't see any signals at all, even after attempting to patch drivers to output test signals. Still my monitor detects Linux. Interestingly my USB to VGA setup detects both BIOS and Linux. Maybe the 'Input not supported' is related to HSYNC/VSYNC polarity?

I believe the capacitors I desoldered were connected between analog ground and the red/green/blue signals. I'd imagine they're very low values around 10pF or something.

Soldering the inductors back seems like the next step so I can hopefully try to reproduce the Alpine Linux login prompt.

I did probe the area with a multimeter. I think the middle ESD diode has failed. Could that be pulling things low?

== Rebuilding solder failure ==
I managed to spend a lot of time and effort rebuilding the stuff I messed up to the point it works again. I actually couldn't find the ceramic capacitors I ordered so I skipped them. I documented this failure in [[AMD Geode/Video 2]] .

Alpine Linux does give a nice login prompt again.

Looking at the ESD diodes again shows they have correct values, so it was likely just me measuring an incomplete circuit.

== More tests and measurements ==
I did a lot of software tests:

* Entering test mode gives a blank screen
* Datasheet seems wrong: "MBD_MSR_DIAG[18:16] = 101h" should be "MSR_DIAG_VP[18:16] = 0b101"
*Still, DAC test mode doesn't seem to work for me
* DAC status registers always say the DAC is below 0.35V
* VGA to HDMI gives the same weird signal on a monitor
* One time VGA output was proper white
* Another time VGA output was blue
* Another time it didn't show up at all

I did the following measurements:

* The DVREF shunt looks okay and produces the correct voltage
* The DVREF shunt has a 12nF cap in parallel
* The DVREF shunt has 13k to the 3V3 rail
* The DRSET resistor is 1.21K to GND
* The VGA resistors seem fine and correct values
*When external DVREF is used, the shunt drops to 800mV, the rail somehow drops to 960mV

I reflowed some solder pins that looked a little lacking to me. I managed to break the second Ethernet chip doing this. Fixing it would require running wires for some PCI lines probably. I put nail polish on the exposed pins to protect from the elements.

I put some contact cleaner on the VGA connector pins and that didn't help.

== Even more measurements ==
Still heading down this trail of madness:

* DAVdd drops to around when Linux initializes video
* Bridging DAVdd and the 3V line causes Linux to display white text
I removed a capacitor and replaced a resistor with a wire, this fixed the DAVdd rail I think.

At this point I believe the DAC is working, so the problem might be related to clocking.

I added in a fresh clock battery and it had no effect.

== BIOS modding ==
I checked the VGA output and it seems fine when Linux uses it, but during boot it just doesn't use VGA out at all. It's seemingly likely that the BIOS is deciding not to output to VGA. I decided to try flashing a known good BIOS, and if that didn't help I'd tweak the settings using a BIOS editor.

I installed FreeDOS 1.2 and used the following commands in AUTOEXEC.BAT to output over serial:

 MODE COM1: 9600,N,8,1,P
 CTTY COM1

This didn't work in FreeDOS 1.3 for some reason, perhaps its serial is broken.

I downloaded the following tools:

* [http://awdbedit.sourceforge.net/ awdbedit-1.0_bin.zip]
*[http://www.lejabeach.com/CBROM.html CBROM32_198.EXE] (listed as CBROM 1.98)
* [http://www.lejabeach.com/Modbin6.html modbin62.04.03.exe]
* [http://ftp.emacinc.com/LegacyProducts/SBC/pcm-9375/Drivers/BIOS/ awd829.EXE]
* [https://www.wimsbios.com/awardflasher.jsp awdflash899.exe]
* [https://downloadt.advantech.com/download/downloadsr.aspx?File_Id=1-1E1MK Awdflash.exe]
I downloaded the following BIOS versions:
* [http://ftp.emacinc.com/LegacyProducts/SBC/pcm-9375/Drivers/BIOS/9375f113.zip 9375f113.zip]
* [http://ftp.emacinc.com/LegacyProducts/SBC/pcm-9375/Drivers/BIOS/9375f114.zip 9375f114.zip]
* [http://ftp.emacinc.com/LegacyProducts/SBC/pcm-9375/Drivers/BIOS/9375v115.zip 9375v115.zip]
* [http://ftp.emacinc.com/LegacyProducts/SBC/pcm-9375/Drivers/BIOS/9375v117.zip 9375v117.zip]
* [https://www.advantech.com/support/details/bios?id=1-22Y526 9375V115.BIN]
* [https://www.advantech.com/support/details/bios?id=1-2DYMY1 9375V116.BIN]
* [https://www.advantech.com/support/details/bios?id=1-2DYMZE 9375V117.BIN]
* [https://www.advantech.com/support/details/bios?id=1-2DYN1X 9375V118.BIN]
I also found [http://ftp.emacinc.com/LegacyProducts/SBC/pcm-9375/Drivers/BIOS/readme.txt a warning on CRT support with BIOS 1.17] and [https://www.advantech.com/support/details/faq?id=1+FY+68 how to make a CRT-only PCM-5862E BIOS].

Using modbin6 I was able to check the BIOS settings and found that it's already set to CRT and doesn't have the broken default 1.17 has, so I'm not even sure this method is going to work.

Using awdbedit I was able to decompress and extract parts of the BIOS.

I wasn't able to use any version of awdflash to flash any BIOS, at least I didn't think it worked. It requires VGA output to display its menus. I might try Uniflash or flashrom in the future.

This is also risky in case I break the BIOS. The chip is a PM49FL004TJ chip and I don't have a flasher for it.

I ordered 5 of those chips off eBay, my plan is to hot swap the chip and use that for BIOS foolery and keep the current one as a known good.

== Ethernet desoldering ==
I attempted to desolder the RTL8139DL chip I had broken earlier. Even with a large amount of flux and 40 minutes of hot air rework at various it didn't come off, instead making a flux mess and possibly damaging the board.

I tried again the next day, changing my technique to use a larger nozzle, putting the nozzle closer to the chip and using tweezers to push the chip upwards to try and defeat surface tension. It worked!

I prepared to re-solder the 6 lost pads, belonging to the following pins:

* GND
* AD21
* AD20
* AD17
* IRDYB
* TRDYB

Only to find out only AD17, TRDYB were visibly connected. It's unclear where IRDYB connects, if at all.

== BIOS flashing ==
One night I was bored so I installed Debian 12 in QEMU then transferred it to the board's CompactFlash. The remaining Ethernet connector didn't work but I'm not sure if it ever connected.

I had some trouble with the RAM detecting. The board wouldn't beep due to lack of RAM but it also wouldn't initialize. Under a microscope the joints look bad, but I'm a little more suspicious of the SODIMM connection pins in the socket itself. The RAM doesn't feel like it snaps it properly, or maybe it's not making proper contact.

Only one Ethernet port shows, but even that one doesn't work. Maybe I've fried that chip? Though it wouldn't really enumerate. There was a resistor I lost when desoldering, that might be important.

The Molex USB to power connector also seems to have failed, because the crimping on it failed its once job of handling physical strain. I decided to just solder the crimp and be gentle with it for now.

Once Debian was up and running I used flashrom and chip hot swapping to try BIOS versions v1.18, v1.17, v1.16 and v1.15. None of them had any changes.

I gave up after a while until I decided to try the TV connector jumper on the board. I had tried this previously and had mixed results as it just seems to stop the boot altogether. When I tried booting it the machine didn't power at all. Removing the jumper also kept the system not booting.

I tried swapping RAM sticks but overall it seems like the slot really is failing in some way. But when it did work something interesting happened: The screen completely worked. I managed to see the BIOS screens and GRUB.

I'm tempted to believe this whole thing is some kind of memory issue, but that doesn't make much sense given OSes seemed to run just fine.

== Ethernet pins ==
While working on video 3 I was seriously, seriously confused about why IRDYB and TRDYB wouldn't be connected. It mades no sense, this is REQUIRED for bus transfer. The mystery is that I was looking at the chip upside down.

The real broken pins are:

* CLKRUNB
* REQB2
* GNTB2
* AVDD
* RTT3
* LWAKE

Most of these are test pins. The only ones that were connected was AVDD and RTT3, both should be fine left disconnected!

== Ethernet testing ==
I decided to try and figure out if the reason the remaining Ethernet was broken was due to me using the wrong type of Ethernet cable: I usually run small cross-over cables from the switch on my desk but the board may need an actual cable.

I booted the board and confirmed VGA works, but I was having serious problems with serial garbage. I had to send a break to get the Linux prompt to show itself.

Surprisingly, the board no longer acknowledges it has any Ethernet ports. It doesn't seem to show up in lspci either. Really not sure what is happening here.

I tried to reboot the board and go in to setup to check that. The delete key didn't work. In fact only some keys on my PS/2 keyboard worked. Same with a USB to PS/2 adapter. I did try to 

VGA turned garbled very fast while the machine was running. I lifted the board up and found the RAM stick fell out. That's not normal, right? RAM sticks aren't supposed to do that.

Also the reset button isn't working. I think this started when working on the VGA port, not the BIOS or Ethernet.

I think the next step is clear: Fix the RAM slot.

== To-do ==
I need to:

* Repair traces
* Reinstall Ethernet
* Validate the USB power cable
* Fix the RAM
* Run a memory test using memtest86+ and its serial mode
* Debug why remaining Ethernet is broken
* Debug why PS/2 is broken
* Check if it was the BIOS breaking the VGA
*Repair the reset button