Telling the story of the dying SAAO CCD instruments in layman terms. The SAAO CCD on the 74in telescope was driven by a combination of a PC and a ‘Merlin’ crate. These crates were built in the early days of the personal computer. In those times, directly accessing the memory was a matter left to hardware – there was a chip on the motherboard that one could fire off an interrupt to, which would instruct the memory to be dumped. The Merlin crate of those times had little or no intelligence built into it to handle timings. Timing was precise simply because it was done in hardware. And so times moved on…In today’s world, a lot of previously hardware-controlled “stuff” is now handed off to software. Specifically, direct memory access (DMA) that was used to, in effect, read the computer’s memory in one single instruction, is now done via software. So the Merlin crate used as an interface between the PC and the CCD, issues an instruction to read out the memory, but in fact that DMA request does not arrive as precisely as it used to when it was done in hardware. And the Merlin crate is unable to handle this slight change in timing. According to Jeran, who did the testing of this in the electronics lab in Cape Town with an oscilloscope – the Merlin crate is “super picky” about it’s timing. Even being out of sync by 0.1ns, causes the Merlin crate/PC to hand requiring a full reboot of both the crate and the PC.
Why does the old imag40 still work on the 40in? This is somewhat of a miracle really. If we could find a PC that has that chipset with that memory and all other things being equal, we may get this to work. But that’s harder than one would expect! Typically even two identical computers with precisely the same specifications often do not necessarily use the same chipsets under the hood. In fact we bought a number of industrial PC’s to replace these aging PC’s. Despite the invoice specifying identical machines, some of them differed in really subtle and often surprising ways which only became apparent when we started testing with them. Also, it’s clear that this is now old hardware. Buying old hardware, some of which has gone out of production is relegated to cruising Ebay. And this could prove to be both time consuming and costly. In doing a half-an-hour search, we came up with some close hits…..but those all cost in the order of $350 per board, excluding shipping. Since this is all second-hand hardware, there are huge disclaimers on the site that states they’ll not be held liable if it does not work when it arrives – and there are NO returns. That deals with the PC. Then there’s the potential of modifying the hardware (i.e. the Merlin crate).
DaveC and I discussed this and he estimates about 8 weeks work by a senior electronics technician followed by an indeterminate amount of time from a software developer to rewrite/change the necessary software is what’ll be needed to make the Merlin crates compatible with this new rather hit-and-miss affair of dealing with DMA access.
I’ve included Jerans Technical Report for those of you who are geeky enough to want the blood-and-gore. The main investigators in this sordid tale, who spent a good amount of wasted time chasing down this bug (in no order of seniority, priority, time-sink order are):
John Menzies – who I was certain was simply asking for another industrial machine to see how many he could actually squeeze into that office of his
Garith Dugmore – who got Linux 18.104.22.168 to boot on this old hardware. Ably assisted by Jean Bernardo
Jeran Cloete – The highly technical electronics guru who pulled himself away from his 3D printer/Pi projects long enough to scratch around
Me – Oh how I hated going through rebuilding the Linux kernel with RTAI on top of it!
DaveC – Probably the one with the higest vested interest in seeing the Merlin crates die – ok, perhaps not. But who’s opinion of these things we all defer to.
Others not mentioned by name who also hacked these things to try to keep them alive. To you’all. I salute you.