Zpectrometer setup configuration and change log

2008 Mar 7: Installed broadband filter and adjusted padding for total power gain  monitor.  See session 40.  Power supply monitor point logging now running.

2008 Aug 25-29: Changed backplanes on all correlators to eliminate plug-socket Phoenix connector. 
                            Added band-limiting filters to Port A inputs on BE to try to recover spectrum across lower 2 GHz of band (cwcal dies off earlier than Rx noise?)
                            Replaced fiber interface and hub with integrated unit.
2008 Oct 30: Install Zpectrometer on GBT. 
2008 Nov 3: Finished installation.  Complete total power chain, changed high-lag amp module on BE3 (turned out to be a bad pad at input)

2008 Dec 16: Migrate software to RH Enterprise 5 software announcement.

2009 Jan 6: Galen moved BE0 filters from Port A to Port B, tightened connectors in receiver.  Zero lag now inside correlator.
2009 Jan 7:  Sess. 23 follows for test.  First remarks about pointing difficulties, but weather is very bad.  Passband gain 1.6 lower than previous.
2009 Jan 8: Galen looked at IF outputs from cryostat etc.  email "Ka Receiver Ripple, Jan 8." 

2009 Jan 12: Rx off telescope to look for ripple, change cold head components
2009 Jan 13: Lab work, warm Rx.  Added isolators at outside of cryostat wall, swapped tripler/LO splitter units between arms.  Rebiased to minimize second harmonic. 
                     Unexplained step changes in power output at J4, about 4 dB.
2009 Jan 16: Back on telescope, first tests Jan 16 (session 24).  Set attens 2,2,2,2 dB (sess. 24).  Still ripples.  See sess. 26 scan 40 for large example

2009 Jan 23: Galen redoes insulation arount total power circuit, offset back to normal at -0.216 V.  But total power is now about 0.5 V on sky after receiver work. 
                     Total power values were bad from 09:15 on Jan 6 (filter change?) to 14:45 on Jan 23, so sessions 23-26 are affected.

2009 Jan 24: Check levels and set attens at 0,0,0,0 dB (sess. 27).  No idea why power has dropped. 
                     Pointing had been difficult to impossible in sess 23-27 plus Smail/Hainline obs, S/N looked lower than usual; related?
                     Pointing S/N seems to improve after restarting LO1 system (to solve LO1B off, which killed pointing) in sess. 32,  1/31/2009

2009 Jan 29: Changed band edges in redBodyC.scr to [55, 492, 55, 492, 55, 492, 55, 57] from [50, 490, 50, 490, 50, 490, 50, 270], trim off BE0

2009 Feb 11: Laura Hainline adds Fourier-component ripple removal, released as redBodyD.scr.

2009 Feb 26: Galen swaps total power monitor from IF downconverter box Mon A to Mon B.  Tapped on various parts of the receiver and Zpectrometer, saw no power variations larger than 0.1 dB.  Total power and gain values should be different now.

2009 Mar 02: Measured receiver gain at downconverter monitor port B with noise diode/demodulator setup: 7 to 9 dB gain difference compared with port A, hard to pin down exact value since gain changes with time (see plot).  Total power tracks shapes, with additional noise.  Power drift seems to be "reset" with relay throw -- sign of a bad relay?  Correlator variances higher, set attens to 1,1,1,1 dB.   cwsigs comparison shows +2.4 dB gain change since last cwsigs on 2009 Feb 15.  8C-9 and 9A-64 had one run each here between swapping total power monitor point and the 8C-9 session that provided these data.

2009 Mar 06: Galen replaces relay after downconverter -- contact came unscrewed from relay instead of cable. Total power monitor still on downconverter port B.   Ray is tracking down a problem that results in scans being logged but not written as files -- something stops the data acquisition program.

2009 Mar 10: Galen moves total power detector back to downconverter monitor port A.  Spectrometer testing with warm load over both horns shows 300 MHz ripple.

2009 Mar 20: Galen replaces pad and cable between PIN switch and downconversion mixer.  This area was sensitive to cold and mechancial stress, and one cable end appeared to have a crack at the connector interface.

2009 Mar 30: Galen replaces oscillating bias cards.  Had previously added thermal blanket around cryostat to reduce cavity mode changes with temperature.  Tests with 9A40 at night showed ripples with varying periods; see "Last night's Ka tests" email 3/31/09.

2009 Mar 31: Decision made to remove Rx on Thu, 4/2, return to telescope in early May for other projects, back down at the beginning of June for the summer.

2009 Apr 2: Receiver removed, Zpectrometer powered down.

2009 May 30: Receiver reinstalled with heavy thermal insulation.   Area near downconverter mixers was very temperature sensitive; downconverter mixers have different thermal sinking than before.  Zpectrometer powered up.  The plan was also to replace the LNA bias cards with a previous generation, as the version that has been in use is prone to oscillations (see Mar 30, initial NRAO wiki notes on Rx).  

2009 May 2-6: First tests in poor weather; looks like zero lag has moved out of correlator.  This is later (May 6) traced to phase flips in the cwsigs data, origin unclear.  On May 5 Galen removes a 3 dB pad to shift lag pattern (which turns out not to be the problem) and to increase power level.

2009 May 10: Observations in moderate weather show many spectra with baseline ripples.

2009 June ~1: Receiver off telescope.
2009 July through December: Receiver tests in lab.

2010 March ~5: Install receiver, Zpectrometer; Zp powered up March 5.  Receiver is in pretty much the same shape as last year:  "The receiver is as it was last year with the additional bracing [diagonal bracing from the feeds to the 50K cold plate].  There is a slightly different dewar window as we used the previous hardware for the unsuccessful Teflon window project, and a slightly different radiation shield covered with sheet absorber around the feeds, which seems to have reduced the feed temperatures by about 10K."

2010 March 22: Tests on 2010 March 18 showed that the 10.5 GHz LO for BE3 in the IF downconverter had lost IF power.  Galen Watts removed the downconverter that night, trouble-shot the system and replaced the DRO today.  Clarified "unstable mixer" remark: one of the downconverter mixers (a Marki unit) had been changed to increase reproducablity in Trec measurements -- this mixer was changed earlier (see 2009 May 30), but may have been changed to a particulary unstable device.  The LO system for the mixer has not been changed.

2010 March 23: Galen reinstalled downconverter.  Tests show all bands work.  Modified cwsigs script so all channels are on the same spacing, saved as zpec10-config-cwsigs.  Ran as check, all looks good.  Stability tests continue to show 1/f region in Allen variance power spectra.  Although narrow features aren't as pronounced, they are clearly visible in be1-3 at lags 242, 247, 250, with some weaker and broader structure near 254; see plot.

2010 Apr 9: Replaced blown fuse at line input of main power supply.  Fuse had blown Mar 25 during mainenance time, which may be a coincidence.

2010 Aug/Sept: Change to 90 degree hybrid, lab tests

2010 Sept ~20: Install receiver and Zpectrometer on telescope.  Apparently added a pad to one input to balance signal levels.

2010 Sept 30: Initial tests with zp3 filters removed to see how high cal signal goes (fades out about 37.8 GHz).  Sess. 61

2010 Oct. 26: Galen replaced LO unit in Ka-band receiver to eliminate spurious signals, especially bad at twice LO, or 29.233 GHz for the Zpectrometer.

2010 Oct. 28: Galen added ~3" cable to J6 to bring zero lag a few lags further into correlators -- after receiver rebuild, it was at lag 256 for several chassis.  Also tests to isolate origin of ripples.  Sess. 63.

2010 Nov. 16: Galen ties IF cables together and then to fixed points to reduce possible phase noise or other effects from cable motion; also installs filters for zp0 (see 2008 Aug 25).

2010 Nov 30: Zpectrometer removed from telescope to make room for Ku-band receiver.
2010 Dec. 8: Ka-band receiver removed from telescope to replace Ka-band amplifiers, which had been damaged by flexure with new mounting method (probably at quad-hybrid rebuild in Fall 2010), coupled with thermal stress from refrigerator failure.
2011 Jan. 6: Ka-band receiver and Zpectrometer reinstalled.  Cryo LNAs are somewhat different than previous versions.

2011 Mar 10: After no signal in session 11A27-02, Galen resets Rx 28V power supply, which had zero output voltage.  This power supply drives relays in IF system.  Note in one of Galen's emails refers to fairly recent lightning strike.

2011 Apr 20: Zpectrometer  removed from telescope to make room for 4 mm receiver.  Zpectrometer has not been communicating since power outages and heavy rains on weekend of Apr 16/17, reason for loss unknown, although many problems in receiver cabin and rumors of water dripping from Zpectrometer enclousre.  zp0 micro has had intermittent communications problems for log data, sometimes telnet, but rarely lag data intermittently since 2010 Nov, probably needs attention.

2011 Sep 28: Zpectrometer installed, all nominal.  9/29: temperatures high, shut down. 10/5: Galen removes protective plate left over from installation, starts up.  Testing runs into cwsigsLoc probelm. 

2011 Oct 7: In session 11A27-10 concerns about bad relay: baseline problems in seq. no. 15, 16, slowly improving.

2011 Nov 11: Fuse in Zp power supply blew, lost 10-15 min of Frayer session and all of Wagg session
2011 Dec: Obs cancelled for high receiver noise due to water inside weather radome, software problems on different days (resolved by restarting manager).

2012 Jan 4: Zpect off telescope

2012 Feb: 2/7 Zpect installed.  2/13 Big power glitch on telescope, Zp power supply fuse blows, other problems in cabin.  DCS pointing dead, point with CCB.  2/14 Obs ok, pointing with CCB.  2/15 and on, calibrator fluxes are zero, cwsigs look different than before. 2/17: Same.  2/21: Testing, then shut down Zpect.

2012 Mar 1: Zpect off telescope.

2012 Mar 27: Actuator in secondary failed, relpaced, but questions remain about recalibration for pointing, focus, etc.

2012 Mar 28: Spent day in GB lab (winds too high for tests on telescope).  Added DMS285 splitter as power combiner for TotPwr monitor, so TotPwr should approximately double.  Put IF processor on network analyzer -- checked out fine.   10 dB gain problem for Spectrometer, work on IF that probably doesn't affect the Zpect. 

2012 May 3: Zpect back on telescope.  Galen has reworked signal path to deal with potentially troublesome relays.  Tests: usual problems with cwsigsLoc, then phase cals.  Signals a factor 3-7 low in power, but passbands are close to earlier ones when all was working fine.  Zero lag is possibly out of correlators, however.

2012 May 8: Galen reinstalls 3" of cable on J6 arm (see 2010 Oct. 28), reworks IF in area bypassing Rx second hybrid, trims power to -20 at J4 and J6.  Tests in 12A376_07.  BE0,2 seem to be ok, BE1,3 seem not to respond to RF -- just noise.  All BE working Apr 3, so why two have dropped out is unclear.  Also, noise diodes are now on when they should be off, and vice versa.
2012 May 15: RF problem traced to 10.5 and 15 GHz LOs, which are dead or nearly so -- oscillate briefly at switch-on, then fade away.  Since both died in the same short period the power supply is suspect and should be replaced too.  Noise diode problem traced to receiver side -- something there is doing the inversion.
2012 May 18: Galen replaces all LOs with spares including 19.0 GHz, which was fine but replaced as a cautionary move.  Problem is thermal in origin.  Bias lines strapped to center pins now.  Noise diode drive was lost at about the same time as oscillator problems started; possible lightning strike?  Hard to see why this would only affect two oscillators deep inside boxes, but hard to see why two should have problems at the same time, particularly close in time to problem with noise diode drive.

2012 May 22: Reinstall Zp and check -- all seems ok.  Noise diode receiver chip removed, set so always off until it can be replaced. Galen notes that noise diode drive in Rx blew at about the same time as LOs failed, so perhaps lightning strike?
2012 June 8: Noise diode is responding to signals again in 12A376/11 (switching on/off, anyway, probably under clock control), could be jammed on by setting poke 0.3 0 and off with poke 0.10 1 (so, backwards again).
2012 June 13: Noise diode driver chip was replaced.  Subsequently wiring patched for correct polarity, no clue why polarity is inverted.

2012 June 16: Powered off Zpectrometer correlator.  Still on telescope, but no promise of observing weather, so no point in keeping it on.
2012 Jul 2: Powered off again, must have restarted after power problems; major storm on 6/29 may have hit GB, micro's timer showed power since Jul 1, 02:32 EDT.

2012 Sept 9: Power up Zpect.
2012 Oct 19: Zpect off telescope
2012 Nov 25: Observing again, Galen finds and fixes connector problem in Rx
2012 Dec 14: Zpect scheduled to go off telescope

2013 Feb 12: Zpect installed
2013 Mar 6-8: Snowstorm and power outage in Green Bank; no contact with micros afterward until Galen replaces power supply fuse Mar 9.

2013 Jul 6: Zp installed.  Log in to turn power off 7/15, but no response; logs end on 7/13.  Replace power supply fuses with medium acting (instead of fast-blow) on 8/22.
2013 Oct 1: No response, but Zpect scheduled to be removed from telescope this week.

2013 Nov 5: Zpect installed; turned off correlator power 11/26
2013 Nov 30: Turn power on.  Observations resume after secondary actuators had been replaced and recalibrated (summer-fall project with no high-frequency observations).
2013 Dec 1: Start observing; zp0 did not respond on ethernet.  Galen unplugged/plugged ethernet cables and all works.  Total power stuck at 13.3 V, no obvious reason.
2013 Dec 3: Bob Garwood fixes filler to accomodate 64-bit upgrade
2013 Dec 27: Total power monitor reading sensible numbers again, no obvious reason for "fix."

2014 Jan 1: ToPwr back to 13.4V -- something intermittent, not broken.
2014 Jan 3: TotPwr -0.196. 
2014 Jan 5: Looking at ripple structure, which has gotten worse and changed frequency cf. previous years.
2014 Jan 10: Zpect off telescope

2014 Feb 20: Zpect installed.  TotPwr -0.170 V.
2014 Feb 26: Observing, but earlier detections seem spurious.
2014 Apr 18: BE1 in an odd state with mean ~20 and variance ~<0.1.  Cycled power and all returned to normal.
2014 Jun 18: Notes of some level problems in zp1.
2014 Jul 3: Power up Zpect.  Note that new synth had server name problem, accounts for no tuning in 2014 Jun 6 cwcal.  Not sure when the synth was changed.
2014 Jul 7: Problems in cwcal from 2014_06_14.  Power down.

Questions or comments?  Please contact Andrew Harris.