Dates: April 10
- 24, 2000
Observers: John Monnier, Rafael Millan-Gabet,
Irene Porro
(1) Short Delay & Fringe Piston Noise
On April 12 we did some tests to map out the parts of the SD track over
which piston noise has appeared in the fringe data since February 2000.
For reference, I include below our original email reporting on this
topic:
Hello,
Based observations of our targets we developped the theory that the so called "vibration problem", which appears as large piston noise in the fringe data (very wide spectral peaks around the expected fringe frequency), happens only when the short delay is on certain parts of the track.
Tonight we tested this idea by taking many sets of scans on the same star, and adjusting the long delay position so that we could take data over the suspected parts of the track at will.
The result is that very consistently the piston noise, showed up while the SD was in the interval [0.2 m,0.6 m] as well as near 0.8 m and 1.6 m (see graph below). In the other parts of the track the data didn't show this problem.
After the test, we continued observing avoiding those parts of the track, which served to re-confirm the theory, in the sense that the data was OK except when we purposefuly ventured into the bad regions of the track.
So, we are thinking that maybe there is something wrong with the track surface, and that perhaps it needs cleaning or polishing.
I believe that you have done something similar before. If it sounds to you like this would be a good idea, can you please suggest us how we could clean/polish the track to try to fix this problem?
Plot of Piston Problems from Apr 12, 2000
---------------------------------------------
Each symbol represents one set of scans (about 150 usually)
.=good piston observation
x=mediocre piston
X=BAD piston observation
. X x Xx .
...... XXxXXX. .XXx..x .... xx x . .
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2
Typical Short Delay Position During Scan
Notice High degree of consistency, esp between 0 and 1.0 meter where we made multiple repeat observations.
At Wes's and Nat's suggestion, we opened the tank to inspect the track and magnets/coil on April 13. Here's the email reporting what we found:
Hello,
We opened the tank to try to identify the problem with the SD.
At the usual air flow rate, the carriage only moved smoothly in the first 20 cm or so near home. Further than that, it was hard to move by hand, and the friction could be clearly felt and heard. The motion became acceptably smooth only at a much higher flow rate (left meter reading 3 instead of 2, right meter off scale).
At both the normal and high flow rate, and at both good and bad places on the track (as determined by the experiments the previous night), I measured that the gap between the coil and the bottom magnet is 0.0375" and the gap between the coil and the top magnet is 0.025" (measured with shims). Therefore, the gap is smaller at the top by 0.0125", or 50%. However, we didn't see actual friction between the coil and the magnets. Therefore, we held off on the fix explained by Wes consisting on changing the height of the table with respect to the granite by adjustment of the 3 screws above the feet.
We felt the granite surface carefully with our hands and didn't really
feel anything spectacular (scratches or dirt). We marked the surface with
black ink and ran the table above it. The table left behind very clear,
and at places very deep, marks on the ink, particularly on the East side.
We cleaned the surface with alcohol several times, and
perhaps improved this a little, but marks still appeared on ink.
Another potentially useful observation is that, when the motion was very rough at the normal flow rate, we could induce smooth motion by pushing down on the cart on the West side.
Attempts to identify the source of the friction noise using a plastic tube stuck in our ears failed.
CONCLUSION:
It is not clear at all that the problem is contact between the coil and magnets.
The ink test appears to show that the East side feet (one or both) touch the granite surface.
SOLUTION: (none yet)
"Key point number 7" in Wes's notes dated Sat. 16 Nov. 96 appears to say that adjusting the screws above the 3 top feet would also fix this problem. However, from looking at the sketch of the thing, we don't really understand how that would be the case. Also, the support for the SD dihedral seems to be bolted above the screw for the West side foot. For those two reasons (mostly the first) we chickened out on making this adjustment until we completely understand what we are doing. Perhaps Marc can explain tomorrow how lifting the table with the screws adjustment results in the feet floating up higher. (Is it because of an actual stronger attractive upward force due to the magnets?). Also, even if this were the case, given that the coil is already closer to the top magnets (0.025" away from them), an issue would be whether we can lift it enough without touching the top magnets.
So, we decided to spend the second half of the night observing using the high flow rate that appeared to work well at atmospheric pressure.
Unfortunately, in vacuum this flow rate resulted in frequent, large and fast oscillations, and so we had to go back to taking data at the normal flow rate, avoiding bad parts of the track by placing the LD appropriately.
We will continue working on this problem tomorrow after talking with
Marc and/or Wes.
The upshot of all of this is that Marc looked at the track and anorad
table and agreed with us that the problem was in the East-North foot
of the table dragging over the granite surface. He did fix it by lifting
the table using the screws above the table ("Key point number 7"
in Wes's notes dated Sat. 16 Nov. 96 is in fact correct, because
there is another set of magnets which pull the table down, and this
is the force that weakens by lifting the table, allowing it to float on
a thicker air cushion).
Thereafter, the piston noise problem in the data, and the SD oscillations apparent in the SD error signal, completely dissapeared. So, as far as we can tell, that was the whole problem and it has been fixed.
Later Note (April 19): While it is still true that we have had virtually
no more occurrences of the so called "generalized vibrations" (apparent
as wide fringe spectra) since Marc worked on the SD, we did have a few
occurences of the so called "longitudinal vibrations" (apparent as ghost
replicas of the fringe spectral signature, at a low but noticeable level).
We tried to relate the presence or absence of this problem to small adjustments
(10-20%) of the air flow rate, unsuccesfully. So, as of yet we haven't
found the cause for this one.
(2) Long Delay Baffling
On one night our alignment happened to result in laser light from the LD interferometer hitting the star tracker CCD (delayed beam). The laser signal was high enough to prevent tracking on faint sources. Since the laser light was right at the edge of the CCD, we could get out of this by slightly re-aiming 45 mirror (last reflection inside the tank), and compensating with yaw. However this shows that we really should baffle the tank near (just North of) the LD dihedral. Wes encouraged us to make a baffle out of black cardboard, but we never got around to it. Any volunteers?
(3) FLUOR Piezo Scanner
We have noticed a very ugly noise coming out, it seems, from the piezo scanner. You can hear this noise even from the computers area (which is how we first noticed) if you pay atention! Listening closer, it sounds like electrical sparks (there are no sparks of course, this is just a description), or like friction of metal against metal. This is intermitent, and we don't know if it has always been there or what, we just report it here so that the FLUOR guys are aware of it.
(4) Star Tracker
We had several instances of a very peculiar behaviour. Since we don't
understand it, I'll just describe what happens: for no apparent reason,
the CCD signal displayed (Value(du)) changes (to lower values), and this
is always followed by complete loss of IR signal on the NICMOS3 pixels.
So, whatever this bug is, it changes the angle of the incoming beams. I
think everytime this happens, the CCD shows (with no star on it) a pattern
of bright columns (column 2 on CCD1 and columns 2 and 9 on CCD2). We "fix"
this by recycling the power to the CCDs, which works most of the time.
See
comments
and explanation by Marc Lacasse.
(5) Star Tracking at High Frequency
We discovered that when star tracking at frequencies above or at about
200 Hz, the photometric NICMOS3 pixels show very clear sinusoidal signals,
with 4-5 periods within a scan. Therefore, it seems that at such rates
the star tracker feedback must get into some kind of resonant state. We
verified that by slowing down the star tracking frequency the photometric
oscillations, also very apparent in the scan power spectra, dissapeared.
(6) South Telescope Roof
The South Telescope roof is becoming pretty hard to open/close again.
(7) Network
During our first 3 or so nights of observing we experienced very frequent
(many times during a night) failures of the internal network, which prevents
observing because it kills the communication between the Telescope and
Star Tracker programs, and with the LD server. We could not find the cause
of this problem. The problem just fixes itself for no known reason
after a few minutes (10-15). During normal network activity we disconnected
our connection to the outside, and the internal network continued to work,
which means that communication between our computers does not depend on
the outside connection (maybe that should have been obvious...). During
one of the failures we disconnected each of our computers in turn from
the network, and this didn't have any effect. Also for no known reason
this problem went away after the third day or so, and for the rest of our
run we had no failures of the network at all.
(8) FLUOR I2 Output
We noticed on April 16 that in this run the FLUOR I2 output was unusually low, compared to previous runs (November 1999 and February 2000).
We taled about this with Vincent and he agreed with our suggestion of trying to clean the fiber head.
This is the email John wrote to some of us after doing so:
In response to a discussion with Vincent about the flagging throughput on the I2 channel (lower by a factor of 2 or 3), we cleaned the fiber tips using alcohol. As of now, we only cleaned the four fibers in the bundle going to the camera. Because of the dramatic results, we would like to also clean the fiber tips at the parabola foci as well.
Results:
I1: 40% Improvement
I2: 250% Improvement !!
P1: 0% Improvement
P2: 4% Improvement
An Example of the SNR Matrix we have now is (under horrible seeing):
I1 I2 = 36.1
27.5
P1 P2
29.0 10.1
Note that the overall alignment is not optimal and we've seen higher throughput through the South telescope recently (50% higher when pupil is better aligned) (P2).
Not sure why I1 and I2 were the dirtiest. The fibers which are plugged and unplugged all the time to allow injection of Maglight are I2 and P1.
Perhaps the recent degradation of the I2 throughput occurred during such an alignment procedure. We are not sure when the degradation occurred, but was present in the Heidelberg data, taken just before our run.
Recall that the 250% improvment to I2 just brings it back to the
November/February throughput, but completely accounts for the large loss
which we discovered this run.
However this seemed not to be the end of the story. On subsequent nights the I2 output appeared low several times again, and cleaning the fiber (by squirting it with alcohol in a light jet, and gently blowing to dry it, and gently tapping to remove any excess alcohol) always brought the signal up. As mentioned above, note that I2 is one of the outputs we unplug/plug every day to do the maglite alignment step. However, we have also observed the drop in I2 signal after having improved it by cleaning and *without* having touched the connector at all.
Another potentially important observation is that I2 often becomes "stuck"
in its connector, in the sense that it is hard to pull out. When this happens,
part of the the connector on the box wants to come out with the fiber,
and one has to hold it and remove the fiber by pulling with some extra
force compared to normal. Note that in that
all the other fibers, including the maglite one, plug and unplug normally
on the I2 box output, which would indicate that the problem is with the
connector on the fiber end, not the box. (maybe the latch mechanism that
covers and uncovers the fiber end).
Later note: The drops in I2 signal have become quite crazy, it happens several times per night, and we now need to also squirt alcohol into the connector attached to the box in order to bring the signal up.
So, we don't quite know what to make of this. Perhaps the FLUOR team would like to come prepared to look at this problem and do some connector/fiber work during their next run.
(9) North Arm Stove Pipe Tip/Tilt Mechanism
When we moved the North telescope from the 35 m station to the 25 m
station, we found out the hard way that the mechanism for adjusting
the tip/tilt was badly damaged. By the way, someone should write up instructions
on how to do this and post them on the web site, because it is non-trivial
for inexperienced observers and there are no
instructions anywhere. In any case, recall that the mechanism
consists of three sets of one bolt and three nuts, separated at 120 degrees
intervals. One is supposed to loosen the lower nuts and get the desired
tilt by adjust the middle nut in the appropriate bolt.
What we found was that the middle nuts could not be moved at all, or
that they would turn the whole bolt+nuts as one piece, as if the bolt's
threads were damaged. We were able to orient the pipe by loosening the
top nuts in addition to the bottom ones, and turning the bolt using the
middle nut. Marc confirmed the next day that the bolts
were damaged (as well as the springy metal section between the top
and bottom sections of the pipe) sometime between February 10 and April
10, 2000. See also
Stovepipes.
(10) Basic Test of SD Control by G3 (April 20)
In order to address the question of the drifting FLUOR Magic Constant
(MC), we did the following simple test.
Setup:
Experiment 1) d2 = 10 cm
Trial # Error (1/1000")
1
+0.25
2
+0.75
3
+0.50
4
+0.75
5
+0.50
6
+1.0
7
+1.0
8
+1.0
9
+1.2
10
+1.2
Experiment 2) d2 = 100 cm
Trial # Error (1/1000")
1
0.0
2
0.0
3
-0.2
4
0.0
5
+0.2
Experiment 3) d2 = 1 cm
Trial # Error (1/1000")
1
+0.25
2
+0.33
3
+0.33
4
+0.33
5
+0.25
6
0.0
7
0.0
8
-0.25
9
-0.50
10
-0.75
Conclusion:
Mean error (of 3 experiments) in coming back to d1 was 0.33/1000" = 8.3 mu, largest error was 1.2/1000" = 30.5 mu. Therefore we think that the G3 control of the SD has passed this test.
Also, we did a tape measure of the actual distance traveled by the SD when commanded to move by 20 cm from the G3, in order to determine whether the G3 has the correct pulses/cm conversion factor. The result was that within the tape measurement accuracy (+-1 mm), the distance traveled was correct.
(11) NICMOS3 Noise Tests (April 20)
We did the following:
a) Turn off all equipment, looking for a source of interference. No effect.
b) Ground dewar and FLUOR table to building earth. No effect.
c) Replace Pentium DIO-32F card with new one, in case it has gone bad and it is putting out ugly pulses. No effect.
d) Look at NICMOS3 power supply outputs, in case it has gone semi-bad.
On the +15V, -15V (analog) and +7.3V (digital) outputs we see 100 MHz
fuzz, of 0.2V amplitude, that is still there even with the supply OFF,
and therefore probably not relevant. On top of that there are larger "spikes"
of 0.05V amplitude (25mV for the digital supply), repeating every 25 us
-> 40 KHz. Those spikes go away when the power supply is turned
OFF.
I do not know yet if that is really a problem. I do not have notes on the detailed look of these levels prior to the appearance of the NICMOS3 pattern noise.
Further progress will involve looking at other parts of the cicuit, after regulation of the supply levels, see if the spikes persist.
Note: The analog supply currently used is a replacement switching supply that Marc gave me after I blew out the original Lambda linear supply (on Nov. 14, 97). The Lambda supply is now fixed and at the site, I could always just put it back and see what happens.
e) (April 23) The Lambda supply does not show the spikes mentioned above... Connect the Lambda poser supply back in the power suplly box: No effect.
f) (April 23) Looked at clocking pulses right at the output of the DIO-32F board: they show large amplitude ringing, which is new. This is not likely due to the board itself, because of experiment (c) above. Could this be due to a problem with the Pentium itself? or due to some weird bad operating system behaviour? Note that this problem first appeared, to my knowledge, in Nov 99 right after the Pentium had a catastrophic hard disk failure. In next run, I'll come prepared to try out a different PC if the bad behaviour of the clocking signals is confirmed.
(12) More SD Tests (April 21)
In view of yesterday's success of the G3 in controlling the position of the SD when it sends it to stationary targets, we did slightly more demanding tests:
Experiment 1)
Re-enact Vx Sgr observations of April 13 from 3 am to 4 am, during which we recorded a MC change of 3 mm. We do this by setting a fake date/time and LD position on the G3. We have communication and data taking with the Pentium enabled. We alternate between source and calibrator exactly as in the real observations. We do *not* step the SD searching for fringes. Note however that we are taking data, and the SD does *something* in this case between scans (as apparent by tracking LED on G3 going ON and OFF between scans ...).
We start the test by defining a zero position using the micrometer as described in the tests of April 20 above, and we will record the error w.r.t that origin at the end of the test (about 1 hour).
Note: a negative error means that the SD fell short of the starting position, a positive error means that the SD overshot the starting position.
Result:
Error = 35/1000" = 0.9 mm or 0.9 x 2 = 1.8 mm in OPD or MC.
(same order of magnitude as 3 mm offset observed in real observations)
Experiment 2)
Repeat, but without alternating between target and calibrator, just tracking on one source, and still recording data.
Result:
Error = 45/1000" = 1.14 mm or 2.28 in OPD or MC.
Experiment 3)
Repeat, but now we just track one target continously for 1 hour, not even recording data.
Result: Error = -30/1000" = 0.76 mm or 1.52 mm in OPD or MC.
(note the odd change of sign!)
Experiment 4)
As a check, repeat yesterday's test of just going out to a stationary target position and back.
Result: for 1 m: Error = 1/1000" = 25.4 um: OK
for 2
m: Error = 1.75/1000" = 44.5 um: OK
Conclusion:
In order to get ~ mm errors in SD position it is enough to have the G3 track the OPD for 1 hour, without doing anything else (Experiment 3).
(13) G3 Data Acquisition Software
John noticed that the last scan in every observation is always bad ...
is this a known bug?