For some convoluted setup reason I can’t remember now, when I started running the MESA (Modules for Experiments in Stellar Astrophysics) stellar evolution package for my Stellar Structure and Evolution class in Winter 2023, I couldn’t get the live-plotting output I was supposed to be able to see. MESA writes its output into data files I was going to have to figure out how to read anyway to do the homeworks, so I wrote my own live-plotting utility to see what was going on! What follows is exactly what I wrote as an appendix to the homework for which I first wrote this code, and my code is at https://github.com/aditya-sengupta/MESA.jl.
My aim here is to create a Julia function that can simultaneously run a MESA simulation and watch its history.data for updates and plot them. Since these will have to happen simultaneously (or I’ll need to carefully sync my ./rn from MESA and when I start watching in the Julia REPL, which is annoying) I’ll have to have two threads. The “run” thread is fairly straightforward – there’s a way to cd into the MESA run directory and to run a bash command from Julia – but file watching will be more complicated.
For file watching, I first have to find the file we have to watch. This is difficult when there’s multiple inlists, as there may be many ways in which a MESA run file/main inlist represents its sub-inlists. In this homework, I have history.data files in two sub-directories, and it’s hard to tell Julia when to switch from one to the other. Fortunately, because of how MESA works, I don’t have to do that. MESA seems to always write to RUN ROOT/LOGS/history.data, then rename it to RUN_ROOT/LOGS_inlist_name/history.data at the end of the run, which means I can just watch the file in LOGS the whole time.
Overall, we want to have a data stream from history.data open, and whenever there’s a new write to the file, we get each new line from it, convert it to data we can plot, plot it, and start waiting for the next. This is pretty well supported by Julia: we loop until the MESA task returns, watching the history file until it changes and reading each line until the end of the file till it does. We grab just the indices corresponding to logTeff and logL (which we’ve got from the header at the start), and plot the new point.
Since we might be dealing with hundreds, thousands, or tens of thousands of data points, replotting the whole time-series at each iteration won’t scale well. To remedy this, we keep the last datapoint around, and when we want to plot a new point, we make a line between the old and the new one. So the plot we see is the composite of n two-point plots streaming in.
The only complication now is what happens when we switch inlists. If we’re on line 20, our data stream will be expecting a new write at line 21, and won’t see what MESA is writing at line 7 of the new file at the same location. I tried solutions involving a second watching thread that alerts the main program when there’s a change to LOGS, but that task doesn’t terminate till we tell it to. And telling another thread to stop involves passing messages through channels, which proved to be really complicated to implement. So my much quicker solution is: keep watching the number of lines in history.data, and continue as normal if it stays the same or increases, but close and reopen the data stream if it ever goes down. This isn’t very scalable, but it was simple to test. If we need scalability, we can change this to checking the file size, which doesn’t require opening the file. A possible edge case here is a switch from an inlist with very few data points to one where the first few iterations are very quick and all get written to history.data at once; it’s possible this condition wouldn’t trigger in that case. But the number of iterations that gets missed here would be pretty small, by the nature of the scenario, so the overall aim of “get a sense of what the simulation is doing in real time” would still be met.