runExternalM2 -- run a Macaulay2 function in a new Macaulay2 process

Usage:

h=runExternalM2(fname,func,params)
Inputs:
- fname, a string, the name of the file that contains the function to run
- func, a string, the function to run. Can also be a named Function or Symbol.
- params, a thing, the parameters for func, usually a Sequence
Optional inputs:
- M2Location => a string, default value null, the location of the M2 executable. If null, then the absolute path of the first entry of commandLine is used.
- KeepStatistics => a Boolean value, default value false, whether to keep statistics on the resources used during execution
- KeepFiles => ..., default value null, indicate whether or not temporary files should be saved
- KeepStatisticsCommand => ..., default value FunctionClosure[../../../../Macaulay2/packages/RunExternalM2.m2:677:27-710:1], an option that tells how to collect statistics
- PreRunScript => ..., default value "true", a way to impose resource limits
Outputs:
- h, a hash table, contains the output of func and other data

Description

This function starts a short-lived new (``child'') Macaulay2 process, loads the file fname, runs the function func with the parameters params, captures the value returned by func, and stores it inside h in the original Macaulay2 process. Optionally, strict resource limits may be imposed on the child process from within Macaulay2, or data may be collected about the resources used by the child process.

Since the child is a new Macaulay2 process, it has no defined variables or functions except those defined in fname. Hence, func and anything it needs (e.g., ring definitions) must be defined in the file fname.

The hash table h stores the exit code of the created Macaulay2 process, the return code of the created Macaulay2 process (see run for details; this is usually 256 times the exit code, plus information about any signals received by the child), the wall-clock time used (as opposed to the CPU time), the name of the output file (unless it was deleted), the name of the answer file (unless it was deleted), any statistics recorded about the resource usage, and the value returned by the function func. If the child process terminates abnormally, then usually the exit code is nonzero and the value returned is null.

For example, we can write a few functions to a temporary file:

i1 : fn=temporaryFileName()|".m2"

o1 = /tmp/M2-94269-0/0.m2

i2 : fn<</// square = (x) -> (stderr<<"Running"<<endl; sleep(1); x^2); ///<<endl;

i3 : fn<</// justexit = () -> ( exit(27); ); ///<<endl;

i4 : fn<</// spin = (x) -> (stderr<<"Spinning!!"<<endl; startTime:=cpuTime(); while(cpuTime()-startTime<x) do for i to 10000000 do i; return(x);); ///<<endl;

i5 : fn<<flush;

and then call them:

i6 : h=runExternalM2(fn,"square",(4));
Running (true && (${prefix}/bin/M2-binary  --stop --no-debug --silent  -q  <"/tmp/M2-94269-0/1.m2" >"/tmp/M2-94269-0/1.out" 2>&1 ))
Finished running.

i7 : h

o7 = HashTable{"answer file" => null}
               "exit code" => 0
               "output file" => null
               "return code" => 0
               "statistics" => null
               "time used" => 3
               value => 16

o7 : HashTable

i8 : h#value===4^2

o8 = true

i9 : h#"exit code"===0

o9 = true

An abnormal program exit will have a nonzero exit code; also, the value will be null, the output file should exist, but the answer file may not exist unless the routine finished successfully.

i10 : h=runExternalM2(fn,"justexit",());
Running (true && (${prefix}/bin/M2-binary  --stop --no-debug --silent  -q  <"/tmp/M2-94269-0/2.m2" >"/tmp/M2-94269-0/2.out" 2>&1 ))
Finished running.
RunExternalM2: expected answer file does not exist

i11 : h

o11 = HashTable{"answer file" => /tmp/M2-94269-0/2.ans}
                "exit code" => 27
                "output file" => /tmp/M2-94269-0/2.out
                "return code" => 6912
                "statistics" => null
                "time used" => 2
                value => null

o11 : HashTable

i12 : fileExists(h#"output file")

o12 = true

i13 : fileExists(h#"answer file")

o13 = false

Here, we use resource limits to limit the routine to 2 seconds of computational time, while the system is asked to use 10 seconds of computational time:

i14 : h=runExternalM2(fn,"spin",10,PreRunScript=>"ulimit -t 2");
Running (ulimit -t 2 && (${prefix}/bin/M2-binary  --stop --no-debug --silent  -q  <"/tmp/M2-94269-0/3.m2" >"/tmp/M2-94269-0/3.out" 2>&1 ))
Killed
Finished running.
RunExternalM2: expected answer file does not exist

i15 : h

o15 = HashTable{"answer file" => /tmp/M2-94269-0/3.ans}
                "exit code" => 0
                "output file" => /tmp/M2-94269-0/3.out
                "return code" => 9
                "statistics" => null
                "time used" => 3
                value => null

o15 : HashTable

i16 : if h#"output file" =!= null and fileExists(h#"output file") then get(h#"output file")

o16 = 
      i1 : -- Script /tmp/M2-94269-0/3.m2 automatically generated by RunExternalM2
           needsPackage("RunExternalM2",Configuration=>{"isChild"=>true});

      i2 : load "/tmp/M2-94269-0/0.m2";

      i3 : runExternalM2ReturnAnswer("/tmp/M2-94269-0/3.ans",spin (10));
      Spinning!!

i17 : if h#"answer file" =!= null and fileExists(h#"answer file") then get(h#"answer file")

We can get quite a lot of detail on the resources used with the KeepStatistics command:

i18 : h=runExternalM2(fn,"spin",3,KeepStatistics=>true);
Running (true && ( (/usr/bin/time --verbose sh -c '${prefix}/bin/M2-binary  --stop --no-debug --silent  -q  <"/tmp/M2-94269-0/4.m2" >"/tmp/M2-94269-0/4.out" 2>&1') >"/tmp/M2-94269-0/4.stat" 2>&1 ))
Finished running.

i19 : h#"statistics"

o19 =         Command being timed: "sh -c ${prefix}/bin/M2-binary  --stop --no-debug --silent  -q  <"/tmp/M2-94269-0/4.m2" >"/tmp/M2-94269-0/4.out" 2>&1"
              User time (seconds): 5.34
              System time (seconds): 0.23
              Percent of CPU this job got: 66%
              Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.34
              Average shared text size (kbytes): 0
              Average unshared data size (kbytes): 0
              Average stack size (kbytes): 0
              Average total size (kbytes): 0
              Maximum resident set size (kbytes): 234000
              Average resident set size (kbytes): 0
              Major (requiring I/O) page faults: 0
              Minor (reclaiming a frame) page faults: 7619
              Voluntary context switches: 1752
              Involuntary context switches: 1948
              Swaps: 0
              File system inputs: 0
              File system outputs: 16
              Socket messages sent: 0
              Socket messages received: 0
              Signals delivered: 0
              Page size (bytes): 4096
              Exit status: 0

We can handle most kinds of objects as return values, although MutableMatrix does not work. Here, we use the built-in identity function:

i20 : v=/// A complicated string^%&C@#CERQVASDFQ#BQBSDH"' ewrjwklsf///;

i21 : (runExternalM2(fn,identity,v))#value===v
Running (true && (${prefix}/bin/M2-binary  --stop --no-debug --silent  -q  <"/tmp/M2-94269-0/6.m2" >"/tmp/M2-94269-0/6.out" 2>&1 ))
Finished running.

o21 = true

Some care is required, however:

i22 : R=QQ[x,y];

i23 : v=coker random(R^2,R^{3:-1})

o23 = cokernel | 9/2x+9/4y 7/9x+7/10y 7x+3/7y |
               | 3/4x+7/4y 7/10x+7/3y 6/7x+6y |

                             2
o23 : R-module, quotient of R

i24 : h=runExternalM2(fn,identity,v)
Running (true && (${prefix}/bin/M2-binary  --stop --no-debug --silent  -q  <"/tmp/M2-94269-0/7.m2" >"/tmp/M2-94269-0/7.out" 2>&1 ))
Finished running.
RunExternalM2: expected answer file does not exist

o24 = HashTable{"answer file" => /tmp/M2-94269-0/7.ans}
                "exit code" => 1
                "output file" => /tmp/M2-94269-0/7.out
                "return code" => 256
                "statistics" => null
                "time used" => 3
                value => null

o24 : HashTable

To view the error message:

i25 : get(h#"output file")

o25 = 
      i1 : -- Script /tmp/M2-94269-0/7.m2 automatically generated by RunExternalM2
           needsPackage("RunExternalM2",Configuration=>{"isChild"=>true});

      i2 : load "/tmp/M2-94269-0/0.m2";

      i3 : runExternalM2ReturnAnswer("/tmp/M2-94269-0/7.ans",identity (cokernel(map(R^2,R^{3:{-1}},{{(9/2)*x+(9/4)*y, (7/9)*x+(7/10)*y, 7*x+(3/7)*y}, {(3/4)*x+(7/4)*y, (7/10)*x+(7/3)*y, (6/7)*x+6*y}}))));
      stdio:4:74:(3):[1]: error: no method for binary operator ^ applied to objects:
                  R (of class Symbol)
            ^     2 (of class ZZ)

Keep in mind that the object you are passing must make sense in the context of the file containing your function! For instance, here we need to define the ring:

i26 : fn<<///R=QQ[x,y];///<<endl<<flush;

i27 : (runExternalM2(fn,identity,v))#value===v
Running (true && (${prefix}/bin/M2-binary  --stop --no-debug --silent  -q  <"/tmp/M2-94269-0/8.m2" >"/tmp/M2-94269-0/8.out" 2>&1 ))
Finished running.

o27 = true

This problem can be avoided by following some suggestions for using RunExternalM2.

The objects may unavoidably lose some internal references, though:

i28 : v=R;

i29 : h=runExternalM2(fn,identity,v);
Running (true && (${prefix}/bin/M2-binary  --stop --no-debug --silent  -q  <"/tmp/M2-94269-0/9.m2" >"/tmp/M2-94269-0/9.out" 2>&1 ))
Finished running.

i30 : h#value

o30 = QQ[x..y]

o30 : PolynomialRing

i31 : v===h#value

o31 = false

but this happens because

i32 : R===value(toExternalString(R))

o32 = false

For the programmer

The object runExternalM2 is a function closure.

The source of this document is in RunExternalM2.m2:894:0.

runExternalM2 -- run a Macaulay2 function in a new Macaulay2 process

Description

See also

For the programmer