138 lines
24 KiB
HTML
138 lines
24 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en"><head><meta charset="UTF-8"/><meta name="viewport" content="width=device-width, initial-scale=1.0"/><title>Running External Programs · The Julia Language</title><script>(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
|
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
|
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
|
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
|
|
|
|
ga('create', 'UA-28835595-6', 'auto');
|
|
ga('send', 'pageview');
|
|
</script><link href="https://cdnjs.cloudflare.com/ajax/libs/normalize/4.2.0/normalize.min.css" rel="stylesheet" type="text/css"/><link href="https://fonts.googleapis.com/css?family=Lato|Roboto+Mono" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.6.3/css/font-awesome.min.css" rel="stylesheet" type="text/css"/><script>documenterBaseURL=".."</script><script src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.2.0/require.min.js" data-main="../assets/documenter.js"></script><script src="../siteinfo.js"></script><script src="../../versions.js"></script><link href="../assets/highlightjs/default.css" rel="stylesheet" type="text/css"/><link href="../assets/documenter.css" rel="stylesheet" type="text/css"/></head><body><nav class="toc"><a href="../index.html"><img class="logo" src="../assets/logo.png" alt="The Julia Language logo"/></a><h1>The Julia Language</h1><select id="version-selector" onChange="window.location.href=this.value" style="visibility: hidden"></select><form class="search" action="../search.html"><input id="search-query" name="q" type="text" placeholder="Search docs"/></form><ul><li><a class="toctext" href="../index.html">Home</a></li><li><span class="toctext">Manual</span><ul><li><a class="toctext" href="introduction.html">Introduction</a></li><li><a class="toctext" href="getting-started.html">Getting Started</a></li><li><a class="toctext" href="variables.html">Variables</a></li><li><a class="toctext" href="integers-and-floating-point-numbers.html">Integers and Floating-Point Numbers</a></li><li><a class="toctext" href="mathematical-operations.html">Mathematical Operations and Elementary Functions</a></li><li><a class="toctext" href="complex-and-rational-numbers.html">Complex and Rational Numbers</a></li><li><a class="toctext" href="strings.html">Strings</a></li><li><a class="toctext" href="functions.html">Functions</a></li><li><a class="toctext" href="control-flow.html">Control Flow</a></li><li><a class="toctext" href="variables-and-scoping.html">Scope of Variables</a></li><li><a class="toctext" href="types.html">Types</a></li><li><a class="toctext" href="methods.html">Methods</a></li><li><a class="toctext" href="constructors.html">Constructors</a></li><li><a class="toctext" href="conversion-and-promotion.html">Conversion and Promotion</a></li><li><a class="toctext" href="interfaces.html">Interfaces</a></li><li><a class="toctext" href="modules.html">Modules</a></li><li><a class="toctext" href="documentation.html">Documentation</a></li><li><a class="toctext" href="metaprogramming.html">Metaprogramming</a></li><li><a class="toctext" href="arrays.html">Multi-dimensional Arrays</a></li><li><a class="toctext" href="linear-algebra.html">Linear algebra</a></li><li><a class="toctext" href="networking-and-streams.html">Networking and Streams</a></li><li><a class="toctext" href="parallel-computing.html">Parallel Computing</a></li><li><a class="toctext" href="dates.html">Date and DateTime</a></li><li><a class="toctext" href="interacting-with-julia.html">Interacting With Julia</a></li><li class="current"><a class="toctext" href="running-external-programs.html">Running External Programs</a><ul class="internal"><li><a class="toctext" href="#command-interpolation-1">Interpolation</a></li><li><a class="toctext" href="#Quoting-1">Quoting</a></li><li><a class="toctext" href="#Pipelines-1">Pipelines</a></li></ul></li><li><a class="toctext" href="calling-c-and-fortran-code.html">Calling C and Fortran Code</a></li><li><a class="toctext" href="handling-operating-system-variation.html">Handling Operating System Variation</a></li><li><a class="toctext" href="environment-variables.html">Environment Variables</a></li><li><a class="toctext" href="embedding.html">Embedding Julia</a></li><li><a class="toctext" href="packages.html">Packages</a></li><li><a class="toctext" href="profile.html">Profiling</a></li><li><a class="toctext" href="stacktraces.html">Stack Traces</a></li><li><a class="toctext" href="performance-tips.html">Performance Tips</a></li><li><a class="toctext" href="workflow-tips.html">Workflow Tips</a></li><li><a class="toctext" href="style-guide.html">Style Guide</a></li><li><a class="toctext" href="faq.html">Frequently Asked Questions</a></li><li><a class="toctext" href="noteworthy-differences.html">Noteworthy Differences from other Languages</a></li><li><a class="toctext" href="unicode-input.html">Unicode Input</a></li></ul></li><li><span class="toctext">Standard Library</span><ul><li><a class="toctext" href="../stdlib/base.html">Essentials</a></li><li><a class="toctext" href="../stdlib/collections.html">Collections and Data Structures</a></li><li><a class="toctext" href="../stdlib/math.html">Mathematics</a></li><li><a class="toctext" href="../stdlib/numbers.html">Numbers</a></li><li><a class="toctext" href="../stdlib/strings.html">Strings</a></li><li><a class="toctext" href="../stdlib/arrays.html">Arrays</a></li><li><a class="toctext" href="../stdlib/parallel.html">Tasks and Parallel Computing</a></li><li><a class="toctext" href="../stdlib/linalg.html">Linear Algebra</a></li><li><a class="toctext" href="../stdlib/constants.html">Constants</a></li><li><a class="toctext" href="../stdlib/file.html">Filesystem</a></li><li><a class="toctext" href="../stdlib/io-network.html">I/O and Network</a></li><li><a class="toctext" href="../stdlib/punctuation.html">Punctuation</a></li><li><a class="toctext" href="../stdlib/sort.html">Sorting and Related Functions</a></li><li><a class="toctext" href="../stdlib/pkg.html">Package Manager Functions</a></li><li><a class="toctext" href="../stdlib/dates.html">Dates and Time</a></li><li><a class="toctext" href="../stdlib/iterators.html">Iteration utilities</a></li><li><a class="toctext" href="../stdlib/test.html">Unit Testing</a></li><li><a class="toctext" href="../stdlib/c.html">C Interface</a></li><li><a class="toctext" href="../stdlib/libc.html">C Standard Library</a></li><li><a class="toctext" href="../stdlib/libdl.html">Dynamic Linker</a></li><li><a class="toctext" href="../stdlib/profile.html">Profiling</a></li><li><a class="toctext" href="../stdlib/stacktraces.html">StackTraces</a></li><li><a class="toctext" href="../stdlib/simd-types.html">SIMD Support</a></li></ul></li><li><span class="toctext">Developer Documentation</span><ul><li><a class="toctext" href="../devdocs/reflection.html">Reflection and introspection</a></li><li><span class="toctext">Documentation of Julia's Internals</span><ul><li><a class="toctext" href="../devdocs/init.html">Initialization of the Julia runtime</a></li><li><a class="toctext" href="../devdocs/ast.html">Julia ASTs</a></li><li><a class="toctext" href="../devdocs/types.html">More about types</a></li><li><a class="toctext" href="../devdocs/object.html">Memory layout of Julia Objects</a></li><li><a class="toctext" href="../devdocs/eval.html">Eval of Julia code</a></li><li><a class="toctext" href="../devdocs/callconv.html">Calling Conventions</a></li><li><a class="toctext" href="../devdocs/compiler.html">High-level Overview of the Native-Code Generation Process</a></li><li><a class="toctext" href="../devdocs/functions.html">Julia Functions</a></li><li><a class="toctext" href="../devdocs/cartesian.html">Base.Cartesian</a></li><li><a class="toctext" href="../devdocs/meta.html">Talking to the compiler (the <code>:meta</code> mechanism)</a></li><li><a class="toctext" href="../devdocs/subarrays.html">SubArrays</a></li><li><a class="toctext" href="../devdocs/sysimg.html">System Image Building</a></li><li><a class="toctext" href="../devdocs/llvm.html">Working with LLVM</a></li><li><a class="toctext" href="../devdocs/stdio.html">printf() and stdio in the Julia runtime</a></li><li><a class="toctext" href="../devdocs/boundscheck.html">Bounds checking</a></li><li><a class="toctext" href="../devdocs/locks.html">Proper maintenance and care of multi-threading locks</a></li><li><a class="toctext" href="../devdocs/offset-arrays.html">Arrays with custom indices</a></li><li><a class="toctext" href="../devdocs/libgit2.html">Base.LibGit2</a></li><li><a class="toctext" href="../devdocs/require.html">Module loading</a></li></ul></li><li><span class="toctext">Developing/debugging Julia's C code</span><ul><li><a class="toctext" href="../devdocs/backtraces.html">Reporting and analyzing crashes (segfaults)</a></li><li><a class="toctext" href="../devdocs/debuggingtips.html">gdb debugging tips</a></li><li><a class="toctext" href="../devdocs/valgrind.html">Using Valgrind with Julia</a></li><li><a class="toctext" href="../devdocs/sanitizers.html">Sanitizer support</a></li></ul></li></ul></li></ul></nav><article id="docs"><header><nav><ul><li>Manual</li><li><a href="running-external-programs.html">Running External Programs</a></li></ul><a class="edit-page" href="https://github.com/JuliaLang/julia/tree/d386e40c17d43b79fc89d3e579fc04547241787c/doc/src/manual/running-external-programs.md"><span class="fa"></span> Edit on GitHub</a></nav><hr/><div id="topbar"><span>Running External Programs</span><a class="fa fa-bars" href="#"></a></div></header><h1><a class="nav-anchor" id="Running-External-Programs-1" href="#Running-External-Programs-1">Running External Programs</a></h1><p>Julia borrows backtick notation for commands from the shell, Perl, and Ruby. However, in Julia, writing</p><pre><code class="language-julia-repl">julia> `echo hello`
|
|
`echo hello`</code></pre><p>differs in several aspects from the behavior in various shells, Perl, or Ruby:</p><ul><li><p>Instead of immediately running the command, backticks create a <code>Cmd</code> object to represent the command. You can use this object to connect the command to others via pipes, run it, and read or write to it.</p></li><li><p>When the command is run, Julia does not capture its output unless you specifically arrange for it to. Instead, the output of the command by default goes to <a href="../stdlib/io-network.html#Base.STDOUT"><code>STDOUT</code></a> as it would using <code>libc</code>'s <code>system</code> call.</p></li><li><p>The command is never run with a shell. Instead, Julia parses the command syntax directly, appropriately interpolating variables and splitting on words as the shell would, respecting shell quoting syntax. The command is run as <code>julia</code>'s immediate child process, using <code>fork</code> and <code>exec</code> calls.</p></li></ul><p>Here's a simple example of running an external program:</p><pre><code class="language-julia-repl">julia> mycommand = `echo hello`
|
|
`echo hello`
|
|
|
|
julia> typeof(mycommand)
|
|
Cmd
|
|
|
|
julia> run(mycommand)
|
|
hello</code></pre><p>The <code>hello</code> is the output of the <code>echo</code> command, sent to <a href="../stdlib/io-network.html#Base.STDOUT"><code>STDOUT</code></a>. The run method itself returns <code>nothing</code>, and throws an <a href="../stdlib/base.html#Core.ErrorException"><code>ErrorException</code></a> if the external command fails to run successfully.</p><p>If you want to read the output of the external command, <a href="../stdlib/io-network.html#Base.readstring"><code>readstring()</code></a> can be used instead:</p><pre><code class="language-julia-repl">julia> a = readstring(`echo hello`)
|
|
"hello\n"
|
|
|
|
julia> chomp(a) == "hello"
|
|
true</code></pre><p>More generally, you can use <a href="../stdlib/io-network.html#Base.open"><code>open()</code></a> to read from or write to an external command.</p><pre><code class="language-julia-repl">julia> open(`less`, "w", STDOUT) do io
|
|
for i = 1:3
|
|
println(io, i)
|
|
end
|
|
end
|
|
1
|
|
2
|
|
3</code></pre><h2><a class="nav-anchor" id="command-interpolation-1" href="#command-interpolation-1">Interpolation</a></h2><p>Suppose you want to do something a bit more complicated and use the name of a file in the variable <code>file</code> as an argument to a command. You can use <code>$</code> for interpolation much as you would in a string literal (see <a href="../devdocs/ast.html#Strings-1">Strings</a>):</p><pre><code class="language-julia-repl">julia> file = "/etc/passwd"
|
|
"/etc/passwd"
|
|
|
|
julia> `sort $file`
|
|
`sort /etc/passwd`</code></pre><p>A common pitfall when running external programs via a shell is that if a file name contains characters that are special to the shell, they may cause undesirable behavior. Suppose, for example, rather than <code>/etc/passwd</code>, we wanted to sort the contents of the file <code>/Volumes/External HD/data.csv</code>. Let's try it:</p><pre><code class="language-julia-repl">julia> file = "/Volumes/External HD/data.csv"
|
|
"/Volumes/External HD/data.csv"
|
|
|
|
julia> `sort $file`
|
|
`sort '/Volumes/External HD/data.csv'`</code></pre><p>How did the file name get quoted? Julia knows that <code>file</code> is meant to be interpolated as a single argument, so it quotes the word for you. Actually, that is not quite accurate: the value of <code>file</code> is never interpreted by a shell, so there's no need for actual quoting; the quotes are inserted only for presentation to the user. This will even work if you interpolate a value as part of a shell word:</p><pre><code class="language-julia-repl">julia> path = "/Volumes/External HD"
|
|
"/Volumes/External HD"
|
|
|
|
julia> name = "data"
|
|
"data"
|
|
|
|
julia> ext = "csv"
|
|
"csv"
|
|
|
|
julia> `sort $path/$name.$ext`
|
|
`sort '/Volumes/External HD/data.csv'`</code></pre><p>As you can see, the space in the <code>path</code> variable is appropriately escaped. But what if you <em>want</em> to interpolate multiple words? In that case, just use an array (or any other iterable container):</p><pre><code class="language-julia-repl">julia> files = ["/etc/passwd","/Volumes/External HD/data.csv"]
|
|
2-element Array{String,1}:
|
|
"/etc/passwd"
|
|
"/Volumes/External HD/data.csv"
|
|
|
|
julia> `grep foo $files`
|
|
`grep foo /etc/passwd '/Volumes/External HD/data.csv'`</code></pre><p>If you interpolate an array as part of a shell word, Julia emulates the shell's <code>{a,b,c}</code> argument generation:</p><pre><code class="language-julia-repl">julia> names = ["foo","bar","baz"]
|
|
3-element Array{String,1}:
|
|
"foo"
|
|
"bar"
|
|
"baz"
|
|
|
|
julia> `grep xylophone $names.txt`
|
|
`grep xylophone foo.txt bar.txt baz.txt`</code></pre><p>Moreover, if you interpolate multiple arrays into the same word, the shell's Cartesian product generation behavior is emulated:</p><pre><code class="language-julia-repl">julia> names = ["foo","bar","baz"]
|
|
3-element Array{String,1}:
|
|
"foo"
|
|
"bar"
|
|
"baz"
|
|
|
|
julia> exts = ["aux","log"]
|
|
2-element Array{String,1}:
|
|
"aux"
|
|
"log"
|
|
|
|
julia> `rm -f $names.$exts`
|
|
`rm -f foo.aux foo.log bar.aux bar.log baz.aux baz.log`</code></pre><p>Since you can interpolate literal arrays, you can use this generative functionality without needing to create temporary array objects first:</p><pre><code class="language-julia-repl">julia> `rm -rf $["foo","bar","baz","qux"].$["aux","log","pdf"]`
|
|
`rm -rf foo.aux foo.log foo.pdf bar.aux bar.log bar.pdf baz.aux baz.log baz.pdf qux.aux qux.log qux.pdf`</code></pre><h2><a class="nav-anchor" id="Quoting-1" href="#Quoting-1">Quoting</a></h2><p>Inevitably, one wants to write commands that aren't quite so simple, and it becomes necessary to use quotes. Here's a simple example of a Perl one-liner at a shell prompt:</p><pre><code class="language-none">sh$ perl -le '$|=1; for (0..3) { print }'
|
|
0
|
|
1
|
|
2
|
|
3</code></pre><p>The Perl expression needs to be in single quotes for two reasons: so that spaces don't break the expression into multiple shell words, and so that uses of Perl variables like <code>$|</code> (yes, that's the name of a variable in Perl), don't cause interpolation. In other instances, you may want to use double quotes so that interpolation <em>does</em> occur:</p><pre><code class="language-none">sh$ first="A"
|
|
sh$ second="B"
|
|
sh$ perl -le '$|=1; print for @ARGV' "1: $first" "2: $second"
|
|
1: A
|
|
2: B</code></pre><p>In general, the Julia backtick syntax is carefully designed so that you can just cut-and-paste shell commands as is into backticks and they will work: the escaping, quoting, and interpolation behaviors are the same as the shell's. The only difference is that the interpolation is integrated and aware of Julia's notion of what is a single string value, and what is a container for multiple values. Let's try the above two examples in Julia:</p><pre><code class="language-julia-repl">julia> A = `perl -le '$|=1; for (0..3) { print }'`
|
|
`perl -le '$|=1; for (0..3) { print }'`
|
|
|
|
julia> run(A)
|
|
0
|
|
1
|
|
2
|
|
3
|
|
|
|
julia> first = "A"; second = "B";
|
|
|
|
julia> B = `perl -le 'print for @ARGV' "1: $first" "2: $second"`
|
|
`perl -le 'print for @ARGV' '1: A' '2: B'`
|
|
|
|
julia> run(B)
|
|
1: A
|
|
2: B</code></pre><p>The results are identical, and Julia's interpolation behavior mimics the shell's with some improvements due to the fact that Julia supports first-class iterable objects while most shells use strings split on spaces for this, which introduces ambiguities. When trying to port shell commands to Julia, try cut and pasting first. Since Julia shows commands to you before running them, you can easily and safely just examine its interpretation without doing any damage.</p><h2><a class="nav-anchor" id="Pipelines-1" href="#Pipelines-1">Pipelines</a></h2><p>Shell metacharacters, such as <code>|</code>, <code>&</code>, and <code>></code>, need to be quoted (or escaped) inside of Julia's backticks:</p><pre><code class="language-julia-repl">julia> run(`echo hello '|' sort`)
|
|
hello | sort
|
|
|
|
julia> run(`echo hello \| sort`)
|
|
hello | sort</code></pre><p>This expression invokes the <code>echo</code> command with three words as arguments: <code>hello</code>, <code>|</code>, and <code>sort</code>. The result is that a single line is printed: <code>hello | sort</code>. How, then, does one construct a pipeline? Instead of using <code>'|'</code> inside of backticks, one uses <a href="../stdlib/base.html#Base.pipeline-Tuple{Any,Any,Any,Vararg{Any,N} where N}"><code>pipeline()</code></a>:</p><pre><code class="language-julia-repl">julia> run(pipeline(`echo hello`, `sort`))
|
|
hello</code></pre><p>This pipes the output of the <code>echo</code> command to the <code>sort</code> command. Of course, this isn't terribly interesting since there's only one line to sort, but we can certainly do much more interesting things:</p><pre><code class="language-julia-repl">julia> run(pipeline(`cut -d: -f3 /etc/passwd`, `sort -n`, `tail -n5`))
|
|
210
|
|
211
|
|
212
|
|
213
|
|
214</code></pre><p>This prints the highest five user IDs on a UNIX system. The <code>cut</code>, <code>sort</code> and <code>tail</code> commands are all spawned as immediate children of the current <code>julia</code> process, with no intervening shell process. Julia itself does the work to setup pipes and connect file descriptors that is normally done by the shell. Since Julia does this itself, it retains better control and can do some things that shells cannot.</p><p>Julia can run multiple commands in parallel:</p><pre><code class="language-julia-repl">julia> run(`echo hello` & `echo world`)
|
|
world
|
|
hello</code></pre><p>The order of the output here is non-deterministic because the two <code>echo</code> processes are started nearly simultaneously, and race to make the first write to the <a href="../stdlib/io-network.html#Base.STDOUT"><code>STDOUT</code></a> descriptor they share with each other and the <code>julia</code> parent process. Julia lets you pipe the output from both of these processes to another program:</p><pre><code class="language-julia-repl">julia> run(pipeline(`echo world` & `echo hello`, `sort`))
|
|
hello
|
|
world</code></pre><p>In terms of UNIX plumbing, what's happening here is that a single UNIX pipe object is created and written to by both <code>echo</code> processes, and the other end of the pipe is read from by the <code>sort</code> command.</p><p>IO redirection can be accomplished by passing keyword arguments stdin, stdout, and stderr to the <code>pipeline</code> function:</p><pre><code class="language-julia">pipeline(`do_work`, stdout=pipeline(`sort`, "out.txt"), stderr="errs.txt")</code></pre><h3><a class="nav-anchor" id="Avoiding-Deadlock-in-Pipelines-1" href="#Avoiding-Deadlock-in-Pipelines-1">Avoiding Deadlock in Pipelines</a></h3><p>When reading and writing to both ends of a pipeline from a single process, it is important to avoid forcing the kernel to buffer all of the data.</p><p>For example, when reading all of the output from a command, call <code>readstring(out)</code>, not <code>wait(process)</code>, since the former will actively consume all of the data written by the process, whereas the latter will attempt to store the data in the kernel's buffers while waiting for a reader to be connected.</p><p>Another common solution is to separate the reader and writer of the pipeline into separate Tasks:</p><pre><code class="language-julia">writer = @async writeall(process, "data")
|
|
reader = @async do_compute(readstring(process))
|
|
wait(process)
|
|
fetch(reader)</code></pre><h3><a class="nav-anchor" id="Complex-Example-1" href="#Complex-Example-1">Complex Example</a></h3><p>The combination of a high-level programming language, a first-class command abstraction, and automatic setup of pipes between processes is a powerful one. To give some sense of the complex pipelines that can be created easily, here are some more sophisticated examples, with apologies for the excessive use of Perl one-liners:</p><pre><code class="language-julia-repl">julia> prefixer(prefix, sleep) = `perl -nle '$|=1; print "'$prefix' ", $_; sleep '$sleep';'`;
|
|
|
|
julia> run(pipeline(`perl -le '$|=1; for(0..9){ print; sleep 1 }'`, prefixer("A",2) & prefixer("B",2)))
|
|
A 0
|
|
B 1
|
|
A 2
|
|
B 3
|
|
A 4
|
|
B 5
|
|
A 6
|
|
B 7
|
|
A 8
|
|
B 9</code></pre><p>This is a classic example of a single producer feeding two concurrent consumers: one <code>perl</code> process generates lines with the numbers 0 through 9 on them, while two parallel processes consume that output, one prefixing lines with the letter "A", the other with the letter "B". Which consumer gets the first line is non-deterministic, but once that race has been won, the lines are consumed alternately by one process and then the other. (Setting <code>$|=1</code> in Perl causes each print statement to flush the <a href="../stdlib/io-network.html#Base.STDOUT"><code>STDOUT</code></a> handle, which is necessary for this example to work. Otherwise all the output is buffered and printed to the pipe at once, to be read by just one consumer process.)</p><p>Here is an even more complex multi-stage producer-consumer example:</p><pre><code class="language-julia-repl">julia> run(pipeline(`perl -le '$|=1; for(0..9){ print; sleep 1 }'`,
|
|
prefixer("X",3) & prefixer("Y",3) & prefixer("Z",3),
|
|
prefixer("A",2) & prefixer("B",2)))
|
|
A X 0
|
|
B Y 1
|
|
A Z 2
|
|
B X 3
|
|
A Y 4
|
|
B Z 5
|
|
A X 6
|
|
B Y 7
|
|
A Z 8
|
|
B X 9</code></pre><p>This example is similar to the previous one, except there are two stages of consumers, and the stages have different latency so they use a different number of parallel workers, to maintain saturated throughput.</p><p>We strongly encourage you to try all these examples to see how they work.</p><footer><hr/><a class="previous" href="interacting-with-julia.html"><span class="direction">Previous</span><span class="title">Interacting With Julia</span></a><a class="next" href="calling-c-and-fortran-code.html"><span class="direction">Next</span><span class="title">Calling C and Fortran Code</span></a></footer></article></body></html>
|